Target detection method and device

Abstract

The invention relates to a target detection method and device, and the method comprises the steps: obtaining an original point cloud collected by a laser radar and an image collected by a camera, and enabling the original point cloud and the image to be in time-space synchronization; points in the original point cloud are projected to the image, a plurality of projection points are obtained, and each projection point in the plurality of projection points corresponds to one point in the original point cloud; determining an image area where each target in the image is located, wherein the image area of each target is a partial area of the image; according to the density distribution information of the projection points in the image area where each target is located, pixel point sampling is carried out in the image area where each target is located, and a sampling point corresponding to each target is obtained; determining a fusion point cloud according to the sampling point corresponding to each target and the original point cloud; and performing fusion target detection on the fusion point cloud and the image to obtain a target detection result. According to the method, the original point cloud is densified, and the target detection effect can be improved by using the densified fusion point cloud and the image to carry out fusion target detection.

Description

Technical Field

[0001] The present application belongs to the field of point cloud data processing technology, and in particular, to a target detection method and device. Background Art

[0002] Target detection methods based on multiple sensor fusion can provide more information, thereby improving the accuracy and reliability of target detection. Therefore, they are increasingly used in fields such as autonomous driving and mobile robots. For example, images generated by cameras can provide rich visual information such as color, shape, texture, etc., but cannot provide 3D (3-Dimension) geometric information and are easily affected by lighting; while the point cloud generated by LiDAR can provide accurate 3D geometric information and is insensitive to ambient light, but the point cloud is sparse and LiDAR is expensive. Therefore, using the complementary characteristics of these two sensors for 3D target detection is currently a popular and effective multi-sensor fusion target detection solution.

[0003] In the process of target detection using LiDAR and camera fusion, the point cloud generated by LiDAR at a long distance or for a small target to be detected is very sparse. Since the sparse point cloud has a poor representation of the features of the target to be detected, the accuracy of LiDAR and camera fusion detection is low. In a longer detection range or for a smaller target to be detected, the missed detection rate and false detection rate of LiDAR and camera fusion target detection are high, and the detection effect is poor.

[0004] Therefore, how to improve the detection effect of LiDAR and camera fusion target detection for targets that are far away or small in size has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The embodiments of the present application provide a target detection method and device, which can solve the technical problem in the prior art that the target detection effect of laser radar and camera fusion is poor in a relatively long detection range or for a smaller target to be detected.

[0006] In a first aspect, an embodiment of the present application provides a target detection method, including: acquiring an original point cloud collected by a laser radar and an image collected by a camera, wherein the original point cloud and the image are synchronized in time and space; projecting points in the original point cloud onto the image to obtain multiple projection points, wherein each projection point in the multiple projection points corresponds to a point in the original point cloud; determining an image area where each target in the image is located, wherein the image area of ​​each target is a partial area of ​​the image; sampling pixels in the image area where each target is located according to density distribution information of projection points in the image area where each target is located, and obtaining sampling points corresponding to each target; determining a fused point cloud according to the sampling points corresponding to each target and the original point cloud; performing fused target detection on the fused point cloud and the image to obtain a target detection result.

[0007] In the above method, a fused point cloud is obtained through the sampling points and the original point cloud of the image area of ​​each target, and then the fused target detection is performed based on the fused point cloud and the image, so that the number of point clouds of each target is increased, and the original point cloud can be densified; the use of the densified fused point cloud and image for fused target detection can improve the effect of target detection. And because the pixel points of the image area of ​​each target are sampled according to the density distribution information of the projection points in the image area of ​​each target to obtain the sampling points corresponding to each target, the obtained sampling points have a stronger correlation with the target, and the consistency between the fused point cloud and the target in the actual situation is improved, thereby improving the accuracy of the final detection result.

[0008] In one of the embodiments, pixel sampling is performed in the image area where each target is located according to density distribution information of projection points in the image area where each target is located to obtain sampling points corresponding to each target, including: dividing the image area where each target is located into multiple sub-areas, in which: each sub-area is a partial area of ​​the image area, and any two sub-areas do not completely overlap; according to the projection points in the image area where each target is located, projection point density information of each sub-area in the multiple sub-areas is determined; the sub-area corresponding to the projection point density information that meets the first preset condition is determined as the target sub-area; pixel sampling is performed in the target sub-area to obtain sampling points corresponding to each target.

[0009] In one embodiment, the image region where each target is located is obtained by expanding the two-dimensional detection frame of each target, and the size of each sub-region of each target is the same as the size of the two-dimensional detection frame of each target.

[0010] In one of the embodiments, the image area where each target is located is divided into multiple sub-areas, including: using the two-dimensional detection frame of each target as a sliding window, performing a sliding window operation in the image area where each target is located, and obtaining multiple window areas, each of the multiple window areas is a sub-area in the multiple sub-areas.

[0011] In one embodiment, determining the image region where each target in the image is located includes: performing target detection on the image to obtain a two-dimensional detection frame of each target in the image; and expanding the two-dimensional detection frame of each target to obtain the image region where each target is located.

[0012] In one of the embodiments, the two-dimensional detection frame of each target is expanded to obtain the image area where each target is located, including: obtaining an expansion parameter corresponding to each target according to the size of each target and the size of the time synchronization error, the time synchronization error indicating the degree of time asynchrony between the camera and the lidar; and expanding the two-dimensional detection frame of each target according to the expansion parameter corresponding to each target to obtain the image area of ​​each target.

[0013] In one embodiment, each target corresponds to multiple sampling points, and a fused point cloud is determined based on the sampling points corresponding to each target and the original point cloud, including: determining a target projection point corresponding to a first sampling point from multiple projection points, the first sampling point being any one of the multiple sampling points corresponding to each target, and the target projection point being part of the multiple projection points; determining a depth value of the first sampling point based on the depth value of the target projection point; using the depth value of the first sampling point, converting the first sampling point to the laser radar coordinate system of the laser radar to obtain a representation point of the first sampling point; fusing multiple representation points corresponding to each target with the original point cloud to obtain a fused point cloud. In this embodiment, by determining the representation point after each sampling is converted to the laser radar coordinate system, the representation point is the point cloud point corresponding to the sampling point in the laser radar coordinate system, and fusing the representation point with the original point cloud to obtain a fused point cloud, the point cloud is densified, thereby improving the accuracy of subsequent target detection.

[0014] In one embodiment, the target projection point includes: a projection point located within a preset range centered at the first sampling point among the multiple projection points.

[0015] In one of the embodiments, determining a target projection point corresponding to a first sampling point from a plurality of projection points includes: if no projection point exists within a preset range centered on the first sampling point, determining a projection point among the plurality of projection points that is closest to the first sampling point as the target projection point corresponding to the first sampling point.

[0016] In one of the embodiments, the depth value of the first sampling point is determined according to the depth value of the target projection point, including: if the first sampling point corresponds to one target projection point, the depth value of the one target projection point is determined as the depth value of the first sampling point; if the first sampling point corresponds to multiple target projection points, the average value of the depth values ​​of the multiple target projection points is determined as the depth value of the first sampling point.

[0017] In a second aspect, an embodiment of the present application provides a target detection device, which includes a unit for executing each step of the method as described in any one of the first aspects above.

[0018] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the methods described in the first aspect when executing the computer program.

[0019] In a fourth aspect, an embodiment of the present application provides a target detection system, the system comprising: a laser radar, a camera, and a computer device as described in any one of the second aspects above.

[0020] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above first aspects is implemented.

[0021] In a sixth aspect, an embodiment of the present application provides a chip, comprising: a processor, configured to call and run a computer program from a memory, so that a computer device equipped with the chip executes a method as described in any one of the above first aspects.

[0022] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 is a schematic diagram of the structure of a target detection system provided by an embodiment of the present application;

[0025] Figure 2 is a flowchart of a target detection method provided by an embodiment of the present application;

[0026] Figure 3 is a schematic diagram of projecting an original point cloud onto an image to form projection points in one embodiment of the present application;

[0027] Figure 4 It is a schematic diagram of using a two-dimensional detection frame as a sliding window to determine multiple sub-regions and a target sub-region and obtain sampling points in an embodiment of the present application;

[0028] Figure 5 is a schematic diagram of a fused point cloud obtained in an embodiment of the present application;

[0029] Figure 6 It is a schematic diagram of a process for determining a fused point cloud in a target detection method provided in an embodiment of the present application;

[0030] Figure 7 is a structural block diagram of a target detection device provided by an embodiment of the present application;

[0031] Figure 8 It is an internal structure diagram of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0032] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0033] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0034] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0035] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0036] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0037] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0038] An embodiment of the present application provides a target detection method, which includes: acquiring an original point cloud collected by a laser radar and an image collected by a camera, wherein the original point cloud and the image are synchronized in time and space; projecting points in the original point cloud onto the image to obtain multiple projection points, wherein each projection point in the multiple projection points corresponds to a point in the original point cloud; determining an image area where each target in the image is located, wherein the image area of ​​each target is a partial area of ​​the image; sampling pixels in the image area where each target is located according to density distribution information of projection points in the image area where each target is located to obtain sampling points corresponding to each target; determining a fused point cloud according to the sampling points corresponding to each target and the original point cloud; performing fused target detection on the fused point cloud and the image to obtain a target detection result.

[0039] The target detection method in the embodiment of the present application obtains a fused point cloud by combining the sampling points and the original point cloud of the image area where each target is located, and then performs fused target detection based on the fused point cloud and the image, so that the number of point clouds of each target will increase, so that the original point cloud can be densified; the use of the densified fused point cloud and the image for fused target detection can improve the effect of target detection. And because the image area of ​​each target is sampled according to the density distribution information of the projection points in the image area of ​​each target, the sampling points corresponding to each target are obtained, so that the obtained sampling points have a stronger correlation with the target, and the consistency of the fused point cloud with the target in the actual situation is improved, thereby improving the accuracy of the final detection result. The target detection method in the embodiment of the present application performs point cloud densification in a targeted manner through the guidance of image and point cloud density, and can also have a good point cloud densification effect for the situation where the laser radar and the camera are not completely synchronized in time, thereby improving the robustness of the target detection method. And because the target detection method in the embodiment of the present application can make the sparse target point cloud dense at the algorithm level, there is no need to make changes in the hardware (such as increasing the number of radar lines, etc.), saving costs.

[0040] The target detection method provided in the present application is exemplarily described below in conjunction with specific embodiments.

[0041] See also Figure 1 , is a schematic diagram of the structure of the target detection system provided in the embodiment of the present application, such as Figure 1 As shown, the target detection system includes a laser radar 110, a camera 120 and a computing device 130, wherein the laser radar 110 and the camera 120 are respectively connected to the computing device 130 through a communication interface, wherein the communication interface can be a wired communication interface or a wireless communication interface, and the present application does not impose any limitation on this.

[0042] The laser radar 110 is used to collect point clouds for the target area to obtain point clouds (i.e., original point clouds) for the target area, and send the obtained point clouds to the computer device 130; the camera user collects images of the target area to obtain images of the target area, and sends the obtained images to the computer device 130. The computer device 130 obtains images and point clouds synchronized in time and space, and projects the points in the point clouds onto the image to obtain multiple projection points; the density distribution information of the image area where the target is located and the projection points is used to perform pixel point sampling in the image area of ​​each target, thereby obtaining sampling points corresponding to each target; the fused point cloud is determined according to the sampling points corresponding to each target and the original point cloud; the fused point cloud and the image are subjected to fused target detection to obtain target detection results.

[0043] It is understandable that the laser radar 110 and the camera 120 in the embodiment of the present application are used to collect point clouds and images of the same area, that is, the laser radar 110 and the camera 120 are two sensors in the multi-sensor fusion detection. For example, the laser radar 110 and the camera 120 can be either roadside sensors or vehicle-mounted sensors. The computer device 130 can be any device that has target detection task requirements. For example, the computer device 130 can be a road-side computer device, a vehicle-side computer device, or a combination system of a road-side computer device and a vehicle-side computer device. This application does not limit this.

[0044] Exemplarily, the laser radar 110 and the camera 120 may be a laser radar and a camera in a roadside fusion perception system. In this case, the computing device 130 may be a road-side computer device, for example, a computing unit in a roadside fusion perception system (also known as a smart base station or a roadside base station), for example, an industrial computer in a roadside fusion perception system.

[0045] It is understandable that the camera 120 in the embodiment of the present application should be understood as an image sensor, rather than a camera in a narrow sense. That is, the camera 120 in the embodiment of the present application is an image sensor that can take images for any user, and the present application does not limit this.

[0046] It should be understood that the target detection system in the embodiments of the present application can be applied to any scenario in which target detection is performed based on lidar point clouds and camera images. For example, the lidar 110, camera 120 and computer device 130 can also be an integrated target detection system, and the present application does not impose any restrictions on this.

[0047] Based on Figure 1 The target detection system shown in the present application provides, in one embodiment, Figure 2 The target detection method shown in the figure is described by taking the method applied to the above-mentioned computer device 130 as an example. It can be understood that the following description is only an example and does not constitute a limitation on the protection scope of the present application. Figure 2 As shown, the method may include S201 to S206. Each step is described below.

[0048] Step S201, obtaining the original point cloud collected by the laser radar and the image collected by the camera, wherein the original point cloud and the image are synchronized in time and space.

[0049] In the embodiment, time-space synchronization refers to time synchronization and space synchronization, that is, the original point cloud and the image are synchronized in time and space. Any method in the prior art can be used to perform time-space synchronization of the original point cloud and the image, which is not described in detail in this application.

[0050] It should be understood that in actual scenarios, it is often difficult to achieve complete synchronization of the original point cloud and the image in time, that is, there will be a small range of errors in the time synchronization of the original point cloud and the image.

[0051] It should be noted that the laser radar and camera in the embodiment of the present application have the same data collection area, that is, the scanning area of ​​the laser radar overlaps with the shooting area of ​​the camera. That is, the laser radar and the camera can collect data for the same target scene. In the embodiment of the present application, the camera and the laser radar are calibrated with external parameters, so that the coordinate transformation relationship between the image and the original point cloud is known.

[0052] Step S202: Project the points in the original point cloud onto the image to obtain a plurality of projection points, each of the plurality of projection points corresponding to a point in the original point cloud.

[0053] In an embodiment, after obtaining the original point cloud and the image, the points in the original point cloud are projected onto the image, wherein the projected points are the pixel points corresponding to the points in the point cloud.

[0054] It can be understood that projection is specifically a coordinate conversion process of converting points in the original point cloud from the lidar coordinate system to the pixel coordinate system. In the projection process, the points in the original point cloud are first converted from the lidar coordinate system to the camera coordinate system, and then converted from the camera coordinate system to the pixel coordinate system.

[0055] It should be understood that the coordinate system transformation can be performed based on the transformation relationship between the coordinate systems. The transformation relationship between the lidar coordinate system and the camera coordinate system in the embodiments of the present application, as well as the transformation relationship between the camera coordinate system and the pixel coordinate system are all known data, and this application will not elaborate on this.

[0056] It is understandable that other feasible coordinate system conversion methods are also within the protection scope of this application.

[0057] Step S203: determine the image region where each target in the image is located, where the image region of each target is a partial region of the image.

[0058] In the embodiment, the target in the image may be different according to the detection purpose. For example, the target in the image may be a traffic participant, and the target in the image may be a pedestrian or a vehicle, etc. The image area where each target in the image is located refers to the image area including the target in the image. For example, a two-dimensional detection box may be used to represent the image area of ​​the target.

[0059] In some embodiments, the image area of ​​each target may be a two-dimensional detection box including the target, and a target detection algorithm may be used to determine the two-dimensional detection box of each target. The target detection algorithm may be YOLO (You Only Look Once), RCNN, etc., and this application does not limit or enumerate this.

[0060] In some other embodiments, the image region of each target is the region after the two-dimensional detection frame is expanded, that is, the area of ​​the image region of each target is larger than the area of ​​the two-dimensional detection frame corresponding to the target.

[0061] Exemplarily, step S203 may include the following steps: performing target detection on the image to obtain a two-dimensional detection frame of each target in the image; and expanding the two-dimensional detection frame of each target to obtain the image region where each target is located. By expanding the two-dimensional detection frame to obtain the image region of each target, more points in the original point cloud can be projected to the image region of the target, so that even if the original point cloud and the image have poor time synchronization, the points corresponding to the target in the original point cloud can be projected to the expanded image region, thereby improving the robustness of the target detection method.

[0062] It is understandable that if the two-dimensional detection frame of the target is expanded too much, the image area will be too large, which will cause noise due to the inclusion of other targets; if the two-dimensional detection frame of the target is expanded too little, the image area will be too small, which will result in the points in the original point cloud being unable to be projected into the target area when the time synchronization error between the original point cloud and the image is large; therefore, the two-dimensional detection frame of the target is expanded to an appropriate range.

[0063] In some embodiments, the process of expanding the two-dimensional detection frame of the target includes the following steps: obtaining an expansion parameter corresponding to each target according to the size of each target and the size of the time synchronization error, the time synchronization error indicating the degree of time asynchrony between the camera and the laser radar; and expanding the two-dimensional detection frame of each target according to the expansion parameter corresponding to each target to obtain the image area of ​​each target. In this embodiment, by associating the expansion of the two-dimensional detection frame with the size of the target and the size of the time synchronization error, the correspondence between the expanded image area and the target is improved, ensuring that the expansion degree of the two-dimensional detection frame is more accurate and has a stronger correspondence with the target, so that the two-dimensional detection frame can be expanded to an appropriate range.

[0064] Figure 3 It is a schematic diagram of projecting an original point cloud onto an image to form projection points in one embodiment of the present application. Figure 3 In figure a of FIG. 3 , the two-dimensional detection frame 301 is used as the image region of the target 302 , and it can be seen that the image region of the target 302 is not aligned with the target 302 on the image. Figure 3In the figure b in FIG. 3 , a new image region 303 is formed after the two-dimensional detection frame is expanded. In this case, the new target image region 303 successfully covers the target 302, so the new image region 303 is a suitable image region.

[0065] Exemplarily, the expansion parameter is expressed as the expansion degree of the two-dimensional detection frame. The larger the expansion parameter, the larger the ratio of the image area of ​​the target to the area of ​​the two-dimensional detection frame. Therefore, the larger the target and the larger the time synchronization error, the larger the expansion parameter; the smaller the target and the smaller the time synchronization error, the smaller the expansion parameter.

[0066] It is understandable that if the target is small, the 2D detection frame is expanded too much, and the obtained image area will contain other targets and cause noise; if the original point cloud and the image differ greatly in time, it means that the offset of the points in the original point cloud projected onto the image will be large, so a larger image area is required to project all the points belonging to the target in the original point cloud into the image area of ​​the target. The expansion parameters of the 2D detection frame are set comprehensively by considering the target size and time synchronization error to improve the accuracy of the image area obtained by the expansion of the 2D detection frame.

[0067] It should be understood that the image area is related to the specific form of the target detection method, that is, the image area of ​​each target can also be any other possible form other than the two-dimensional detection box, and this application will not go into details.

[0068] Step S204: sampling pixels in the image region where each target is located according to the density distribution information of the projection points in the image region where each target is located, to obtain sampling points corresponding to each target.

[0069] In the embodiment, the sampling points are pixel points obtained by random sampling in the pixel area; the density distribution information of the projection points mainly reflects the aggregation of the projection points; generally speaking, in an area, the greater the density of the projection points, the higher the probability that the target appears in the area, so the sampling points obtained by sampling the pixel points in the area have a stronger correspondence with the target.

[0070] In some embodiments, step S204 includes: dividing the image area where each target is located into multiple sub-areas, in which: each sub-area is a partial area of ​​the image area, and any two sub-areas do not completely overlap; determining the projection point density information of each sub-area in the multiple sub-areas according to the projection points in the image area where each target is located; determining the sub-area corresponding to the projection point density information that meets the first preset condition as the target sub-area; sampling pixels in the target sub-area to obtain sampling points corresponding to each target. In this embodiment, the image area of ​​the target is divided into multiple sub-areas to determine the projection point density information in each sub-area, determine the target sub-area from the multiple sub-areas according to the projection point density information, and perform pixel sampling in the target sub-area to obtain sampling points corresponding to the target. In this method, by dividing the image area of ​​the target and filtering the target sub-area according to the projection point density information, the acquisition of sampling points is more targeted, and the correspondence between the sampling points and the target is ensured.

[0071] In some embodiments, the projection point density information of a sub-region may include a projection point density value, which is equal to the ratio of the number of projection points to the area of ​​the sub-region. The sub-region with the largest projection point density value may be determined as the target sub-region.

[0072] In some other embodiments, a coefficient k can be set for each sub-region, and the closer the sub-region is to the center of the image region, the larger the coefficient k is; the projection point density information of a sub-region can include a corrected projection point density value, and the corrected projection point density value is equal to the product of the coefficient k and the projection point density value, and the sub-region with the largest corrected projection point density can be determined as the target sub-region. Since the closer the sub-region is to the center of the image region, the greater the probability that the target is located in the sub-region; by introducing the coefficient k to correct the projection point density value by the position of the sub-region in the image region, the corrected projection point density value is closer to the actual situation, so that the determined target sub-region is more consistent with the actual position of the target in the image.

[0073] In some embodiments, the image region where each target is located is obtained by expanding the two-dimensional detection frame of each target, and the size of each sub-region of each target is the same as the size of the two-dimensional detection frame of each target. In this embodiment, since the image region of the target is obtained by expanding the two-dimensional detection frame of the target, the size of each sub-region is the same as the size of the two-dimensional detection frame. Such a setting makes the correspondence between the sub-region and the target better, thereby ensuring the correspondence between the sampling point and the target.

[0074] Exemplarily, the process of dividing the image area of ​​each target into sub-areas may include: using the two-dimensional detection frame of each target as a sliding window, performing a sliding window operation in the image area where each target is located, and obtaining multiple window areas, each of the multiple window areas being a sub-area in the multiple sub-areas. In this embodiment, the image area is divided by using the two-dimensional detection frame as a sliding window, and the method is simple and easy to implement.

[0075] like Figure 4 FIG. 1 is a schematic diagram showing an embodiment of the present application in which a two-dimensional detection frame is used as a sliding window to determine multiple sub-regions and a target sub-region and obtain sampling points. Figure 4 As shown in Figure a, in the target image area 401, a two-dimensional detection frame 402 is used as a sliding window to perform a sliding window operation, determine the projection point density information of each window, and then convert the projection point density information into Figure 4 The area where the window 403 in the b figure is located is the target sub-area. Figure 4 Random sampling is performed in the target sub-region of the b image in the figure to obtain Figure 4 There are 10 sampling points 404 shown in Figure c (only some sampling points are marked in the figure).

[0076] Step S205: Determine a fused point cloud according to the sampling points and the original point cloud corresponding to each target.

[0077] In the embodiment, generally speaking, each target corresponds to multiple sampling points. The fused point cloud is determined according to the sampling points corresponding to each target and the original point cloud, which can effectively increase the density of points in the point cloud based on the original point cloud.

[0078] Figure 5 FIG. 1 is a schematic diagram of a fused point cloud obtained in an embodiment of the present application. Figure 5 As shown, in Figure 5 There are three images in total, and each dotted rectangle corresponds to one image. Figure 5 There are two images on the left side of the figure. The upper left image 501 is the projected image obtained by projecting the original point cloud onto the image. The lower left image 502 is the fused point cloud at the first target 510 in the projected image. The right image 503 is the fused point cloud at the second target 520 in the projected image. Figure 5 As shown, near the first target 510 and the second target 520, the point cloud density increases.

[0079] Step S206: Perform fusion target detection on the fused point cloud and the image to obtain a target detection result.

[0080] In the embodiment, fused point cloud and image are subjected to fused target detection, specifically, the target detection result is obtained by inputting the fused point cloud and image into a point cloud image fusion detection algorithm.

[0081] It is understandable that the point cloud image fusion detection algorithm can adopt any algorithm currently available in academia or industry, and this application does not impose any limitation on this.

[0082] In the target detection method in the embodiment of the present application, a fused point cloud is obtained through the sampling points and the original point cloud of the image area of ​​each target, and then the fused target detection is performed based on the fused point cloud and the image, so that the number of point clouds of each target is increased, and the original point cloud can be densified; the use of the densified fused point cloud and the image for fused target detection can improve the effect of target detection. And because the image area of ​​each target is sampled for pixels according to the density distribution information of the projection points in the image area of ​​each target to obtain the sampling points corresponding to each target, the obtained sampling points have a stronger correlation with the target, and the consistency between the fused point cloud and the target in the actual situation is improved, thereby improving the accuracy of the final detection result.

[0083] For easier understanding, the process of determining the fused point cloud according to the sampling points and the original point cloud corresponding to each target is exemplarily described below. Figure 6 The following is a flow chart of determining a fused point cloud in a target detection method provided in an embodiment of the present application. Figure 6 As shown, the process of determining the fused point cloud includes: S601 to S604. Each step is described below.

[0084] Step S601: Determine a target projection point corresponding to a first sampling point from among multiple projection points, where the first sampling point is any one of the multiple sampling points corresponding to each target, and the target projection point is part of the multiple projection points.

[0085] Exemplarily, the target projection point corresponding to the first sampling point may be: a projection point located within a preset range centered on the first sampling point among the multiple projection points. For example, a projection point within a circular range with the first sampling point as the center and N pixels as the radius may be determined as the target projection point. In this embodiment, the projection points within the preset range have a stronger correlation with the first sampling point, so the method can be used to very simply obtain a suitable target projection point.

[0086] It is understandable that the preset range can be set according to the actual situation of the target size and the projection point, and this application will not go into details.

[0087] Exemplarily, if there is no projection point within a preset range centered on the first sampling point, a projection point among the multiple projection points closest to the first sampling point is determined as the target projection point corresponding to the first sampling point. This embodiment can ensure that each sampling point can obtain a target projection point.

[0088] Step S602: Determine the depth value of the first sampling point according to the depth value of the target projection point.

[0089] It can be understood that in the process of projecting the original point cloud to the pixel coordinate system to generate projection points, the points in the original point cloud are first converted from the lidar coordinate system to the camera coordinate system, and then converted from the camera coordinate system to the pixel coordinate system. When the points in the point cloud are converted to the camera's coordinate system, a depth value (or depth information) is generated. This depth value is the depth value of the corresponding projection point, so each projection point will correspond to a depth value.

[0090] In the embodiment, the sampling point is a pixel point sampled in the image area, and therefore does not have depth information (or depth value); if you want to convert the sampling point to the coordinate system of the laser radar, you need to assign depth information to the sampling point. In the embodiment of the present application, the target projection point corresponding to the sampling point is used to assign depth information to the sampling point. After obtaining the depth information of the sampling point, the sampling point is first converted from the pixel coordinate system to the camera coordinate system, and then the sampling point is assigned depth information, and then converted from the camera coordinate system to the laser radar coordinate system.

[0091] In some embodiments, the process of determining the depth value of the first sampling point includes: if the first sampling point corresponds to a target projection point, the depth value of the target projection point is determined as the depth value of the first sampling point; if the first sampling point corresponds to multiple target projection points, the average of the depth values ​​of the multiple target projection points is determined as the depth value of the first sampling point. In this embodiment, for sampling points corresponding to multiple target projection points, the average depth value of the multiple projection points is determined as the depth value of the sampling point, so that the depth value of the sampling point is closer to the actual situation; for sampling points corresponding to one target projection point, the depth value of the target projection point is determined as the depth value of the sampling point, ensuring that each sampling point can obtain a depth value through the target projection point, thereby ensuring that each sampling point can obtain a representation point, and then merge into the original point cloud, thereby ensuring the number of points in the fused point cloud.

[0092] Step S603: using the depth value of the first sampling point, convert the first sampling point into a laser radar coordinate system to obtain a representation point of the first sampling point.

[0093] In the embodiment, the three-dimensional coordinates of the characterization point of the first sampling point are: the three-dimensional coordinates obtained by converting the first sampling point into a laser radar coordinate system.

[0094] In some embodiments, the process of obtaining a representation point of a first sampling point includes: converting the first sampling point from a pixel coordinate system to a camera coordinate system, obtaining a camera coordinate point corresponding to the first sampling point in the camera coordinate system, the camera coordinate point including a depth value of the first sampling point; converting the camera coordinate point from the camera coordinate system to a lidar coordinate system, and obtaining a representation point of the first sampling point.

[0095] Step S604: fuse multiple representation points corresponding to each target with the original point cloud to obtain a fused point cloud.

[0096] In the embodiment, a plurality of representation points corresponding to each target are fused with the original point cloud. Specifically, the representation points are used as points in the point cloud, that is, the fused point cloud includes the points in the original point cloud and all the representation points.

[0097] In some other embodiments, the projection point closest to the first sampling point can be directly determined as the target projection point, and the depth value of the target projection point can be determined as the depth value of the sampling point. The subsequent coordinate system transformation of the first sampling point is as described above and will not be repeated. In this embodiment, for each sampling point, the depth value of the nearest projection point is used as the depth value, which is simple and has a small amount of calculation.

[0098] In the above process of determining the fused point cloud, the depth value of the first sampling point is determined according to the depth value of the target projection point corresponding to the first sampling point, wherein the target projection point is a part of all projection points that have a strong correlation with the first sampling point. The depth value of the first sampling point is determined by the depth value of the target projection point with a strong correlation, which can improve the accuracy of the depth value of the first sampling point, thereby ensuring the accuracy of the representation point and improving the quality of the fused point cloud.

[0099] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0100] Corresponding to a target detection method in the above embodiment, Figure 7 A structural block diagram of a target detection device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0101] Reference Figure 7 , the target detection device 700 includes: an acquisition unit 710, a projection unit 720, an image area determination unit 730, a sampling point determination unit 740, a fusion point cloud determination unit 750, and a target detection unit 760, wherein:

[0102] An acquisition unit 710 is used to acquire an original point cloud collected by the laser radar and an image collected by the camera, wherein the original point cloud and the image are synchronized in time and space;

[0103] A projection unit 720, configured to project points in the original point cloud onto the image to obtain a plurality of projection points, each of the plurality of projection points corresponding to a point in the original point cloud;

[0104] An image region determining unit 730 is used to determine an image region where each target in the image is located, and the image region of each target is a partial region of the image;

[0105] A sampling point determination unit 740 is used to perform pixel sampling in the image region where each target is located according to the density distribution information of the projection points in the image region where each target is located, so as to obtain a sampling point corresponding to each target;

[0106] A fused point cloud determining unit 750 is used to determine a fused point cloud according to the sampling points and the original point cloud corresponding to each target;

[0107] The target detection unit 760 is used to perform fusion target detection on the fused point cloud and the image to obtain a target detection result.

[0108] In one embodiment, the sampling point determination unit 740 is used to perform pixel sampling in the image area where each target is located according to the density distribution information of the projection points in the image area where each target is located to obtain the sampling points corresponding to each target, including:

[0109] The image region where each target is located is divided into multiple sub-regions, in which: each sub-region is a partial region of the image region, and any two sub-regions do not completely overlap;

[0110] Determine projection point density information of each sub-region in the plurality of sub-regions according to the projection points in the image region where each target is located;

[0111] Determine the sub-region corresponding to the projection point density information that meets the first preset condition as the target sub-region;

[0112] Pixel sampling is performed in the target sub-region to obtain sampling points corresponding to each target.

[0113] In one embodiment, the image region where each target is located is obtained by expanding the two-dimensional detection frame of each target, and the size of each sub-region of each target is the same as the size of the two-dimensional detection frame of each target.

[0114] In one embodiment, the sampling point determination unit 740 is used to divide the image area where each target is located into multiple sub-areas, including:

[0115] A two-dimensional detection frame of each target is used as a sliding window, and a sliding window operation is performed in each image region where the target is located to obtain multiple window regions, each of which is a sub-region of the multiple sub-regions.

[0116] In one embodiment, the sampling point determination unit 740 is used to determine the image area where each object in the image is located, including:

[0117] Perform object detection on the image to obtain a two-dimensional detection frame for each object in the image;

[0118] The two-dimensional detection box of each target is expanded to obtain the image area where each target is located.

[0119] In one embodiment, the sampling point determination unit 740 is used to expand the two-dimensional detection frame of each target to obtain the image area where each target is located, including:

[0120] According to the size of each target and the size of the time synchronization error, the expansion parameter corresponding to each target is obtained, and the time synchronization error indicates the degree of time asynchrony between the camera and the lidar;

[0121] According to the expansion parameters corresponding to each target, the two-dimensional detection frame of each target is expanded to obtain the image area of ​​each target.

[0122] In one embodiment, each target corresponds to a plurality of sampling points, and the fused point cloud determining unit 750 is used to determine the fused point cloud according to the sampling points corresponding to each target and the original point cloud, including:

[0123] Determine a target projection point corresponding to a first sampling point from among the multiple projection points, where the first sampling point is any one of the multiple sampling points corresponding to each target, and the target projection point is a portion of the multiple projection points;

[0124] Determine the depth value of the first sampling point according to the depth value of the target projection point;

[0125] Using the depth value of the first sampling point, convert the first sampling point to a laser radar coordinate system of the laser radar to obtain a representation point of the first sampling point;

[0126] Multiple representation points corresponding to each target are fused with the original point cloud to obtain a fused point cloud.

[0127] In one embodiment, the target projection point includes: a projection point located within a preset range centered at the first sampling point among the multiple projection points.

[0128] In one embodiment, the fused point cloud determining unit 750 is used to determine a target projection point corresponding to the first sampling point from a plurality of projection points, including:

[0129] If there is no projection point within the preset range centered at the first sampling point, a projection point among the multiple projection points that is closest to the first sampling point is determined as a target projection point corresponding to the first sampling point.

[0130] In one embodiment, the fused point cloud determining unit 750 is used to determine the depth value of the first sampling point according to the depth value of the target projection point, including:

[0131] If the first sampling point corresponds to a target projection point, the depth value of the target projection point is determined as the depth value of the first sampling point; if the first sampling point corresponds to multiple target projection points, the average of the depth values ​​of the multiple target projection points is determined as the depth value of the first sampling point.

[0132] Figure 8 A schematic diagram of the structure of a computer device 8 provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the computer device 8 of this embodiment includes: at least one processor 800 ( Figure 8 Only one processor is shown in the figure), a memory 801, and a computer program 802 stored in the memory 801 and executable on the at least one processor 800, wherein the processor 800 implements the steps in any of the above-mentioned target detection method embodiments when executing the computer program 802.

[0133] The computer device may include, but is not limited to, a processor 800 and a memory 801. Those skilled in the art will appreciate that Figure 8 It is only an example of the computer device 8 and does not constitute a limitation on the computer device 8. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0134] The processor 800 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0135] The memory 801 may be an internal storage unit in some embodiments, such as a hard disk or a memory. The memory 801 may also be an external storage device in other embodiments, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 801 may also include both an internal storage unit and an external storage device. The memory 801 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 801 may also be used to temporarily store data that has been output or is to be output.

[0136] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0137] In one embodiment, the present application also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0138] Obtain the original point cloud collected by the lidar and the image collected by the camera. The original point cloud and the image are synchronized in time and space;

[0139] Projecting points in the original point cloud onto the image to obtain multiple projection points, each of the multiple projection points corresponds to a point in the original point cloud;

[0140] Determine the image region where each target in the image is located, and the image region of each target is a partial region of the image;

[0141] According to the density distribution information of the projection points in the image area where each target is located, pixel sampling is performed in the image area where each target is located to obtain the sampling points corresponding to each target;

[0142] Determine the fused point cloud according to the sampling points and original point cloud corresponding to each target;

[0143] Perform fusion target detection on the fused point cloud and image to obtain the target detection result.

[0144] In one embodiment, the processor implements the following steps when executing the computer program:

[0145] The image region where each target is located is divided into multiple sub-regions, in which: each sub-region is a partial region of the image region, and any two sub-regions do not completely overlap;

[0146] Determine projection point density information of each sub-region in the plurality of sub-regions according to the projection points in the image region where each target is located;

[0147] Determine the sub-region corresponding to the projection point density information that meets the first preset condition as the target sub-region;

[0148] Pixel sampling is performed in the target sub-region to obtain sampling points corresponding to each target.

[0149] In one embodiment, the image region where each target is located is obtained by expanding the two-dimensional detection frame of each target, and the size of each sub-region of each target is the same as the size of the two-dimensional detection frame of each target.

[0150] In one embodiment, the processor implements the following steps when executing a computer program: using the two-dimensional detection frame of each target as a sliding window, performing a sliding window operation in the image area where each target is located to obtain multiple window areas, each of the multiple window areas being a sub-area in the multiple sub-areas.

[0151] In one embodiment, the processor implements the following steps when executing the computer program:

[0152] Perform object detection on the image to obtain a two-dimensional detection frame for each object in the image;

[0153] The two-dimensional detection box of each target is expanded to obtain the image area where each target is located.

[0154] In one embodiment, the processor implements the following steps when executing the computer program:

[0155] According to the size of each target and the size of the time synchronization error, the expansion parameter corresponding to each target is obtained, and the time synchronization error indicates the degree of time asynchrony between the camera and the lidar;

[0156] According to the expansion parameters corresponding to each target, the two-dimensional detection frame of each target is expanded to obtain the image area of ​​each target.

[0157] In one embodiment, each target corresponds to a plurality of sampling points, and the processor implements the following steps when executing the computer program:

[0158] Determine a target projection point corresponding to a first sampling point from among the multiple projection points, where the first sampling point is any one of the multiple sampling points corresponding to each target, and the target projection point is a portion of the multiple projection points;

[0159] Determine the depth value of the first sampling point according to the depth value of the target projection point;

[0160] Using the depth value of the first sampling point, convert the first sampling point to a laser radar coordinate system of the laser radar to obtain a representation point of the first sampling point;

[0161] Multiple representation points corresponding to each target are fused with the original point cloud to obtain a fused point cloud.

[0162] In one embodiment, the target projection point includes: a projection point located within a preset range centered at the first sampling point among the multiple projection points.

[0163] In one embodiment, the processor implements the following steps when executing the computer program: if there is no projection point within a preset range centered on the first sampling point, a projection point among multiple projection points that is closest to the first sampling point is determined as a target projection point corresponding to the first sampling point.

[0164] In one embodiment, the processor implements the following steps when executing the computer program: if the first sampling point corresponds to a target projection point, the depth value of the target projection point is determined as the depth value of the first sampling point; if the first sampling point corresponds to multiple target projection points, the average value of the depth values ​​of the multiple target projection points is determined as the depth value of the first sampling point.

[0165] The implementation principles and technical effects of the steps implemented when the processor executes the computer program in this embodiment are similar to the principles of the above-mentioned target detection method and will not be repeated here.

[0166] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0167] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0168] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0169] Obtain the original point cloud collected by the lidar and the image collected by the camera. The original point cloud and the image are synchronized in time and space;

[0170] Projecting points in the original point cloud onto the image to obtain multiple projection points, each of the multiple projection points corresponds to a point in the original point cloud;

[0171] Determine the image region where each target in the image is located, and the image region of each target is a partial region of the image;

[0172] According to the density distribution information of the projection points in the image area where each target is located, pixel sampling is performed in the image area where each target is located to obtain the sampling points corresponding to each target;

[0173] Determine the fused point cloud according to the sampling points and original point cloud corresponding to each target;

[0174] Perform fusion target detection on the fused point cloud and image to obtain the target detection result.

[0175] In one embodiment, the computer program, when executed by a processor, implements the following steps:

[0176] The image region where each target is located is divided into multiple sub-regions, in which: each sub-region is a partial region of the image region, and any two sub-regions do not completely overlap;

[0177] Determine projection point density information of each sub-region in the plurality of sub-regions according to the projection points in the image region where each target is located;

[0178] Determine the sub-region corresponding to the projection point density information that meets the first preset condition as the target sub-region;

[0179] Pixel sampling is performed in the target sub-region to obtain sampling points corresponding to each target.

[0180] In one embodiment, the image region where each target is located is obtained by expanding the two-dimensional detection frame of each target, and the size of each sub-region of each target is the same as the size of the two-dimensional detection frame of each target.

[0181] In one embodiment, when the computer program is executed by a processor, the following steps are implemented: using the two-dimensional detection frame of each target as a sliding window, performing a sliding window operation in the image area where each target is located, and obtaining multiple window areas, each of the multiple window areas is a sub-area in the multiple sub-areas.

[0182] In one embodiment, the computer program, when executed by a processor, implements the following steps:

[0183] Perform object detection on the image to obtain a two-dimensional detection frame for each object in the image;

[0184] The two-dimensional detection box of each target is expanded to obtain the image area where each target is located.

[0185] In one embodiment, the computer program, when executed by a processor, implements the following steps:

[0186] According to the size of each target and the size of the time synchronization error, the expansion parameter corresponding to each target is obtained, and the time synchronization error indicates the degree of time asynchrony between the camera and the lidar;

[0187] According to the expansion parameters corresponding to each target, the two-dimensional detection frame of each target is expanded to obtain the image area of ​​each target.

[0188] In one embodiment, the computer program, when executed by a processor, implements the following steps:

[0189] Determine a target projection point corresponding to a first sampling point from among the multiple projection points, where the first sampling point is any one of the multiple sampling points corresponding to each target, and the target projection point is a portion of the multiple projection points;

[0190] Determine the depth value of the first sampling point according to the depth value of the target projection point;

[0191] Using the depth value of the first sampling point, convert the first sampling point to a laser radar coordinate system of the laser radar to obtain a representation point of the first sampling point;

[0192] Multiple representation points corresponding to each target are fused with the original point cloud to obtain a fused point cloud.

[0193] In one embodiment, the target projection point includes: a projection point located within a preset range centered at the first sampling point among the multiple projection points.

[0194] In one embodiment, when the computer program is executed by a processor, the following steps are implemented: if there is no projection point within a preset range centered on the first sampling point, a projection point among multiple projection points that is closest to the first sampling point is determined as a target projection point corresponding to the first sampling point.

[0195] In one embodiment, when the computer program is executed by a processor, the following steps are implemented: if the first sampling point corresponds to a target projection point, the depth value of the target projection point is determined as the depth value of the first sampling point; if the first sampling point corresponds to multiple target projection points, the average value of the depth values ​​of the multiple target projection points is determined as the depth value of the first sampling point.

[0196] The implementation principles and technical effects of the steps implemented when the computer program in this embodiment is executed by the processor are similar to the principles of the above-mentioned target detection method and will not be repeated here.

[0197] An embodiment of the present application provides a computer program product. When the computer program product runs on a laser radar, the laser radar can implement the steps in the above-mentioned method embodiments when executed.

[0198] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0199] Obtain the original point cloud collected by the lidar and the image collected by the camera. The original point cloud and the image are synchronized in time and space;

[0200] Projecting points in the original point cloud onto the image to obtain multiple projection points, each of the multiple projection points corresponds to a point in the original point cloud;

[0201] Determine the image region where each target in the image is located, and the image region of each target is a partial region of the image;

[0202] According to the density distribution information of the projection points in the image area where each target is located, pixel sampling is performed in the image area where each target is located to obtain the sampling points corresponding to each target;

[0203] Determine the fused point cloud according to the sampling points and original point cloud corresponding to each target;

[0204] Perform fusion target detection on the fused point cloud and image to obtain the target detection result.

[0205] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0206] The image region where each target is located is divided into multiple sub-regions, in which: each sub-region is a partial region of the image region, and any two sub-regions do not completely overlap;

[0207] Determine projection point density information of each sub-region in the plurality of sub-regions according to the projection points in the image region where each target is located;

[0208] Determine the sub-region corresponding to the projection point density information that meets the first preset condition as the target sub-region;

[0209] Pixel sampling is performed in the target sub-region to obtain sampling points corresponding to each target.

[0210] In one embodiment, the image region where each target is located is obtained by expanding the two-dimensional detection frame of each target, and the size of each sub-region of each target is the same as the size of the two-dimensional detection frame of each target.

[0211] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: using the two-dimensional detection frame of each target as a sliding window, performing a sliding window operation in the image area where each target is located, and obtaining multiple window areas, each of the multiple window areas is a sub-area in the multiple sub-areas.

[0212] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0213] Perform object detection on the image to obtain a two-dimensional detection frame for each object in the image;

[0214] The two-dimensional detection box of each target is expanded to obtain the image area where each target is located.

[0215] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0216] According to the size of each target and the size of the time synchronization error, the expansion parameter corresponding to each target is obtained, and the time synchronization error indicates the degree of time asynchrony between the camera and the lidar;

[0217] According to the expansion parameters corresponding to each target, the two-dimensional detection frame of each target is expanded to obtain the image area of ​​each target.

[0218] In one embodiment, each target corresponds to a plurality of sampling points, and when the computer program is executed by the processor, the following steps are also implemented:

[0219] Determine a target projection point corresponding to a first sampling point from among the multiple projection points, where the first sampling point is any one of the multiple sampling points corresponding to each target, and the target projection point is a portion of the multiple projection points;

[0220] Determine the depth value of the first sampling point according to the depth value of the target projection point;

[0221] Using the depth value of the first sampling point, convert the first sampling point to a laser radar coordinate system of the laser radar to obtain a representation point of the first sampling point;

[0222] Multiple representation points corresponding to each target are fused with the original point cloud to obtain a fused point cloud.

[0223] In one embodiment, the target projection point includes: a projection point located within a preset range centered at the first sampling point among the multiple projection points.

[0224] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: if there is no projection point within a preset range centered on the first sampling point, a projection point among multiple projection points that is closest to the first sampling point is determined as a target projection point corresponding to the first sampling point.

[0225] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: if the first sampling point corresponds to a target projection point, the depth value of the target projection point is determined as the depth value of the first sampling point; if the first sampling point corresponds to multiple target projection points, the average value of the depth values ​​of the multiple target projection points is determined as the depth value of the first sampling point.

[0226] An embodiment of the present application further provides a chip, including: a processor, configured to call and run a computer program from a memory, so that a computer device equipped with the chip executes the steps in the above-mentioned method embodiments.

[0227] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0228] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0229] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are merely schematic, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0230] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0231] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0232] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0233] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A target detection method, characterized in that: The method comprises: Acquire an original point cloud collected by a laser radar and an image collected by a camera, wherein the original point cloud and the image are synchronized in time and space; Projecting points in the original point cloud onto the image to obtain a plurality of projection points, each of the plurality of projection points corresponding to a point in the original point cloud; Determine an image region where each target in the image is located, wherein the image region of each target is a partial region of the image; According to the density distribution information of the projection points in the image area where each target is located, pixel sampling is performed in the image area where each target is located to obtain the sampling points corresponding to each target; Determine a fused point cloud according to the sampling points corresponding to each target and the original point cloud; Perform fusion target detection on the fused point cloud and the image to obtain a target detection result.

2. The method according to claim 1, characterized in that The method of sampling pixels in the image region where each target is located according to the density distribution information of the projection points in the image region where each target is located to obtain sampling points corresponding to each target includes: Divide the image region where each target is located into a plurality of sub-regions, wherein: each sub-region is a partial region of the image region, and any two sub-regions do not completely overlap; Determine projection point density information of each sub-region in the plurality of sub-regions according to the projection points in the image region where each target is located; Determine the sub-region corresponding to the projection point density information that meets the first preset condition as the target sub-region; Pixel sampling is performed in the target sub-area to obtain sampling points corresponding to each target.

3. The method according to claim 2, characterized in that The image region where each target is located is obtained by expanding the two-dimensional detection frame of each target, and the size of each sub-region of each target is the same as the size of the two-dimensional detection frame of each target.

4. The method according to claim 3, characterized in that: The image area where each target is located is divided into multiple sub-areas, including: Using the two-dimensional detection frame of each target as a sliding window, a sliding window operation is performed in the image region where each target is located to obtain multiple window regions, each of which is a sub-region of the multiple sub-regions.

5. The method according to claim 1, characterized in that The step of determining the image region where each target in the image is located comprises: Performing object detection on the image to obtain a two-dimensional detection frame of each object in the image; The two-dimensional detection box of each target is expanded to obtain the image area where each target is located.

6. The method according to claim 5, characterized in that The step of expanding the two-dimensional detection frame of each target to obtain the image region where each target is located includes: Obtaining an extended parameter corresponding to each target according to the size of each target and the size of a time synchronization error, wherein the time synchronization error indicates the degree of time asynchrony between the camera and the laser radar; The two-dimensional detection frame of each target is expanded according to the expansion parameters corresponding to each target to obtain the image region of each target.

7. The method according to any one of claims 1 to 6, characterized in that: Each target corresponds to a plurality of sampling points, and determining the fused point cloud according to the sampling points corresponding to each target and the original point cloud includes: Determine a target projection point corresponding to a first sampling point from the multiple projection points, wherein the first sampling point is any one of the multiple sampling points corresponding to each target, and the target projection point is a portion of the multiple projection points; Determine the depth value of the first sampling point according to the depth value of the target projection point; Using the depth value of the first sampling point, convert the first sampling point to a laser radar coordinate system of the laser radar to obtain a representation point of the first sampling point; A plurality of representation points corresponding to each target are fused with the original point cloud to obtain the fused point cloud.

8. The method according to claim 7, characterized in that The target projection point includes: a projection point among the multiple projection points that is located within a preset range centered on the first sampling point.

9. The method according to claim 7, characterized in that: The step of determining a target projection point corresponding to the first sampling point from the plurality of projection points comprises: If there is no projection point within a preset range centered at the first sampling point, a projection point among the multiple projection points that is closest to the first sampling point is determined as a target projection point corresponding to the first sampling point.

10. The method according to any one of claims 7 to 9, characterized in that: The step of determining the depth value of the first sampling point according to the depth value of the target projection point includes: If the first sampling point corresponds to a target projection point, the depth value of the target projection point is determined as the depth value of the first sampling point; if the first sampling point corresponds to multiple target projection points, the average value of the depth values ​​of the multiple target projection points is determined as the depth value of the first sampling point.

11. A target detection device, characterized in that: The device comprises means for performing the steps of the method according to any one of claims 1 to 10.

12. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 10 is implemented.

13. A target detection system, characterized in that: include: A laser radar, a camera and a computer device as claimed in claim 12.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

15. A chip, characterized in that: include: A processor, configured to call and run a computer program from a memory, so that a computer device equipped with the chip executes a method as claimed in any one of claims 1 to 10.

Images