A category determination method, apparatus, device, and storage medium

Abstract

A category determination method, device, equipment and storage medium are disclosed. The method comprises: obtaining a projection matrix between at least one camera and at least one laser radar; controlling each camera to obtain image data of a target object; controlling each laser radar to obtain point cloud data corresponding to the target object; and determining target category information of the target object according to the projection matrix, the image data of the target object and the point cloud data corresponding to the target object. The embodiment of the present application obtains the projection matrix between at least one camera and at least one laser radar, controls each camera to obtain image data of a target object, controls each laser radar to obtain point cloud data corresponding to the target object, and determines target category information of the target object according to the projection matrix, the image data of the target object and the point cloud data corresponding to the target object, thereby solving the problem that the detection results of the camera and the detection results of the laser radar cannot be effectively fused in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-sensor data fusion, and in particular to a category determination method, device, equipment and storage medium. BACKGROUND

[0002] Multi-sensor data fusion is an emerging field, which is a research on data processing for the specific problem of using multiple sensors in a system. In the past decade, with the rapid development of unmanned vehicles and mobile robots, a precise and efficient perception system can ensure the safety of robots and other surrounding moving objects, and multi-sensor fusion plays an important role in the perception system. In recent years, laser radar is widely used in unmanned vehicle perception systems, and two-dimensional object detection based on cameras has developed rapidly in unmanned vehicle systems, but more improvements are needed in the mutual fusion of objects detected by cameras and objects detected by laser radars.

[0003] This multi-sensor fusion task usually involves sensors such as laser radars, millimeter wave radars and cameras. Laser radar sensors are often used in object segmentation and object detection algorithms based on 3D world coordinate systems. The images produced by cameras usually have more semantic information, so they are often used in object segmentation and detection algorithms based on image planes. However, due to the detection based on different reference coordinate systems by the two sensors, and the lower accuracy of laser radar detection than camera accuracy, there are often missed detections, but the distance accuracy of laser radar is better than the distance accuracy provided by cameras. For camera detection algorithms, the accuracy of target object detection is higher than the detection accuracy of laser radars, but the distance accuracy is much lower than the distance accuracy provided by laser radars. Therefore, if the detection results of the two sensors can be effectively fused, the safety of the autonomous driving system can be ensured.

[0004] There are still many practical application problems in existing multi-sensor fusion algorithms. For example: 1) a unified fusion theory and effective generalized fusion model have not been established; 2) the research on specific data fusion methods is still in its infancy; 3) the existing algorithms have not well solved the fault tolerance and robustness problems in the fusion system; 4) the associated ambiguity is the main obstacle in data fusion. SUMMARY

[0005] The present application provides a category determination method, device, equipment and storage medium to solve the problem that the detection results of cameras and laser radars cannot be effectively fused.

[0006] According to an aspect of the present application, a category determination method is provided, which comprises:

[0007] acquire a projection matrix between at least one camera and at least one lidar;

[0008] control each of the cameras to acquire image data of a target object;

[0009] control each of the lidars to acquire point cloud data corresponding to the target object;

[0010] determine target category information of the target object according to the projection matrix, the image data of the target object, and the point cloud data corresponding to the target object.

[0011] According to another aspect of the present application, there is provided a category determination apparatus, comprising:

[0012] a first acquisition module configured to acquire a projection matrix between at least one camera and at least one lidar;

[0013] a first control module configured to control each of the cameras to acquire image data of a target object;

[0014] a second control module configured to control each of the lidars to acquire point cloud data corresponding to the target object;

[0015] a determination module configured to determine target category information of the target object according to the projection matrix, the image data of the target object, and the point cloud data corresponding to the target object.

[0016] According to another aspect of the present application, there is provided an electronic device, comprising:

[0017] at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the category determination method according to any one of the embodiments of the present application.

[0018] According to another aspect of the present application, there is provided a computer readable storage medium storing computer instructions for causing a processor to perform the category determination method according to any one of the embodiments of the present application when executed.

[0019] The technical scheme of the embodiment of the present application obtains a projection matrix between at least one camera and at least one laser radar, controls each camera to obtain image data of a target object, controls each laser radar to obtain point cloud data corresponding to the target object, determines target category information of the target object according to the projection matrix, the image data of the target object and the point cloud data corresponding to the target object, and solves the problem that the detection result of the camera and the detection result of the laser radar cannot be effectively fused in the prior art.

[0020] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1 is a flow chart of a category determination method according to an embodiment of the present application;

[0023] Figure 2 is a structural schematic diagram of a category determination device according to an embodiment of the present application;

[0024] Figure 3 is a structural schematic diagram of an electronic device for implementing the category determination method of the present application. DETAILED DESCRIPTION

[0025] In order to make those skilled in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0026] It is to be understood that the terminology "first", "target", and the like used in the specification and the claims of the application and the above-described drawings is intended to distinguish similar objects, and is not necessarily intended to describe a particular sequential or chronological order. It should be understood that the data thus used can be interchanged, where appropriate, so that the embodiments of the application described herein can be carried out in other than the order shown or described herein. Furthermore, the terms "comprising" and "having", and any variations thereof, are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or apparatus that includes a list of steps or units need not be limited to those steps or units that are clearly listed, but can include other steps or units that are not clearly listed or inherent to such processes, methods, products, or apparatuses.

[0027] Embodiment one

[0028] Figure 1 is a flowchart of a category determination method according to an embodiment one of the application, the embodiment can be applicable to a category determination case, the method can be executed by a category determination device, the category determination device can be realized in the form of hardware and / or software, and the category determination device can be integrated in any electronic device providing a category determination function. As shown in the figure, the method comprises: Figure 1

[0029] S101, obtaining a projection matrix between at least one camera and at least one laser radar.

[0030] In the embodiment, the camera can be at least one, which is used to obtain the image of the person or object in the environment around the vehicle body, and the type of the camera is not limited in the embodiment. The camera can be installed on the vehicle body, and the installation position of the camera is not limited in the embodiment, for example, one can be installed on each of the front, rear, left and right of the vehicle body, and one can be installed on the roof of the vehicle body.

[0031] In the embodiment, the laser radar can be at least one, which is used to obtain the point cloud data of the person or object in the environment around the vehicle body, and the type of the laser radar is not limited in the embodiment. The laser radar can be installed on the vehicle body, and the installation position of the laser radar is not limited in the embodiment, for example, three can be installed on the roof of the vehicle body.

[0032] It should be explained that the projection matrix can be understood as the conversion matrix of the positional relationship between each camera and each laser radar. For example, the projection matrix can be a 4x4 matrix. In the embodiment, the projection matrix can include a rotation matrix and a translation matrix.

[0033] ​Specifically, the extrinsic calibration is performed on the at least one camera to calculate the position coordinates of each camera relative to the center of the rear wheel of the vehicle, the extrinsic calibration is performed on the at least one lidar to calculate the position coordinates of each lidar relative to the center of the rear wheel of the vehicle, and the projection matrix between each camera and each lidar is determined according to the position coordinates of each camera relative to the center of the rear wheel of the vehicle and the position coordinates of each lidar relative to the center of the rear wheel of the vehicle.

[0034] S102, control each camera to acquire image data of a target object.

[0035] For example, the target object can be a person or an object in the environment around the body of the current vehicle, and the current vehicle can be a vehicle driven by a user and installed with at least one camera and at least one lidar.

[0036] It should be noted that each camera corresponds to an image coordinate system, and the image data can be the data information of the image of the target object captured by the camera in the image coordinate system corresponding to the camera.

[0037] Specifically, each camera is controlled to acquire images of the target object around the body in real time, and the image data of the image of the target object captured by each camera in the image coordinate system corresponding to the camera is acquired.

[0038] S103, control each lidar to acquire point cloud data corresponding to the target object.

[0039] It can be understood that the point cloud data refers to a set of vectors in a three-dimensional coordinate system. In this embodiment, the point cloud data can be the point cloud data corresponding to the target object obtained by scanning the target object around the body by the lidar.

[0040] Specifically, each lidar is controlled to acquire point cloud data corresponding to the target object around the body in real time.

[0041] S104, determine target category information of the target object according to the projection matrix, the image data of the target object, and the point cloud data corresponding to the target object.

[0042] It should be noted that the target category information can be information of which category the target object specifically belongs to. In this embodiment, the target category information can include six categories: pedestrians, trucks, cars, vans, cyclists (motorcycle and bicycle), and buses.

[0043] Specifically, the target category information of which category the target object specifically belongs to is determined according to the projection matrix, the image data of the target object, and the point cloud data corresponding to the target object.

[0044] The technical scheme of the embodiment of the present application obtains a projection matrix between at least one camera and at least one laser radar, controls each camera to obtain image data of a target object, controls each laser radar to obtain corresponding point cloud data of the target object, determines target category information of the target object according to the projection matrix, the image data of the target object and the corresponding point cloud data of the target object, and solves the problem that the detection result of the camera and the detection result of the laser radar cannot be effectively fused in the prior art.

[0045] Optionally, the corresponding point cloud data of the target object comprises first point cloud data and second point cloud data.

[0046] In the embodiment, the first point cloud data can be corresponding point cloud data of an irregular segmented object of which the target object around the vehicle body is a polygon, obtained by the laser radar. Specifically, the first point cloud data based on a world coordinate system is generated according to the point cloud data of the target object around the vehicle body obtained by the laser radar, based on a laser radar segmentation algorithm.

[0047] In the embodiment, the second point cloud data can be corresponding point cloud data of a detection object of which the target object around the vehicle body is a 3D bounding box, i.e., the target object is a regular object, obtained by the laser radar. Specifically, the second point cloud data based on a world coordinate system is generated according to the point cloud data of the target object around the vehicle body obtained by the laser radar, based on a laser radar target detection algorithm.

[0048] Correspondingly, the target category information of the target object is determined according to the projection matrix, the image data of the target object and the corresponding point cloud data of the target object, comprising:

[0049] The first category information of the target object and a first confidence degree corresponding to the first category information are determined according to the image data of the target object.

[0050] It should be noted that the first category information can be information of which category the target object specifically belongs to, determined according to the image data of the target object obtained by the camera. The first confidence degree can be an accuracy confidence degree of the information of which category the target object specifically belongs to, determined according to the image data of the target object obtained by the camera.

[0051] Specifically, the first category information of the target object and the first confidence degree corresponding to the first category information can be determined according to the image data of the target object, based on a target detection algorithm.

[0052] The first contour information of the target object is determined according to the first point cloud data, the projection matrix and the image data of the target object.

[0053] It needs to be explained that the first contour information can be the contour information of the target object when the first point cloud data corresponding to the irregular segmented object of the target object around the vehicle body is projected to the plane where the image data of the target object is located, and the contour information of the target object after data association between the image data of the target object obtained by the camera.

[0054] Specifically, the irregular segmented object of the polygon is projected to the image coordinate system corresponding to each camera, that is, the first point cloud data corresponding to each irregular segmented object of the polygon is multiplied by the projection matrix corresponding to each camera, and the result is associated with the image data of the target object obtained by the camera to obtain the first contour information of the target object.

[0055] In actual operation, after obtaining the first contour information of the target object, the contour of the irregular segmented object of the polygon is fitted based on the L-Fitting algorithm, that is, the 3D bounding box of the irregular segmented object of the polygon is fitted.

[0056] According to the first contour information of the target object and the first category information of the target object, the first result category information of the target object is determined.

[0057] It needs to be explained that the first result category information can be the category information of the target object when the target object around the vehicle body is the irregular segmented object of the polygon.

[0058] Specifically, when the target object is the irregular segmented object of the polygon, the first category information of the target object is assigned to the first contour information of the target object, that is, when the target object around the vehicle body is the irregular segmented object of the polygon, the first category information of the target object is taken as the first result category information of the target object.

[0059] According to the second point cloud data, the second category information of the target object and the second confidence corresponding to the second category information are determined.

[0060] It needs to be explained that the second category information can be the information of which category the target object determined by the second point cloud data corresponding to the detection object of the 3D bounding box of the target object around the vehicle body obtained by the laser radar belongs to when the target object is a regular object. The second confidence can be the accuracy confidence of the information of which category the target object determined by the second point cloud data corresponding to the detection object of the 3D bounding box of the target object around the vehicle body obtained by the laser radar belongs to when the target object is a regular object.

[0061] Specifically, the second point cloud data based on the world coordinate system can be generated based on the laser radar target detection algorithm according to the point cloud data of the target object around the vehicle body acquired by the laser radar, and the second category information of the target object and the second confidence corresponding to the second category information can be determined according to the second point cloud data.

[0062] The second contour information of the target object is determined according to the second point cloud data, the projection matrix and the image data of the target object.

[0063] It should be explained that the second contour information can be a detection object of the target object around the vehicle body as a 3D bounding box, that is, the contour information of the target object when the second point cloud data corresponding to the target object as a regular object is projected to the plane where the image data of the target object is located, and the contour information of the target object after data association of the image data of the target object acquired by the camera.

[0064] Specifically, the detection object of the 3D bounding box is projected to the image coordinate system corresponding to each camera, that is, the second point cloud data corresponding to each detection object of the 3D bounding box is multiplied by the projection matrix corresponding to each camera, and the result is associated with the image data of the target object acquired by the camera to finally obtain the second contour information of the target object.

[0065] The second result category information of the target object is determined according to the second contour information of the target object, the first category information of the target object, the first confidence corresponding to the first category information, the second category information of the target object and the second confidence corresponding to the second category information.

[0066] It should be noted that the second result category information can be a detection object of the target object around the vehicle body as a 3D bounding box, that is, the category information of the target object when the target object is a regular object.

[0067] Specifically, when the first category information of the target object determined according to the image data of the target object is the same as the second category information of the target object determined according to the second point cloud data, the target object is a detection object of a 3D bounding box, that is, the second result category information of the target object when the target object is a regular object can be the first category information of the target object or the second category information of the target object; when the first category information of the target object determined according to the image data of the target object is different from the second category information of the target object determined according to the second point cloud data, the size of the first confidence corresponding to the first category information of the target object and the second confidence corresponding to the second category information of the target object is judged, and the category information corresponding to the maximum confidence is taken as the second result category information of the target object.

[0068] The target category information of the target object is determined according to the first result category information of the target object and the second result category information of the target object.

[0069] Specifically, the first result category information when the target object is an irregular segmentation object and the second result category information when the target object is a 3D bounding box detection object, i.e., the target object is a regular object, are fitted by 3D IOU (Intersection over Union), and the target category information of the target object is determined, i.e., the final category information result of the target object is determined, thereby solving the problem of the target category information normalization of the repeated target object.

[0070] Optionally, the first contour information of the target object is determined according to the first point cloud data, the projection matrix, and the image data of the target object, and the method comprises the following steps.

[0071] The first contour data of the target object is determined according to the first point cloud data and the projection matrix.

[0072] It should be noted that the first contour data can be the contour information data of the target object when the target object around the vehicle body is an irregular segmentation object of a polygon, and the first point cloud data corresponding to the target object is projected to the plane where the image data of the target object is located.

[0073] Specifically, the irregular segmentation object of a polygon is projected to the image coordinate system corresponding to each camera, i.e., the first point cloud data corresponding to each irregular segmentation object of a polygon is multiplied by the projection matrix corresponding to each camera, and the result obtained is the first contour data of the target object.

[0074] In actual operation, after obtaining the first contour data of the target object, the value of the data of the contour boundary in the obtained first contour data that exceeds the length and width of the image collected by the camera is modified to the value of the length and width of the image collected by the camera, i.e., the boundary value of the first contour data is limited. For example, the resolution of the image collected by the camera can be 1920 (width) x 1208 (height), and the width of the resolution of the irregular segmentation object of a polygon in the obtained first contour data can be 1980, which exceeds the width 1920 of the resolution of the image collected by the camera, so the 1980 is forcibly limited to 1920.

[0075] The first contour data of the target object and the image data of the target object are associated to determine the first contour information of the target object.

[0076] In this embodiment, the data association can be to establish a corresponding relationship between the first contour data of the target object and the image data of the target object.

[0077] Specifically, the first contour data of the target object and the image data of the target object are associated based on the Hungarian algorithm when the target object is an irregular segmented object, and the first contour information of the target object is determined.

[0078] Optionally, the second contour information of the target object is determined according to the second point cloud data, the projection matrix and the image data of the target object, and the method comprises the following steps of:

[0079] The second contour data of the target object is determined according to the second point cloud data and the projection matrix.

[0080] It should be noted that the second contour data can be the contour information data of the target object when the target object is a 3D bounding box detection object, i.e., a regular object, and the corresponding second point cloud data of the target object is projected to the plane where the image data of the target object is located.

[0081] Specifically, the 3D bounding box detection object is projected to the image coordinate system corresponding to each camera, i.e., the second point cloud data corresponding to each 3D bounding box detection object is multiplied by the projection matrix corresponding to each camera, and the result is the second contour data of the target object.

[0082] In actual operation, after obtaining the second contour data of the target object, the data value of the contour boundary data in the obtained second contour data that exceeds the length and width of the image collected by the camera is modified to the data value of the length and width of the image collected by the camera, i.e., the boundary value of the second contour data is limited. For example, the resolution of the image collected by the camera can be 1920 (width) x 1208 (height), and the resolution of the 3D bounding box detection object in the obtained second contour data can be 1980 (width), which exceeds the width 1920 in the resolution of the image collected by the camera. Therefore, 1980 is forcibly limited to 1920.

[0083] The second contour data of the target object and the image data of the target object are associated to determine the second contour information of the target object.

[0084] In this embodiment, the data association can be to establish a corresponding relationship between the second contour data of the target object and the image data of the target object.

[0085] Specifically, the second contour data of the target object and the image data of the target object are associated based on the Hungarian algorithm when the target object is a 3D bounding box detection object, i.e., a regular object, and the second contour information of the target object is determined.

[0086] Optionally, the second result category information of the target object is determined according to the second contour information of the target object, the first category information of the target object, the first confidence corresponding to the first category information, the second category information of the target object, and the second confidence corresponding to the second category information, and the method comprises the following steps.

[0087] If the first category information of the target object and the second category information of the target object are different, the category information corresponding to the confidence with the maximum value between the first confidence corresponding to the first category information and the second confidence corresponding to the second category information is determined as the second result category information of the target object.

[0088] Specifically, if the first category information of the target object and the second category information of the target object are different, the first confidence corresponding to the first category information and the second confidence corresponding to the second category information are compared, and the category information corresponding to the confidence with the maximum value between the first confidence corresponding to the first category information and the second confidence corresponding to the second category information is determined as the second result category information of the target object.

[0089] Optionally, the projection matrix between the at least one camera and the at least one laser radar is obtained, and the method comprises the following steps.

[0090] The coordinate information of each camera relative to the target position of the vehicle is obtained.

[0091] In the embodiment, the target position of the vehicle can be any position of the vehicle, and preferably, the target position of the vehicle can be the center of the rear wheel of the vehicle.

[0092] Specifically, the coordinate information of all cameras mounted on the vehicle relative to the target position of the vehicle is obtained.

[0093] The coordinate information of each laser radar relative to the target position of the vehicle is obtained.

[0094] Specifically, the coordinate information of all laser radars mounted on the vehicle relative to the target position of the vehicle is obtained.

[0095] The projection matrix between each camera and each laser radar is determined according to the coordinate information of each camera relative to the target position of the vehicle and the coordinate information of each laser radar relative to the target position of the vehicle.

[0096] Specifically, the conversion matrix of the positional relationship between each camera and each laser radar, i.e., the projection matrix between each camera and each laser radar, is determined according to the coordinate information of each camera relative to the target position of the vehicle and the coordinate information of each laser radar relative to the target position of the vehicle.

[0097] Optionally, before the target category information of the target object is determined according to the projection matrix, the image data of the target object, and the point cloud data corresponding to the target object, the method further comprises the following steps.

[0098] acquire a first timestamp of the image data.

[0099] In the embodiment, the first timestamp can be understood as the time when each camera acquires the image data of the target object, for example, each camera can acquire image data once every 1 second, and each camera can acquire image data 10 times within 10 seconds from the current time.

[0100] Specifically, the first timestamp of the image data of the target object is acquired.

[0101] acquire a second timestamp of the point cloud data.

[0102] In the embodiment, the second timestamp can be understood as the time when each laser radar acquires the point cloud data corresponding to the target object, for example, each laser radar can acquire point cloud data once every 1 second, and each laser radar can acquire point cloud data 10 times within 10 seconds from the current time.

[0103] Specifically, the second timestamp of the point cloud data corresponding to the target object is acquired.

[0104] Time synchronization is performed according to the first timestamp and the second timestamp.

[0105] Specifically, before controlling each camera to acquire the image data of the target object and controlling each laser radar to acquire the point cloud data corresponding to the target object, time synchronization of GNSS (Global Navigation Satellite System) is performed on each camera and each laser radar, and then time synchronization is performed according to the first timestamp of the image data acquired by each camera and the second timestamp of the point cloud data acquired by each laser radar, that is, the image data and the point cloud data acquired at the same time are fused.

[0106] In actual operation, spatial synchronization also needs to be performed according to the coordinate information of each camera relative to the target position of the vehicle and the coordinate information of each laser radar relative to the target position of the vehicle.

[0107] The technical scheme of the embodiment of the application acquires the projection matrix between at least one camera and at least one laser radar, controls each camera to acquire the image data of the target object, controls each laser radar to acquire the point cloud data corresponding to the target object, determines the target category information of the target object according to the projection matrix, the image data of the target object and the point cloud data corresponding to the target object, solves the problem that the detection results of the camera and the detection results of the laser radar cannot be effectively fused in the prior art, and effectively fuses the detection results of the two kinds of sensors, that is, the camera and the laser radar, thereby providing safety guarantee for automatic driving.

[0108] Embodiment Two

[0109] Figure 2 is a structural schematic diagram of a category determination apparatus provided according to Embodiment Two of the present application. As shown in the figure, the apparatus comprises a first acquisition module 201, a first control module 202, a second control module 203 and a determination module 204. Figure 2

[0110] The first acquisition module 201 is configured to acquire a projection matrix between at least one camera and at least one laser radar.

[0111] The first control module 202 is configured to control each of the cameras to acquire image data of a target object.

[0112] The second control module 203 is configured to control each of the laser radars to acquire point cloud data corresponding to the target object.

[0113] The determination module 204 is configured to determine target category information of the target object according to the projection matrix, the image data of the target object and the point cloud data corresponding to the target object.

[0114] Optionally, the point cloud data corresponding to the target object comprises first point cloud data and second point cloud data.

[0115] Correspondingly, the determination module 204 comprises:

[0116] A first determination unit configured to determine first category information of the target object and a first confidence corresponding to the first category information according to the image data of the target object.

[0117] A second determination unit configured to determine first contour information of the target object according to the first point cloud data, the projection matrix and the image data of the target object.

[0118] A third determination unit configured to determine first result category information of the target object according to the first contour information of the target object and the first category information of the target object.

[0119] A fourth determination unit configured to determine second category information of the target object and a second confidence corresponding to the second category information according to the second point cloud data.

[0120] A fifth determination unit configured to determine second contour information of the target object according to the second point cloud data, the projection matrix and the image data of the target object.

[0121] ​The sixth determination unit is configured to determine second result category information of the target object according to the second contour information of the target object, the first category information of the target object, the first confidence corresponding to the first category information, the second category information of the target object, and the second confidence corresponding to the second category information.

[0122] The seventh determination unit is configured to determine target category information of the target object according to the first result category information of the target object and the second result category information of the target object.

[0123] Optionally, the second determination unit comprises:

[0124] The first determination sub-unit is configured to determine first contour data of the target object according to the first point cloud data and the projection matrix.

[0125] The second determination sub-unit is configured to determine first contour information of the target object by data association between the first contour data of the target object and the image data of the target object.

[0126] Optionally, the fifth determination unit comprises:

[0127] The third determination sub-unit is configured to determine second contour data of the target object according to the second point cloud data and the projection matrix.

[0128] The fourth determination sub-unit is configured to determine second contour information of the target object by data association between the second contour data of the target object and the image data of the target object.

[0129] Optionally, the sixth determination unit comprises:

[0130] The fifth determination sub-unit is configured to determine, if the first category information of the target object and the second category information of the target object are different, the category information corresponding to the confidence with the maximum value between the first confidence corresponding to the first category information and the second confidence corresponding to the second category information as the second result category information of the target object.

[0131] Optionally, the first acquisition module 201 comprises:

[0132] The first acquisition unit is configured to acquire coordinate information of each camera relative to a target position of the vehicle.

[0133] The second acquisition unit is configured to acquire coordinate information of each laser radar relative to the target position of the vehicle.

[0134] An eighth determining unit is configured to determine a projection matrix between each camera and each lidar according to coordinate information of each camera relative to a target position of the vehicle and coordinate information of each lidar relative to the target position of the vehicle.

[0135] Optionally, the category determining apparatus further comprises:

[0136] A second obtaining module is configured to obtain a first time stamp of the image data before determining the target category information of the target object according to the projection matrix, the image data of the target object, and the point cloud data corresponding to the target object.

[0137] A third obtaining module is configured to obtain a second time stamp of the point cloud data before determining the target category information of the target object according to the projection matrix, the image data of the target object, and the point cloud data corresponding to the target object.

[0138] A time synchronizing module is configured to perform time synchronization according to the first time stamp and the second time stamp before determining the target category information of the target object according to the projection matrix, the image data of the target object, and the point cloud data corresponding to the target object.

[0139] The category determining apparatus provided by the embodiments of the present application can perform the category determining method provided by any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of performing the method.

[0140] Embodiment three

[0141] Figure 3 A structural schematic diagram of an electronic device 30 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0142] As Figure 3As shown, the electronic device 30 includes at least one processor 31, and a memory, such as a read-only memory (ROM) 32, a random access memory (RAM) 33, etc., connected to the at least one processor 31 in communication. The memory stores computer programs executable by the at least one processor, and the processor 31 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 32 or loaded from the storage unit 38 into the random access memory (RAM) 33. In the RAM 33, various programs and data required for the operation of the electronic device 30 can also be stored. The processor 31, the ROM 32, and the RAM 33 are connected to each other through a bus 34. An input / output (I / O) interface 35 is also connected to the bus 34.

[0143] Various components in the electronic device 30 are connected to the I / O interface 35, including an input unit 36, such as a keyboard, a mouse, etc., an output unit 37, such as various types of displays, speakers, etc., a storage unit 38, such as a magnetic disk, an optical disk, etc., and a communication unit 39, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 39 allows the electronic device 30 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunications networks.

[0144] The processor 31 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 31 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 31 performs various methods and processes described above, such as the category determination method:

[0145] obtaining a projection matrix between at least one camera and at least one lidar;

[0146] controlling each of the cameras to obtain image data of a target object;

[0147] controlling each of the lidars to obtain point cloud data corresponding to the target object;

[0148] determining target category information of the target object according to the projection matrix, the image data of the target object, and the point cloud data corresponding to the target object.

[0149] In some embodiments, the category determination method can be implemented as a computer program tangibly embodied in a computer readable storage medium, e.g., storage unit 38. In some embodiments, portions of or all of the computer program can be loaded onto the electronic device 30 via, e.g., ROM 32 and / or communication unit 39. When a computer program is loaded onto RAM 33 and executed by processor 31, one or more of the steps of the above described category determination method can be performed. Alternatively, in other embodiments, processor 31 can be configured to perform the category determination method by other means, e.g., with the aid of firmware.

[0150] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0151] Computer programs used to implement the methods of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor of the machine, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0152] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0153] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0154] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0155] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0156] It should be understood that the various forms of flow shown above can be used to reorder, add or delete steps. For example, each step described in the present application can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, which is not limited herein.

[0157] The above detailed description does not constitute a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A category determining method characterized by comprising: The method comprises: acquiring a projection matrix between at least one camera and at least one lidar; controlling each of the cameras to acquire image data of a target object; controlling each of the lidars to acquire point cloud data corresponding to the target object; determining target category information of the target object according to the projection matrix, the image data of the target object, and the point cloud data corresponding to the target object; wherein the point cloud data corresponding to the target object comprises first point cloud data and second point cloud data; correspondingly, determining the target category information of the target object according to the projection matrix, the image data of the target object, and the point cloud data corresponding to the target object comprises: determining first category information of the target object and a first confidence corresponding to the first category information according to the image data of the target object; determining first contour information of the target object according to the first point cloud data, the projection matrix, and the image data of the target object; determining first result category information of the target object according to the first contour information of the target object and the first category information of the target object; determining second category information of the target object and a second confidence corresponding to the second category information according to the second point cloud data; determining second contour information of the target object according to the second point cloud data, the projection matrix, and the image data of the target object; determining second result category information of the target object according to the second contour information of the target object, the first category information of the target object, the first confidence corresponding to the first category information, the second category information of the target object, and the second confidence corresponding to the second category information; determining target category information of the target object according to the first result category information of the target object and the second result category information of the target object.

2. The method of claim 1, wherein, determining the first contour information of the target object according to the first point cloud data, the projection matrix, and the image data of the target object comprises: determining first contour data of the target object according to the first point cloud data and the projection matrix; determining the first contour information of the target object by data association between the first contour data of the target object and the image data of the target object.

3. The method of claim 1, wherein, determining the second contour information of the target object according to the second point cloud data, the projection matrix, and the image data of the target object comprises: determining second contour data of the target object according to the second point cloud data and the projection matrix; determining the second contour information of the target object by data association between the second contour data of the target object and the image data of the target object.

4. The method of claim 1, wherein, determining the second result category information of the target object according to the second contour information of the target object, the first category information of the target object, the first confidence corresponding to the first category information, the second category information of the target object, and the second confidence corresponding to the second category information comprises: If the first category information of the target object and the second category information of the target object are not the same, a category information corresponding to a confidence degree with a maximum value between a first confidence degree corresponding to the first category information and a second confidence degree corresponding to the second category information is determined as second result category information of the target object.

5. The method of claim 1, wherein, Obtaining a projection matrix between at least one camera and at least one laser radar includes: Obtaining coordinate information of each camera relative to a target position of a vehicle; Obtaining coordinate information of each laser radar relative to the target position of the vehicle; Determining the projection matrix between each camera and each laser radar according to the coordinate information of each camera relative to the target position of the vehicle and the coordinate information of each laser radar relative to the target position of the vehicle.

6. The method of claim 1, wherein, Before determining target category information of the target object according to the projection matrix, image data of the target object, and point cloud data corresponding to the target object, the method further includes: Obtaining a first timestamp of the image data; Obtaining a second timestamp of the point cloud data; Time synchronizing according to the first timestamp and the second timestamp.

7. A category determining apparatus characterized by comprising: The method includes: A first obtaining module is configured to obtain a projection matrix between at least one camera and at least one laser radar; A first control module is configured to control each camera to obtain image data of a target object; A second control module is configured to control each laser radar to obtain point cloud data corresponding to the target object; A determining module is configured to determine target category information of the target object according to the projection matrix, the image data of the target object, and the point cloud data corresponding to the target object; The point cloud data corresponding to the target object includes first point cloud data and second point cloud data. Correspondingly, the determining module includes: A first determining unit is configured to determine first category information of the target object and a first confidence degree corresponding to the first category information according to the image data of the target object; A second determining unit is configured to determine first contour information of the target object according to the first point cloud data, the projection matrix, and the image data of the target object; A third determining unit is configured to determine first result category information of the target object according to the first contour information of the target object and the first category information of the target object; A fourth determining unit is configured to determine second category information of the target object and a second confidence degree corresponding to the second category information according to the second point cloud data; A fifth determining unit is configured to determine second contour information of the target object according to the second point cloud data, the projection matrix, and the image data of the target object; A sixth determining unit is configured to determine second result category information of the target object according to the second contour information of the target object, the first category information of the target object, the first confidence degree corresponding to the first category information, the second category information of the target object, and the second confidence degree corresponding to the second category information; A seventh determining unit is configured to determine target category information of the target object according to the first result category information of the target object and the second result category information of the target object.

8. An electronic device, comprising: The electronic device includes: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the category determination method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to implement the category determination method of any one of claims 1-6 when executed.

Images