Object detection method and apparatus, device, computer-readable storage medium, and computer program product

Abstract

The present application provides an object detection method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product, applicable to the fields of intelligent driving and security. The method comprises: performing first object detection on a target image to obtain a first object and a first detection box corresponding to the first object, and performing second object detection on point cloud data corresponding to the target image so as to obtain a second object and a second detection box corresponding to the second object; projecting the second detection box onto the target image to obtain a projection detection box of the second detection box on the target image; matching the first detection box and the projection detection box in N dimensions to obtain the degrees of match of the first detection box and the projection detection box in the N dimensions, wherein N is an integer greater than or equal to two; and on the basis of the degrees of match of the first detection box and the projection detection box in the N dimensions, determining an object detection result indicating whether the first object is the same as the second object. The present application can improve the accuracy of object detection.

Description

Target detection method, device, equipment, computer readable storage medium and computer program product

[0001] This application claims priority to the Chinese patent application No. 202410533142.8, filed on April 29, 2024 in the China Patent Office, and entitled "Target detection method, device, equipment, computer readable storage medium and computer program product", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to computer perception technology, and in particular to a target detection method, device, equipment, storage medium and program product. BACKGROUND

[0003] With the development of computer vision and perception technology, two-dimensional target detection has become an indispensable technology in many application fields. By analyzing two-dimensional images, various targets such as pedestrians and vehicles in the two-dimensional images are detected and recognized. However, two-dimensional target detection has some limitations, such as lack of depth and position information of the target. Although radar point cloud can provide three-dimensional position information of the target compared to two-dimensional images, due to the sparsity of radar point cloud data itself, the total amount of point cloud data falling on the target is small, which can easily cause some false detection problems. Therefore, the accuracy of target detection based on two-dimensional images or radar point cloud is relatively low. TECHNICAL PROBLEM

[0004] The embodiments of the present application provide a target detection method, device, electronic equipment, computer readable storage medium and computer program product, which can improve the accuracy of target detection. TECHNICAL SOLUTION

[0005] The technical scheme of the embodiments of the present application is implemented as follows:

[0006] The embodiments of the present application provide a target detection method, comprising:

[0007] performing first target detection on a target image to obtain a first target and a first detection box corresponding to the first target, and performing second target detection on point cloud data corresponding to the target image to obtain a second target and a second detection box corresponding to the second target;

[0008] projecting the second detection box onto the target image to obtain a projection detection box of the second detection box on the target image;

[0009] performing N-dimensional matching on the first detection box and the projection detection box to obtain a matching degree of the first detection box and the projection detection box in N dimensions, N being an integer greater than or equal to 2;

[0010] determine, based on the matching degree of the first detection box and the projected detection box in the N dimensions, a target detection result of whether the first target and the second target are the same.

[0011] Embodiments of the present application provide a target detection device, comprising:

[0012] The detection module is configured to perform first target detection on a target image to obtain a first target and a first detection box corresponding to the first target, and perform second target detection on point cloud data corresponding to the target image to obtain a second target and a second detection box corresponding to the second target.

[0013] The projection module is configured to project the second detection box onto the target image to obtain a projected detection box of the second detection box on the target image.

[0014] The matching module is configured to perform matching of N dimensions on the first detection box and the projected detection box to obtain a matching degree of the first detection box and the projected detection box in the N dimensions, where N is an integer greater than or equal to 2.

[0015] The determination module is configured to determine, based on the matching degree of the first detection box and the projected detection box in the N dimensions, a target detection result of whether the first target and the second target are the same.

[0016] In the above scheme, the matching module is further configured to perform at least two of the following matching operations on the first detection box and the projected detection box: performing matching of a region dimension on the first detection box and the projected detection box to obtain a region matching degree between the first detection box and the projected detection box; performing matching of a shape dimension on the first detection box and the projected detection box to obtain a shape matching degree between the first detection box and the projected detection box; performing matching of a position dimension on the first detection box and the projected detection box to obtain a position matching degree between the first detection box and the projected detection box; performing matching of a category dimension on the first target and the second target to obtain a category matching degree between the first detection box and the projected detection box; performing matching of a color dimension on the first target and the second target to obtain a color matching degree between the first detection box and the projected detection box; and performing matching of a texture dimension on the first target and the second target to obtain a texture matching degree between the first detection box and the projected detection box.

[0017] In the above scheme, the matching module is further configured to obtain a first intersection region and a first union region between the first detection box and the projected detection box; and determine a ratio between the first intersection region and the first union region as a region matching degree between the first detection box and the projected detection box.

[0018] In the scheme, the matching module is further configured to perform center alignment on the first detection frame and the projection detection frame; obtain a second intersection region between the first detection frame and the projection detection frame after the center alignment, and obtain a second union region between the first detection frame and the projection detection frame after the center alignment; determine a ratio between the second intersection region and the second union region as a shape matching degree between the first detection frame and the projection detection frame.

[0019] In the scheme, the matching module is further configured to determine a distance between the first detection frame and the projection detection frame, and a proportion of an average value of a width and a height of the first detection frame and the projection detection frame in the target image; multiply the distance and the proportion to obtain a reference value for chi-square distribution table query, and query a target probability value from a constructed chi-square distribution table based on the reference value; obtain a mapping relationship between the target probability value and the position matching degree, and determine the position matching degree between the first detection frame and the projection detection frame based on the target probability value and the mapping relationship.

[0020] In the scheme, the matching module is further configured to determine a positioning coordinate of the first detection frame, and determine a coordinate difference value between each corner point coordinate of the projection detection frame and the positioning coordinate; average each coordinate difference value corresponding to each corner point to obtain an average coordinate difference value; normalize the average coordinate difference value based on the positioning coordinate to obtain a distance between the first detection frame and the projection detection frame.

[0021] In the scheme, the average coordinate difference value includes an average horizontal coordinate difference value and an average vertical coordinate difference value; the matching module is further configured to obtain a first difference value between a maximum horizontal coordinate and a minimum horizontal coordinate in the positioning coordinate, and a second difference value between a maximum vertical coordinate and a minimum vertical coordinate; determine a ratio of the average horizontal coordinate difference value to the first difference value as a horizontal coordinate normalization result, and a ratio of the average vertical coordinate difference value to the second difference value as a vertical coordinate normalization result; obtain a first weight of the horizontal coordinate normalization result and a second weight of the vertical coordinate normalization result, and fuse the horizontal coordinate normalization result and the vertical coordinate normalization result based on the first weight and the second weight to obtain the distance between the first detection frame and the projection detection frame.

[0022] In the scheme, the matching module is further configured to obtain a corresponding relationship among a first category of the target in the image, a second category of the target in the point cloud data, and a target similarity, wherein the target similarity is a similarity between the first category and the second category; determine the category of the first target and the category of the second target, and determine a category matching degree between the first detection frame and the projection detection frame based on the category of the first target, the category of the second target, and the corresponding relationship.

[0023] In the scheme, the matching module is further configured to obtain color information in the second target, and map the color information to a two-dimensional color space to obtain a mapping feature; perform color coding on the mapping feature to obtain a coded feature, and perform image conversion on the coded feature to obtain a first converted image corresponding to the second target; perform color extraction on the first target to obtain a first color feature, and perform color extraction on the first converted image to obtain a second color feature; and perform matching on the first color feature and the second color feature to obtain a color matching degree between the first detection frame and the projection detection frame.

[0024] In the scheme, the matching module is further configured to obtain texture information in the second target, and perform image conversion on the texture information to obtain a second converted image corresponding to the second target; perform texture extraction on the first target to obtain a first texture feature, and perform texture extraction on the second converted image to obtain a second texture feature; and perform matching on the first texture feature and the second texture feature to obtain a texture matching degree between the first detection frame and the projection detection frame.

[0025] In the scheme, the determining module is further configured to determine a weight of the matching degree of each of the N dimensions; and perform weighted summation on the matching degrees of the first detection frame and the projection detection frame in the N dimensions based on the weight of the matching degree of each of the N dimensions, to obtain a target detection result of whether the first target and the second target are the same.

[0026] In the scheme, the determining module is further configured to, for the matching degree of each of the N dimensions, compare the matching degree with a preset matching degree threshold, and determine an initial detection result of whether the first target and the second target are the same in the corresponding dimension based on a comparison result; count a first number of the initial detection results indicating that the first target and the second target are the same in the corresponding dimension, and a second number of the initial detection results indicating that the first target and the second target are different in the corresponding dimension; and determine a target detection result of whether the first target and the second target are the same based on the first number and the second number.

[0027] In the scheme, the detection module is further configured to perform sparse conversion on the point cloud data corresponding to the target image to obtain a plurality of voxels corresponding to the point cloud data; perform mapping on each voxel to obtain a mapping image corresponding to each voxel; and perform second target detection on the mapping image corresponding to each voxel to obtain a second target and a second detection box corresponding to the second target.

[0028] In the scheme, the projection module is further configured to determine a conversion relationship between a radar coordinate system and a camera coordinate system, wherein the radar coordinate system is a coordinate system of a radar that collects the point cloud data, and the camera coordinate system is a coordinate system of a camera that collects the target image; and project the second detection box onto the target image based on the conversion relationship to obtain a projection detection box of the second detection box on the target image.

[0029] In the scheme, the second detection box includes a plurality of corner points, and the projection module is further configured to obtain position information of the second detection box in the radar coordinate system, and perform projection transformation on each corner point based on the position information and the conversion relationship to obtain pixel coordinates of each corner point on the target image; and filter a minimum horizontal coordinate, a maximum horizontal coordinate, a minimum vertical coordinate, and a maximum vertical coordinate from the pixel coordinates of each corner point on the target image; and determine the projection detection box of the second detection box on the target image based on a detection box composed of the minimum horizontal coordinate, the maximum horizontal coordinate, the minimum vertical coordinate, and the maximum vertical coordinate.

[0030] An electronic device is provided in an embodiment of the present application, and includes:

[0031] A memory is configured to store computer executable instructions or computer programs.

[0032] A processor is configured to execute the computer executable instructions or computer programs stored in the memory to implement the target detection method provided in the embodiments of the present application.

[0033] A computer readable storage medium is provided in an embodiment of the present application, and stores computer executable instructions or computer programs, which are configured to be executed by a processor to implement the target detection method provided in the embodiments of the present application.

[0034] A computer program product is provided in an embodiment of the present application, and includes a computer program or computer executable instructions, which are executed by a processor to implement the target detection method provided in the embodiments of the present application. Advantages

[0035] The embodiments of the present application have the following advantages:

[0036] By applying the embodiment of the present application, target detection is performed in combination with a target image and point cloud data corresponding to the target image, a first detection box obtained by detecting the target image and a projection detection box obtained by detecting the point cloud data are matched in multiple dimensions, and a final target detection result is determined in combination with matching results in the multiple dimensions. In this way, various types of information in the target image and the point cloud data can be fully utilized, and the accuracy and reliability of the final detection result can be improved by matching in multiple dimensions. BRIEF DESCRIPTION OF DRAWINGS

[0037] FIG. 1 is a schematic diagram of an architecture of a target detection system 100 provided by an embodiment of the present application;

[0038] FIG. 2 is a schematic diagram of a structure of an electronic device 500 provided by an embodiment of the present application;

[0039] FIG. 3A is a schematic diagram of a flow of a target detection method provided by an embodiment of the present application;

[0040] FIG. 3B is a schematic diagram of a projection method of a detection box provided by an embodiment of the present application;

[0041] FIG. 3C is a schematic diagram of a matching method of a detection box provided by an embodiment of the present application;

[0042] FIG. 3D is a schematic diagram of a matching method of a position dimension provided by an embodiment of the present application;

[0043] FIG. 3E is a schematic diagram of a matching method of a color dimension provided by an embodiment of the present application;

[0044] FIG. 3F is a schematic diagram of a matching method of a texture dimension provided by an embodiment of the present application;

[0045] FIG. 4 is a schematic diagram of a detection result of a target image provided by an embodiment of the present application;

[0046] FIG. 5 is a schematic diagram of a detection result of point cloud data provided by an embodiment of the present application;

[0047] FIG. 6 is a schematic diagram of a composition of a second detection box provided by an embodiment of the present application;

[0048] FIG. 7 is a schematic diagram of a projection of a detection box provided by an embodiment of the present application;

[0049] FIG. 8 is a schematic diagram of matching in a region dimension provided by an embodiment of the present application;

[0050] FIG. 9 is a schematic diagram of matching in a shape dimension provided by an embodiment of the present application;

[0051] FIG. 10 is a schematic diagram of a projection of point cloud data provided by an embodiment of the present application. Embodiments of the present application

[0052] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings, and the described embodiments should not be regarded as limitations to the present application. All other embodiments obtained by those of ordinary skill in the art without creative work under the premise that the described embodiments do not fall within the scope of the present application.

[0053] It can be understood that, in the embodiments of the present application, the data related to user information (for example, images related to users and the like) need to obtain the permission or consent of the user when the embodiments of the present application are applied to specific products or technologies, and the collection, use and processing of the related data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0054] In the following description, "some embodiments" are related to a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0055] In the following description, the term "first\second" is only to distinguish similar objects, and does not represent the specific order of the objects. It can be understood that "first\second" can interchange the specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0056] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0058] The embodiment of the present application provides a target detection method and device, electronic equipment, computer readable storage medium and computer program product, which can improve the accuracy of target detection. The following describes an exemplary application of the electronic equipment provided by the embodiment of the present application. The electronic equipment provided by the embodiment of the present application can be implemented as a notebook computer, a tablet computer, a desktop computer, a set-top box, a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated message device, a portable game device), a smart phone, a smart speaker, a smart watch, a smart television, a vehicle-mounted terminal, a security terminal, and various types of user terminals. It can also be implemented as a server. The following describes an exemplary application when the device is implemented as a terminal.

[0059] Referring to FIG. 1, FIG. 1 is an architecture schematic diagram of a target detection system 100 provided by the embodiment of the present application. To realize an exemplary application, the terminal (exemplarily shown as terminal 400-1 and terminal 400-2) connects the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0060] In some embodiments, a client or application program with target detection function is arranged on the terminal, which is used to realize the target detection method provided by the embodiment of the present application, such as performing first target detection on the target image to obtain a first target and a first detection box corresponding to the first target, and performing second target detection on the point cloud data corresponding to the target image to obtain a second target and a second detection box corresponding to the second target; projecting the second detection box onto the target image to obtain a projection detection box of the second detection box on the target image; performing N-dimensional matching on the first detection box and the projection detection box, obtaining a matching degree of the first detection box and the projection detection box in N dimensions, N being an integer greater than or equal to 2; and determining a target detection result of whether the first target and the second target are the same based on the matching degree of the first detection box and the projection detection box in N dimensions.

[0061] In some embodiments, the target detection method provided by the embodiments of the present application can also be implemented by a terminal and a server, that is, the terminal acquires a target image to be detected and point cloud data corresponding to the target image, and sends the acquired target image and the point cloud data corresponding to the target image to the server 200; the server 200 performs first target detection on the target image to obtain a first target and a first detection box corresponding to the first target, and performs second target detection on the point cloud data corresponding to the target image to obtain a second target and a second detection box corresponding to the second target; the second detection box is projected onto the target image to obtain a projection detection box of the second detection box on the target image; the first detection box and the projection detection box are matched in N dimensions, where N is an integer greater than or equal to 2, to obtain a matching degree of the first detection box and the projection detection box in the N dimensions; based on the matching degree of the first detection box and the projection detection box in the N dimensions, a target detection result of whether the first target and the second target are the same is determined, and the final target detection result is returned to the terminal.

[0062] In some embodiments, the server 200 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms, etc. Basic cloud computing services. The terminal and the server can be connected directly or indirectly through wired or wireless communication, which is not limited in the embodiments of the present application.

[0063] Referring to FIG. 2, FIG. 2 is a structural schematic diagram of an electronic device 500 provided by the embodiments of the present application. Taking the terminal in FIG. 1 as an example, the electronic device 500 shown in FIG. 2 includes at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between the components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the purpose of clear illustration, all kinds of buses are marked as the bus system 540 in FIG. 2.

[0064] The processor 510 can be an integrated circuit chip with signal processing capability, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor.

[0065] The memory 550 includes volatile memory or nonvolatile memory, and can include both volatile and nonvolatile memory. Among them, the nonvolatile memory can be a read-only memory (ROM), and the volatile memory can be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory. The memory 550 can optionally include one or more storage devices that are physically separate from the processor 510.

[0066] In some embodiments, the memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, which are exemplarily illustrated below.

[0067] The operating system 551 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks; the network communication module 552 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 520, and exemplary network interfaces 520 include Bluetooth, wireless compatibility authentication (WiFi), and universal serial bus (USB), and the like.

[0068] In some embodiments, the target detection apparatus provided by the embodiments of the present application can be realized in a software manner, and the target detection apparatus provided by the embodiments of the present application can be provided in various software embodiments, including various forms such as application programs, software, software modules, scripts, or codes. FIG. 2 shows a target detection apparatus 555 stored in the memory 550, which can be software in the form of programs and plug-ins, and includes a series of modules, including a detection module 5551, a projection module 5552, a matching module 5553, and a determination module 5554. These modules are logical, and thus can be combined or further split according to the functions implemented, and the functions of each module will be described below.

[0069] In some embodiments, the apparatus provided by the embodiments of the present application can be implemented in a hardware manner. For example, the apparatus provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to perform the target detection method provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can be implemented by using one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs), Field-Programmable Gate Arrays (FPGAs), or other electronic elements.

[0070] In some embodiments, the terminal or server can implement the target detection method provided by the embodiments of the present application by running various computer-executable instructions or computer programs. For example, the computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. The computer program can be a native program in an operating system or a software module; can be a Native APPlication (APP), i.e., a program that needs to be installed in an operating system to run, such as a vehicle navigation APP, a security APP; or can be a small program that can be embedded into any APP, i.e., a program that only needs to be downloaded into a browser environment to run. In summary, the above computer-executable instructions can be any form of instructions, and the above computer programs can be any form of application programs, modules, or plug-ins.

[0071] The target detection method provided by the embodiments of the present application will be described in conjunction with an exemplary application and implementation of an electronic device provided by the embodiments of the present application. As described above, the target detection method provided by the embodiments of the present application can be implemented by various types of electronic devices, for example, can be performed by any one of the terminal and the server 200 in FIG. 1 individually, or can be performed by the terminal and the server 200 in FIG. 1 cooperatively. Therefore, the execution subject of each step will not be described again hereinafter.

[0072] It should be noted that in the following examples of target detection, the target image is taken as an example for illustration. Those skilled in the art can apply the target detection method provided by the embodiments of the present application to the detection of a plurality of video frames continuously acquired according to the understanding of the following.

[0073] Referring to FIG. 3A, FIG. 3A is a flowchart of a target detection method provided in an embodiment of the present application. The steps shown in FIG. 3A will be described in combination.

[0074] In step 101, first target detection is performed on the target image to obtain a first target and a first detection box corresponding to the first target, and second target detection is performed on the point cloud data corresponding to the target image to obtain a second target and a second detection box corresponding to the second target.

[0075] In actual applications, before performing first target detection on the two-dimensional target image, a two-dimensional detection model for first target detection can be trained. When training the two-dimensional detection model, a training data set and a test data set containing target category labels can be prepared first. Each training sample in the training data set includes an image sample and a detection target and a detection box labeled for the image sample. Then, a YOLO (You Only Look Once) model or a single-stage target detection model (SSD, Single Shot MultiBox Detector) is used as a pre-training model for training. For example, each training sample in the training data set is input into the pre-training model, and a predicted target and a predicted detection box for each training sample are output. A loss function is constructed based on the predicted target and the labeled detection target, the predicted detection box and the labeled detection box. The model parameters of the pre-training model are updated based on the constructed loss function, so as to train the two-dimensional detection model.

[0076] When performing first target detection on the two-dimensional target image, a checkerboard calibration method can be used to calibrate the camera for collecting the target image to obtain the intrinsic parameters of the camera. The collected target image is rectified according to the intrinsic parameters of the camera to obtain a target image without distortion. The obtained target image without distortion is input into the trained target detection model to perform first target detection on the target image without distortion by the target detection model to obtain a two-dimensional detection result. Referring to FIG. 4, FIG. 4 is a schematic diagram of a detection result of a target image provided in an embodiment of the present application. As shown in FIG. 4, the two-dimensional detection result includes a recognized first target (or a category of the first target) and a first detection box corresponding to the first target. The number of the first target can be one or more. Correspondingly, the number of the first detection box is also one or more. The first detection box carries position information (x, y, w, h) and a confidence. x and y represent the offset of the center point of the first detection box relative to the top left corner of the grid cell. w and h represent the width and height of the first detection box. The confidence is used to indicate the probability that the first target exists in the first detection box. When the confidence is greater than a confidence threshold, it is considered that the first target exists in the first detection box.

[0077] In this way, before the second target detection is performed on the target image to be detected, the target image is first subjected to distortion correction, thereby improving the image quality of the target image and facilitating improvement of the accuracy of subsequent target detection.

[0078] When the point cloud data corresponding to the target image (the so-called correspondence refers to the time and position at which the camera collects the target image, which corresponds to the time and position at which the radar collects the point cloud data, that is, the target image and the point cloud data are data obtained at the same time and at the same position) is subjected to the second target detection, the second target and the second detection box corresponding to the second target are obtained by performing the second target detection on the point cloud data corresponding to the target image in step 101 shown in FIG. 3A, which can be implemented in the following manner: first, the point cloud data corresponding to the target image is subjected to sparse conversion to obtain a plurality of voxels corresponding to the point cloud data, so that the point cloud data is included in the plurality of voxels to form a regular and densely distributed voxel set; then, each voxel is mapped to obtain a mapping image corresponding to each voxel, such as projection mapping processing of each voxel in the voxel set from the front view and the top view to obtain a plurality of pseudo images (i.e., mapping images) for the convolutional neural network processing; and finally, the mapping image corresponding to each voxel is subjected to the second target detection, such as inputting the mapping image into the trained detection model (such as the PointPillars model) to obtain the second target and the second detection box corresponding to the second target.

[0079] In addition, before the point cloud data is subjected to the second target detection, the three-dimensional detection model for the second target detection can still be trained (the training manner of the three-dimensional detection model and the two-dimensional detection model is consistent, which will not be described herein again), and the point cloud data is directly input into the trained three-dimensional detection model to obtain a three-dimensional detection result, that is, the second target and the second detection box corresponding to the second target.

[0080] Referring to FIG. 5, FIG. 5 is a schematic diagram of a detection result of point cloud data according to an embodiment of the present application. As shown in FIG. 5, after the point cloud data is subjected to the second target detection processing, the output visualized three-dimensional detection result includes the recognized second target (or the category of the second target) and the second detection box (a three-dimensional detection box, such as a cuboid or a cube) corresponding to the second target. The number of the second target can be one or more, and the number of the second detection box is also one or more. The second detection box carries position information (x, y, z, l, w, h, R), wherein (x, y, z) refers to the three-dimensional coordinates of the center point of the second detection box in the radar coordinate system, (l, w, h) represents the length, width and height of the second detection box, and R is a rotation matrix. Different colors of different second detection boxes represent different categories of the corresponding second target, such as blue representing a car and red representing a pedestrian.

[0081] By the above manner, since the two-dimensional target image can provide color, texture and other information, the three-dimensional point cloud data can provide spatial position information, and target detection is combined with multi-dimensional information such as color, texture, spatial position information, which is beneficial to improve the accuracy of target detection.

[0082] In step 102, the second detection frame is projected onto the target image to obtain a projection detection frame of the second detection frame on the target image.

[0083] In some embodiments, referring to FIG. 3B, which is a schematic diagram of a projection method of a detection frame provided by the embodiments of the present application, the step 102 shown in FIG. 3A can be implemented by steps 1021-1022 shown in FIG. 3B:

[0084] In step 1021, a conversion relationship between a radar coordinate system and a camera coordinate system is determined, wherein the radar coordinate system is a coordinate system of a radar collecting point cloud data, and the camera coordinate system is a coordinate system of a camera collecting a target image.

[0085] Here, first, the camera is calibrated according to the target image collected by the camera, such as calibrating the camera by Zhang Dingyou calibration method to obtain camera intrinsic parameters and distortion coefficients, and the target image is rectified according to the distortion coefficients to obtain a non-distortion image; then, more than 4 pairs of sensing points in the point cloud data and corresponding image pixel points are selected, the three-dimensional coordinates (X, Y, Z) of the sensing points and the pixel point coordinates (u, v) of the sensing points projected onto the target image are used, and the PnP (Perspective-n-Point) algorithm is used to determine the extrinsic parameters (i.e. the rotation matrix R and the translation matrix T) between the camera and the radar, complete joint calibration, so as to correspond the pixel points of the camera and the sensing points of the radar, wherein the corresponding relationship (i.e. the conversion relationship) between the radar coordinate system and the camera coordinate system can be represented by formula (1):

[0086] Wherein, f u and f v are the focal length in horizontal and vertical directions respectively, u0 and v0 are the image center point coordinates, R and T are the rotation matrix and translation matrix between the camera and the radar respectively.

[0087] In step 1022, based on the conversion relationship, the second detection frame is projected onto the target image to obtain a projection detection frame of the second detection frame on the target image.

[0088] In some embodiments, the second detection box includes a plurality of corner points, and step 1022 can be implemented by: obtaining position information of the second detection box in the radar coordinates, and performing projection transformation on each corner point based on the position information and the conversion relationship to obtain pixel coordinates of each corner point on the target image; and selecting a minimum horizontal coordinate, a maximum horizontal coordinate, a minimum vertical coordinate and a maximum vertical coordinate from the pixel coordinates of each corner point on the target image; and determining a projection detection box of the second detection box on the target image based on a detection box composed of the minimum horizontal coordinate, the maximum horizontal coordinate, the minimum vertical coordinate and the maximum vertical coordinate.

[0089] As described above, the second detection box carries position information (x, y, z, l, w, h, R), wherein (x, y, z) refers to three-dimensional coordinates of a center point of the second detection box in the radar coordinate system, (l, w, h) represents length, width and height of the second detection box, and R is a rotation matrix. The second detection box can be regarded as a cube including eight corner points, as shown in FIG. 6, which is a composition diagram of the second detection box provided by an embodiment of the present application. The positive direction of the xyz axis of the point cloud coordinate system is shown by an arrow in FIG. 6. According to the position information (x, y, z, l, w, h, R) carried on the second detection box, the three-dimensional coordinates of the eight corner points of the second detection box can be calculated by using formula (2).

[0090] Wherein, * represents matrix multiplication.

[0091] Then, the eight corner points of the second detection box are projected onto the target image respectively, and the projection pixel point coordinates (i.e., pixel coordinates) (u, v) of the eight corner points on the target image are calculated by using formula (1). The minimum horizontal coordinate u min , the maximum horizontal coordinate u max , the minimum vertical coordinate v min and the maximum vertical coordinate v max are selected from the eight projection pixel point coordinates. The projection detection box obtained after projection can be represented as (u min , v min , u max , v max ).

[0092] For example, referring to FIG. 7, which is a projection diagram of a detection box provided by an embodiment of the present application, after the three-dimensional detection result (including a plurality of second detection boxes) obtained by performing the second target detection on the point cloud data is projected onto the target image, the projection detection box corresponding to each second detection box is obtained, and the projection detection box is represented by a rectangular box 2. The rectangular box 1 in the rectangular box 2 is used to represent the first detection box obtained by performing the first target detection on the target image.

[0093] By the above manner, the second detection box obtained by performing the second target detection on the point cloud data is projected onto the target image to obtain a projection detection box. Since the projection detection box and the first detection box obtained by performing the first target detection on the target image are in the same dimension, the three-dimensional detection result and the two-dimensional detection result have matchability.

[0094] In step 103, the first detection box and the projection detection box are matched in N dimensions to obtain a matching degree of the first detection box and the projection detection box in the N dimensions.

[0095] Wherein, N is an integer greater than or equal to 2, here, after the second detection box obtained by performing the second target detection on the point cloud data is projected onto the target image to obtain the projection detection box, when the projection detection box and the first detection box obtained by performing the first target detection on the target image are matched, the matching is performed from N (i.e. multiple) dimensions, and finally the final target detection result is determined according to the matching result of the N dimensions, wherein the N dimensions include but are not limited to: region, shape, position, category, color, texture. Since the multi-dimensional matching fully considers the matching degree of various information, the accuracy of the final target detection result is greatly improved.

[0096] In some embodiments, referring to FIG. 3C, which is a schematic diagram of a matching method of a detection box provided in an embodiment of the present application, step 103 shown in FIG. 3A can be implemented by performing at least two steps of steps 1031-1036 shown in FIG. 3C on the first detection box and the projection detection box:

[0097] In step 1031, the first detection box and the projection detection box are matched in the region dimension to obtain a region matching degree between the first detection box and the projection detection box.

[0098] In some embodiments, step 1031 can be implemented in the following manner: obtaining a first intersection region and a first union region between the first detection box and the projection detection box; determining a ratio between the first intersection region and the first union region as the region matching degree between the first detection box and the projection detection box.

[0099] Referring to FIG. 8, which is a schematic diagram of matching in the region dimension provided in an embodiment of the present application, the first detection box is represented by A, and the projection detection box is represented by B. First, a first intersection region (A∩B) between the first detection box and the projection detection box is obtained, then a first union region (A∪B) between the first detection box and the projection detection box is obtained, and then the first intersection region (A∩B) is divided by the first union region (A∪B) to obtain the region matching degree between the first detection box and the projection detection box, that is, the region matching degree (simIOU) can be represented as: simIOU=(A∩B) / (A∪B).

[0100] In step 1032, shape dimension matching is performed on the first detection frame and the projection detection frame to obtain a shape matching degree between the first detection frame and the projection detection frame.

[0101] In some embodiments, step 1032 can be implemented by the following manner: center alignment is performed on the first detection frame and the projection detection frame; a second intersection region between the center-aligned first detection frame and the projection detection frame is obtained, and a second union region between the center-aligned first detection frame and the projection detection frame is obtained; a ratio between the second intersection region and the second union region is determined as the shape matching degree between the first detection frame and the projection detection frame.

[0102] Referring to FIG. 9, which is a shape dimension matching diagram provided by the embodiments of the present application, the first detection frame is represented by C, and the projection detection frame is represented by D. When shape dimension matching is performed on the first detection frame and the projection detection frame, the center points of the first detection frame C and the projection detection frame D need to be moved to the same position (i.e., center alignment). After center alignment, a second intersection region (C∩D) between the first detection frame and the projection detection frame is obtained, and then a second union region (C∪D) between the first detection frame and the projection detection frame is obtained. Then, the second intersection region (C∩D) is divided by the second union region (C∪D) to obtain the shape matching degree between the first detection frame and the projection detection frame, i.e., the shape matching degree (simShape) can be represented as: simShape=(C∩D) / (C∪D).

[0103] In this way, the position influence between the first detection frame and the projection detection frame is excluded. When the length or width of the first detection frame and the projection detection frame is closer, the calculated shape matching degree will be larger, i.e., the shape similarity is higher.

[0104] In step 1033, position dimension matching is performed on the first detection frame and the projection detection frame to obtain a position matching degree between the first detection frame and the projection detection frame.

[0105] In some embodiments, referring to FIG. 3D, which is a position dimension matching method diagram provided by the embodiments of the present application, step 1033 shown in FIG. 3C can be implemented by steps 10331-10333 shown in FIG. 3D.

[0106] In step 10331, a distance between the first detection frame and the projection detection frame is determined, and a ratio of an average width and height of the first detection frame and the projection detection frame in the target image is determined.

[0107] In some embodiments, the ratio of the average width and height of the first detection frame and the projection detection frame in the target image can be determined by formula (3), which can be represented as: ratio=(((W 2D +W3D ) / 2 / img_width)+((H 2D +H 3D ) / 2 / img_height)) / 2 (3)

[0108] wherein, W 2D and H 2D are the width and height of the first bounding box respectively, W 3D and H 3D are the width and height of the projected bounding box respectively, img_width and img_height are the width and height of the target image respectively.

[0109] In some embodiments, the distance between the first bounding box and the projected bounding box can be determined by: determining a positioning coordinate of the first bounding box, and determining a coordinate difference between a positioning coordinate of each corner point of the projected bounding box and the positioning coordinate; averaging the coordinate difference corresponding to each corner point respectively to obtain an average coordinate difference; and normalizing the average coordinate difference based on the positioning coordinate to obtain the distance between the first bounding box and the projected bounding box.

[0110] Here, when determining the positioning coordinate of the first bounding box, the left upper corner point and the right lower corner point of the first bounding box can be uniquely determined as one bounding box, and the positioning coordinate of the first bounding box can be represented as ((x min ,y min ), (x max ,y max )); the positioning coordinate of the projected bounding box can be represented by the corner point coordinates of the four corner points, such as the positioning coordinate u of the projected bounding box, wherein u = (u1, u2, u3, u4), u1 = (u 1x ,u 1y ), u2 = (u 2x ,u 2y ), u3 = (u 3x ,u 3y ), u4 = (u 4x ,u 4y ), u 1x and u 1y represent the coordinates of the corner point u1 on the x and y axes, u 2x and u 2y represent the coordinates of the corner point u2 on the x and y axes, u 3x and u 3y represent the coordinates of the corner point u3 on the x and y axes, and u 4x and u 4y represent the coordinates of the corner point u4 on the x and y axes.

[0111] Then, the coordinate difference value of each corner point of the projection detection frame and the first detection frame in the x and y axis directions is calculated respectively, for example, the coordinate difference value of the corner point u1 and the first detection frame in the x and y axis directions is calculated, which can be calculated by formula (4):

[0112] wherein, du 1x and du 1y are the coordinate difference value of the corner point u1 of the projection detection frame and the first detection frame in the x and y axis directions respectively; in the same way, the coordinate difference value of the corner point u2 and the first detection frame in the x and y axis directions du 2x and du 2y , the coordinate difference value of the corner point u3 and the first detection frame in the x and y axis directions du 3x and du 3y , and the coordinate difference value of the corner point u4 and the first detection frame in the x and y axis directions du 4x and du 4y .

[0113] Then, the coordinate difference value corresponding to each corner point is averaged respectively to obtain the average coordinate difference value, that is, the average coordinate difference value in the x axis (referred to as the average horizontal coordinate difference value) is obtained by averaging the coordinate difference values du 1x , du 2x , du 3x and du 4x in the x axis, and the average coordinate difference value in the y axis (referred to as the average vertical coordinate difference value) is obtained by averaging the coordinate difference values du 1y , du 2y , du 3y and du 4y in the y axis, which can be shown in formula 5:

[0114] wherein, meandx is the average horizontal coordinate difference value, and meandy is the average vertical coordinate difference value.

[0115] Finally, based on the positioning coordinate, the average coordinate difference value is normalized to obtain the distance between the first detection frame and the projection detection frame, for example, the first difference value (x max -x min ) between the maximum horizontal coordinate and the minimum horizontal coordinate in the positioning coordinate of the first detection frame, and the second difference value (y max -y min) as a horizontal coordinate normalization result and a ratio of the average vertical coordinate difference value to the second difference value as a vertical coordinate normalization result; obtaining a first weight of the horizontal coordinate normalization result and a second weight of the vertical coordinate normalization result, and fusing the horizontal coordinate normalization result and the vertical coordinate normalization result based on the first weight and the second weight to obtain the distance between the first detection frame and the projection detection frame.

[0116] wherein the average horizontal coordinate difference value and the average vertical coordinate difference value can be normalized by formula (6), and the distance between the first detection frame and the projection detection frame is obtained based on the weights, and formula (6) is expressed as follows:

[0117] wherein normx represents the horizontal coordinate normalization result, normy represents the vertical coordinate normalization result, square_norm is the distance between the first detection frame and the projection detection frame, x_std_dev is the first weight of the horizontal coordinate normalization result, and y_std_dev is the second weight of the vertical coordinate normalization result, and the first weight and the second weight can be set according to actual conditions, for example, according to experience, x_std_dev can be set as 0.4 and y_std_dev can be set as 0.5.

[0118] In step 10332, the product of the distance and the proportion is taken as a reference value for chi-square distribution table query, and the target probability value is queried from the constructed chi-square distribution table based on the reference value.

[0119] Here, after the distance square_norm between the first detection frame and the projection detection frame and the proportion ratio of the average width and height of the first detection frame and the projection detection frame in the target image are determined, the product of the two is taken as the reference value for chi-square distribution table query, that is, the reference value dist=square_norm*ratio.

[0120] In actual application, when constructing the chi-square distribution table, the degrees of freedom and the significance level of the chi-square distribution need to be known, wherein the degrees of freedom are usually related to the number of observation values, and the significance level is a probability threshold for determining the rejection of the null hypothesis, here, the degrees of freedom can be assumed to be 2, and the significance level is assumed to be 0.05, when constructing the chi-square distribution table, the value is set from 0 to 0.999953, after the above parameters are determined, a fixed chi-square distribution table can be generated for subsequent query.

[0121] For example, a chi-square distribution table Table = [0.00000, 0.024690, 0.048771, 0.072257, 0.095163, 0.117503, 0.139293, 0.160543, 0.181269, 0.201484, 0.221199, 0.240428, 0.259182, 0.277473, 0.295312, 0.312711, 0.329680, 0.346230, 0.362372, 0.378115, 0.393469, 0.408445, 0.423050, …, 0.999953] is built by code, and the purpose of this part of code is to query the pre-computed chi-square distribution table and use linear interpolation to approximate the cumulative probability of any chi-square value, which can avoid complex numerical integration calculation during code running, thereby improving efficiency.

[0122] The chi-square distribution table is built by the above method, assuming that the number of values in the chi-square distribution table is table_size, and the chi-square distribution table contains chi-square values and corresponding probability values. In subsequent table lookup, the target probability value can be queried from the built chi-square distribution table based on the reference value dist, and the specific lookup method is as follows:

[0123] step = 0.050000; dist_ind = dist / step, where dist_ind is of floating point type; dist_int = int(dist_ind), where dist_int is the integer part of dist_ind; w = dist_ind-dist_int; if dist_ind >= table_size-1, then the position matching degree simLoc between the first detection frame and the projection detection frame is 0; otherwise, step 10333 is executed.

[0124] In step 10333, the mapping relationship between the target probability value and the position matching degree is obtained, and based on the target probability value and the mapping relationship, the position matching degree between the first detection frame and the projection detection frame is determined.

[0125] Here, the mapping relationship between the target probability value and the position matching degree can be represented by formula (7): simLoc = 1-(table[dist_int]*(1-w)+table[dist_int+1]*w) (7)

[0126] Where table[dist_int] represents the target probability value corresponding to the dist_intth number in the chi-square distribution table, and simLoc represents the position matching degree between the first detection frame and the projection detection frame.

[0127] For example, assuming that dist conforms to a Gaussian distribution, dist = 2.3, step = 0.05, table_size = 400, dist_ind = 46, dist_int = 46, w = 0, the queried table[dist_int] = 0.683363, and the formula (7) is used, the position matching degree simLoc between the first detection frame and the projection detection frame can be calculated as simLoc = 1-0.683363 = 0.316637.

[0128] In the above manner, the position matching degree between the first detection frame and the projection detection frame is calculated by using the chi-square test method, and the accuracy of target detection can be improved.

[0129] In step 1034, the category dimension of the first target and the second target is matched to obtain the category matching degree between the first detection frame and the projection detection frame.

[0130] In some embodiments, the above step 1034 can be implemented in the following manner: obtaining a corresponding relationship between a first category of a target in an image, a second category of the target in point cloud data, and a target similarity, wherein the target similarity is a similarity between the first category and the second category; determining the category of the first target and the category of the second target, and determining the category matching degree between the first detection frame and the projection detection frame based on the category of the first target, the category of the second target, and the corresponding relationship.

[0131] In actual applications, the corresponding relationship between the target similarities can be the first category of a target obtained by performing target detection on an image, and the second category of the target obtained by performing target detection on point cloud data corresponding to the image. Assuming that the first category obtained by performing first target detection on a target image is respectively: pedestrian, car, truck, motorcycle, bicycle, traffic pyramid, and trailer, and the second category obtained by performing second target detection on the point cloud data is respectively: pedestrian, car, truck, motorcycle, bicycle, traffic pyramid, and other vehicles, a similarity table is constructed based on experience, as shown in Table 1.

[0132] Table 1

[0133] Here, after the first target detection on the target image obtains the category of the first target (i.e., the category corresponding to the first detection box), and the second target detection on the point cloud data corresponding to the target image obtains the category of the second target (i.e., the category corresponding to the projection detection box), the category similarity is directly queried in Table 1 based on the category of the first target and the category of the second target as the category matching degree (simClass) between the first detection box and the projection detection box. For example, if the category of the first target is a pedestrian and the category of the second target is also a pedestrian, the category similarity between the first target and the second target is 1, that is, the category matching degree between the first detection box and the projection detection box is 1.

[0134] In step 1035, the color dimension matching of the first target and the second target is performed to obtain the color matching degree between the first detection box and the projection detection box.

[0135] In some embodiments, referring to FIG. 3E, which is a schematic diagram of a color dimension matching method provided by an embodiment of the present application, the step 1035 shown in FIG. 3C can be implemented by steps 10351-10354 shown in FIG. 3E:

[0136] In step 10351, color information in the second target is obtained, and the color information is mapped to a two-dimensional color space to obtain a mapping feature.

[0137] Here, since the second target is obtained by the second target detection on the three-dimensional point cloud data, color information (such as reflectivity or intensity value) needs to be extracted from the second target, and the color information is quantized or normalized to be mapped to a two-dimensional color space to obtain a mapping feature.

[0138] In the above manner, the color information of the second target of the three-dimensional point cloud data is mapped to a two-dimensional color space, which facilitates the matching with the color information of the two-dimensional first target.

[0139] In step 10352, the mapping feature is color-coded to obtain a coded feature, and the coded feature is image-converted to obtain a first converted image corresponding to the second target.

[0140] Here, after obtaining the mapping feature of the color information of the second target in the two-dimensional color space, the mapping feature is color-coded, that is, a corresponding color code is assigned to each color information mapping feature value, and the color-coded coded feature is converted into an image, such as being mapped to a corresponding pixel position according to a coding rule to generate a first converted image with color features.

[0141] In this way, the three-dimensional color information in the radar detection result is converted into color features in the two-dimensional image, and the detection result of the target image is a unified two-dimensional representation, so as to realize color matching between the first detection frame and the second detection frame.

[0142] In step 10353, color extraction is performed on the first target to obtain a first color feature, and color extraction is performed on the first converted image to obtain a second color feature.

[0143] In actual applications, color extraction techniques such as color histogram, color space conversion, or color filtering can be used to extract the first color feature from the second target and the corresponding second color feature from the first converted image for subsequent color matching.

[0144] In step 10354, the first color feature and the second color feature are matched to obtain a color matching degree between the first detection frame and the projected detection frame.

[0145] Here, after obtaining the first color feature of the first target and the corresponding second color feature of the second target, the first color feature and the second color feature can be matched, such as calculating the Euclidean distance or cosine similarity of the first color feature and the second color feature, and the obtained distance or similarity value is taken as the color matching degree (denoted as simCol) between the first detection frame and the projected detection frame.

[0146] In this way, the color features of the target in the image detection result and the radar detection result can be effectively matched, thereby realizing more accurate and reliable target detection.

[0147] In step 1036, the first target and the second target are matched in texture dimension to obtain a texture matching degree between the first detection frame and the projected detection frame.

[0148] In some embodiments, referring to FIG. 3F, which is a schematic diagram of a texture dimension matching method according to an embodiment of the present application, step 1036 shown in FIG. 3C can be implemented by steps 10361-10363 shown in FIG. 3F:

[0149] In step 10361, texture information in the second target is obtained, and the texture information is converted to obtain a second converted image corresponding to the second target.

[0150] Here, first, texture information such as reflectivity or echo intensity distribution of the second target is extracted from the radar detection result, which is usually in the form of one-dimensional data or two-dimensional matrix; then, the extracted texture information is processed by image processing, which is converted into a data format similar to an image, such as a gray-scale image, a pseudo-color image, etc. in the form of a second conversion image according to the distribution rule and characteristics of the texture information, so as to facilitate subsequent texture feature extraction.

[0151] In step 10362, texture extraction is performed on the first target to obtain the first texture feature, and texture extraction is performed on the second conversion image to obtain the second texture feature.

[0152] In actual applications, texture extraction techniques such as gray-level co-occurrence matrix, local binary pattern, and histogram of oriented gradients can be used to extract the first texture feature in the first target and the second texture feature in the second conversion image for subsequent texture matching.

[0153] In step 10363, the first texture feature and the second texture feature are matched to obtain the texture matching degree between the first detection frame and the projection detection frame.

[0154] Here, after obtaining the first texture feature of the first target and the second texture feature corresponding to the second target, the first texture feature and the second texture feature can be matched, such as calculating the Euclidean distance or cosine similarity of the first texture feature and the second texture feature, and the obtained distance or similarity value is taken as the texture matching degree (denoted as simTex) between the first detection frame and the projection detection frame.

[0155] In this way, the texture features of the target in the image detection result and the radar detection result can be effectively matched, thereby realizing more accurate and reliable target detection.

[0156] In step 104, based on the matching degrees of the first detection frame and the projection detection frame in N dimensions, the target detection result of whether the first target and the second target are the same is determined.

[0157] In actual applications, after determining the matching degrees of the first detection frame and the projection detection frame in multiple dimensions, the matching degrees of the first detection frame and the projection detection frame in multiple dimensions can be combined to determine the target detection result of whether the first target and the second target are the same.

[0158] In some embodiments, the above step 104 can be implemented in the following way: determining the weight of the matching degree of each dimension in N dimensions; based on the weight of the matching degree of each dimension in N dimensions, the matching degrees of the first detection frame and the projection detection frame in N dimensions are weighted and summed to obtain the target detection result of whether the first target and the second target are the same.

[0159] As an example, after obtaining the region matching degree (simIOU), shape matching degree (simShape), location matching degree (simLoc), category matching degree (simClass), color matching degree (simCol), and texture matching degree (simTex) of the first detection box and the projected detection box, the weights of the matching degrees in each dimension are obtained, and the weights of the above matching degrees are α IOU Shape Loc Class Col Tex That is, based on the weights, the matching degrees in each dimension are weighted and summed to obtain the comprehensive matching degree sim, and the comprehensive matching degree is taken as the final target detection result, as shown in formula (8):

[0160] Wherein, α IOU Shape Loc Class Col Tex The corresponding values can be set according to actual conditions to give each dimension a corresponding degree of attention or importance.

[0161] In actual application, when the target detection result represents that the comprehensive matching degree sim exceeds the target matching degree (which can be set according to actual conditions), it is considered that the first target and the second target are the same target; when the target detection result represents that the comprehensive matching degree sim is lower than the target matching degree (which can be set according to actual conditions), it is considered that the first target and the second target are not the same target.

[0162] In the above manner, the matching degrees of the first detection box and the projected detection box in multiple dimensions and the weights of the matching degrees in each dimension are comprehensively considered, which can improve the degree of attention to important dimensions, so that the comprehensive matching degree obtained finally can better reflect whether the first target and the second target are the same target, and the accuracy of target detection is improved.

[0163] In some embodiments, the above step 104 can be implemented in the following manner: for the matching degree of each dimension in the N dimensions, the matching degree is compared with a preset matching degree threshold, and based on the comparison result, an initial detection result of whether the first target and the second target in the corresponding dimension are the same is determined; the first number of initial detection results indicating that the first target and the second target in the corresponding dimension are the same, and the second number of initial detection results indicating that the first target and the second target in the corresponding dimension are different are counted; based on the first number and the second number, a target detection result of whether the first target and the second target are the same is determined.

[0164] ​​​​​​​​​​As an example, after obtaining the region matching degree (simIOU), shape matching degree (simShape), location matching degree (simLoc), category matching degree (simClass), color matching degree (simCol) and texture matching degree (simTex) of the first detection box and the projected detection box, the matching degrees of each dimension can be compared with the corresponding preset matching degree threshold (the matching degree threshold corresponding to each dimension can be set according to actual conditions, and the matching degree thresholds corresponding to each dimension can be the same or different), such as comparing the region matching degree (simIOU) with the region matching degree threshold to obtain the initial detection result corresponding to the region dimension (that is, judging whether the first target and the second target are the same target from the region dimension), such as when the region matching degree (simIOU) exceeds the region matching degree threshold, the initial detection result indicates that the first target and the second target are the same target from the region dimension.

[0165] In the above manner, the initial detection results of other dimensions can be counted, such as the initial detection result corresponding to the shape dimension, the initial detection result corresponding to the location dimension, the initial detection result corresponding to the category dimension, the initial detection result corresponding to the color dimension, and the initial detection result corresponding to the texture dimension. The first quantity indicating that the first target and the second target are the same in the corresponding dimension and the second quantity indicating that the first target and the second target are different in the corresponding dimension are counted according to the initial detection results of each dimension, such as assuming that the initial detection results corresponding to the region dimension, the shape dimension, the location dimension, the category dimension and the color dimension all indicate that the first target and the second target are the same, and the initial detection result corresponding to the texture dimension indicates that the first target and the second target are different, then the first quantity can be determined as 5 and the second quantity can be determined as 1. When determining the final target detection result based on the first quantity and the second quantity, the initial detection result corresponding to the largest quantity can be selected as the final target detection result, such as the initial detection result indicating that the first target and the second target are the same is selected as the target detection result, that is, the final target detection result indicates that the first target and the second target are the same target.

[0166] In the above manner, the matching degrees of the first detection box and the projected detection box in multiple dimensions are comprehensively considered, and the final target detection result is determined by voting, which can avoid the problem that the final detection target is inaccurate due to occasional misjudgment, and improves the accuracy of target detection.

[0167] In the following, an exemplary application of the embodiments of the present application in the practical application scenario of intelligent driving of a vehicle will be described. In order to improve the accuracy and reliability of target detection, the embodiments of the present application match the camera detection result (i.e., the two-dimensional detection result obtained by performing target detection on the target image) and the radar detection result (i.e., the three-dimensional detection result obtained by performing target detection on the radar point cloud data corresponding to the target image), which can realize consistency of the target in two-dimensional and three-dimensional spaces, filter three-dimensional false detection, and solve the problem that two-dimensional detection does not have target depth and size information. In addition, in the actual matching process, considering the following possible situations: the motion of the ego vehicle, the motion of the target, the inaccuracy of the external parameter calibration, or the situation that the radar and camera time stamps cannot be completely aligned, which can cause that simply using the region dimension (i.e., IOU) cannot match the camera detection result and the radar detection result.

[0168] To this end, the embodiments of the present application provide an improved matching method, which matches from N (greater than or equal to 2) dimensions, and determines the final target detection result based on the matching result of the N dimensions. Next, taking the region dimension, the position dimension, the shape dimension, and the category dimension as examples, the target detection method provided by the embodiments of the present application will be described.

[0169] Next, the target detection, point cloud projection, and detection result fusion strategy will be described one by one.

[0170] 1. Target detection

[0171] 1.1. Two-dimensional target detection

[0172] When performing first target detection on the two-dimensional target image, the checkerboard calibration method can be used to calibrate the camera for collecting the target image to obtain the intrinsic parameters of the camera, so as to correct the distortion of the collected target image according to the intrinsic parameters of the camera to obtain a non-distorted target image, and input the obtained non-distorted target image into the trained two-dimensional target detection model (such as a model that can be trained based on YOLO, SSD, etc.) to perform first target detection on the non-distorted target image through the target detection model to obtain a two-dimensional detection result.

[0173] As shown in FIG. 4, the two-dimensional detection result includes a recognized first target (i.e., the category of the first target) and a first detection box, wherein the number of the first target can be one or more, and the number of the first detection box is also one or more. The first detection box carries position information (x, y, w, h) and a confidence (only the confidence is shown in FIG. 4), wherein x and y refer to the offset of the center point of the first detection box relative to the upper left corner of the grid cell, w and h represent the width and height of the first detection box, and the confidence is used to indicate the probability that the first target exists in the first detection box. When the confidence is greater than a confidence threshold, it is considered that the first target exists in the first detection box.

[0174] 1.2, Three-dimensional target detection

[0175] Firstly, point cloud data corresponding to the target image needs to be acquired (referring to the time and position of the camera collecting the target image, which corresponds to the time and position of the radar collecting the point cloud data, that is, the target image and the point cloud data corresponding to the target image are data from the same time and the same position) ; then, the point cloud data is input into the trained three-dimensional target detection model, and a three-dimensional detection result is output.

[0176] As shown in FIG. 5, after the second target detection processing on the point cloud data, the output visualized three-dimensional detection result includes the recognized second target (or the category of the second target) and the second detection box (that is, the three-dimensional detection box, which has a shape such as a cuboid or a cube), wherein the number of the second target can be one or more, and the number of the second detection box is also one or more. The second detection box carries position information (x, y, z, l, w, h, R), wherein (x, y, z) refers to the three-dimensional coordinates of the center point of the second detection box in the radar coordinate system, (l, w, h) represents the length, width and height of the second detection box, R is a rotation matrix, and different shapes of the second detection box represent different categories corresponding to the corresponding second target.

[0177] 2. Point cloud projection

[0178] 2.1, Project the point cloud data onto the image.

[0179] When projecting the point cloud data onto the image, the camera intrinsic parameters need to be calibrated first, and then the image is rectified based on the camera intrinsic parameters to obtain an undistorted image. Then, the extrinsic parameters between the camera and the radar are calibrated based on the undistorted image. Finally, the point cloud data and the target image at the same time are acquired, and the point cloud data is projected onto the target image. The visual effect of the projection is shown in FIG. 10, which is a projection diagram of point cloud data provided by an embodiment of the present application. In FIG. 10, the colored horizontal stripes represent the projection of the sensing points in the point cloud data onto the target image, and different colors represent different distances of the target, such as color 1 (such as orange) representing a closer target and color 2 (such as blue) representing a farther target.

[0180] Specifically, the projection method is as follows: given the sensing point coordinates P = (X, Y, Z, 1) in the point cloud, and the extrinsic parameters C_T_L between the radar coordinate system and the camera coordinate system, the camera coordinates of the sensing point in the point cloud can be obtained by using formula (9): C = (X C ,Y C ,Z C ) = C_T_L * P (9)

[0181] According to the conversion formula of camera coordinates to pixel coordinates shown in formula (10), the projection (u, v) of the sensing point in the point cloud on the target image can be obtained:

[0182] where (u, v) represents the pixel coordinates, (X C ,Y C ,Z C ) represents the camera coordinates, fx, f y , u0, v0 are camera intrinsic parameters.

[0183] 2.2, Project the three-dimensional detection result into the target image.

[0184] The projection of the three-dimensional detection result into the target image is essentially the projection of the second detection box in the three-dimensional detection result into the target image, which carries position information (x, y, z, l, w, h, R) on the second detection box, where (x, y, z) refers to the three-dimensional coordinates of the center point of the second detection box in the radar coordinate system, (l, w, h) represents the length, width and height of the second detection box, and R is the rotation matrix. The second detection box can be regarded as a cube including 8 corner points, and the positive direction of the xyz axis of the point cloud coordinate system is shown by the arrow in FIG. 6. According to the position information (x, y, z, l, w, h, R) carried on the second detection box, the three-dimensional coordinates of the 8 corner points of the second detection box can be calculated by formula (2).

[0185] Given the extrinsic parameters C_T_L of the radar coordinate system and the camera coordinate system, the projection method in section 2.1 is used to project the 8 corner points of the second detection box onto the target image, and after obtaining the 8 projected pixel coordinates, the minimum horizontal coordinate u min , the maximum horizontal coordinate u max , the minimum vertical coordinate v min and the maximum vertical coordinate v max are selected from the 8 projected pixel coordinates. The projected detection box obtained after projection can be represented as (u min , v min , u max , v max ), as shown in FIG. 7. After projecting the three-dimensional detection result obtained by the second target detection on the point cloud data (including multiple second detection boxes) onto the target image, the projection detection box corresponding to each second detection box is obtained. The projection detection box is represented by a dashed box, and the solid line box in it is used to represent the first detection box obtained by the first target detection on the target image.

[0186] In the above manner, the second detection box obtained by performing a second target detection on the point cloud data is projected onto the target image to obtain a projected detection box. Since the projected detection box and the first detection box obtained by performing a first target detection on the target image are in the same dimension, this makes the three-dimensional detection result and the two-dimensional detection result matchable.

[0187] 3. Detection Result Fusion Strategy

[0188] The first detection box and the projected detection box are matched in the region dimension, position dimension, shape dimension and category dimension respectively to obtain four matching degrees, and the final target detection result is determined based on the four matching results.

[0189] 3.1 Regional Matching Degree

[0190] As shown in Figure 8, let the first detection box be represented by A and the projected detection box by B. First, the first intersection region (A∩B) between the first detection box and the projected detection box is obtained. Then, the first union region (A∪B) between the first detection box and the projected detection box is obtained. Then, the first intersection region (A∩B) and the first union region (A∪B) are divided to obtain the region matching degree between the first detection box and the projected detection box. That is, the region matching degree (simIOU) can be expressed as: simIOU=(A∩B) / (A∪B).

[0191] 3.2 Location matching degree

[0192] First, the ratio of the average width and height of the first detection box and the projected detection box in the target image is calculated using the formula (3) above.

[0193] Then, the top-left and bottom-right corner points of the first detection box are used to uniquely define a detection box. The positioning coordinates of this first detection box can then be expressed as: ((x...) min ,y min ),(x max ,y max The positioning coordinates of the projection detection frame can be represented by the coordinates of its four corner points. For example, the positioning coordinates u of the projection detection frame can be represented as: u = (u1, u2, u3, u4), where u1 = (u2, u3, u4) 1x ,u 1y ), u2=(u 2x ,u 2y ), u3=(u 3x ,u 3y ), u4=(u 4x ,u 4y ), u 1x and u 1y This represents the coordinates of corner point u1 on the x and y axes, u 2x and u 2ydenotes the coordinates of the corner point u2 in the x and y axes, u 3x denotes the coordinates of the corner point u3 in the x and y axes, u 3y denotes the coordinates of the corner point u4 in the x and y axes. 4x denotes the coordinates of the corner point u4 in the x and y axes. 4y denotes the coordinates of the corner point u4 in the x and y axes.

[0194] Then, the coordinate difference of each corner point of the projected detection frame and the first detection frame in the x and y axes is calculated respectively, for example, the coordinate difference of the corner point u1 and the first detection frame in the x and y axes can be calculated by the above formula (4), and in the same way, the coordinate difference of the corner point u2 and the first detection frame in the x and y axes can be determined as du 2x and du 2y , the coordinate difference of the corner point u3 and the first detection frame in the x and y axes can be determined as du 3x and du 3y , and the coordinate difference of the corner point u4 and the first detection frame in the x and y axes can be determined as du 4x and du 4y .

[0195] Then, the average coordinate difference of each corner point is calculated by the above formula (5), that is, the average coordinate difference in the x axis (referred to as the average horizontal coordinate difference meandx) is obtained by averaging du 1x , du 2x , du 3x and du 4x , and the average coordinate difference in the y axis (referred to as the average vertical coordinate difference meandy) is obtained by averaging du 1y , du 2y , du 3y and du 4y .

[0196] Then, based on the positioning coordinates, the average coordinate difference is normalized to obtain the distance between the first detection frame and the projected detection frame, for example, the distance between the first detection frame and the projected detection frame square_norm can be obtained by the above formula (6) normx 2 / x_std_dev 2 +normy 2 / y_std_dev 2 , wherein the horizontal coordinate normalization result normx = meanx / (x max -x min ), and the vertical coordinate normalization result normy = meany / (y max -y min), x_std_dev is the first weight of the horizontal coordinate normalization result, y_std_dev is the second weight of the vertical coordinate normalization result, and the first weight and the second weight can be set according to actual conditions, for example, according to experience, x_std_dev can be set as 0.4 and y_std_dev can be set as 0.5.

[0197] After the distance square_norm between the first detection box and the projection detection box and the proportion ratio of the average width and height of the first detection box and the projection detection box in the target image are determined, the product of the two is taken as a reference value for chi-square distribution table query, that is, the reference value dist=square_norm*ratio.

[0198] Next, the position matching degree of the two detection boxes is calculated by using the chi-square test method, a chi-square distribution table table is prepared, it is assumed that the number of values in the chi-square distribution table is table_size, the chi-square distribution table contains chi-square values and corresponding probability values, in subsequent table lookup, the target probability value can be queried from the constructed chi-square distribution table based on the reference value dist, and the specific lookup method is as follows:

[0199] step=0.050000; dist_ind=dist / step, where the type of dist_ind is floating point; dist_int=int(dist_ind), where dist_int is the integer part of dist_ind; w=dist_ind-dist_int; if dist_ind>=table_size-1, the position matching degree simLoc between the first detection box and the projection detection box is 0; otherwise, simLoc=1-(table[dist_int]*(1-w)+table[dist_int+1]*w), where table[dist_int] represents the target probability value corresponding to the dist_int number in the chi-square distribution table, and simLoc represents the position matching degree between the first detection box and the projection detection box.

[0200] For example, it is assumed that dist conforms to a Gaussian distribution, dist=2.3, step=0.05, and table_size=400, then dist_ind=46, dist_int=46, w=0, and the queried table[dist_int]=0.683363, then the position matching degree simLoc between the first detection box and the projection detection box can be calculated by using formula (7) as follows: simLoc=1-0.683363=0.316637.

[0201] By the above-mentioned manner, the position matching degree of the first detection frame and the projection detection frame is calculated by using the chi-square test method, so that the accuracy of target detection can be improved.

[0202] 3.3, shape matching degree

[0203] As shown in FIG. 9, the first detection frame is represented by C, and the projection detection frame is represented by D. When the shape dimension of the first detection frame and the projection detection frame is matched, the center points of the first detection frame C and the projection detection frame D need to be moved to the same position (i.e., center alignment). After the center alignment, the second intersection area (C∩D) between the first detection frame and the projection detection frame is obtained, and then the second union area (C∪D) between the first detection frame and the projection detection frame is obtained. Then, the second intersection area (C∩D) is divided by the second union area (C∪D) to obtain the shape matching degree between the first detection frame and the projection detection frame, i.e., the shape matching degree (simShape) can be represented as: simShape=(C∩D) / (C∪D).

[0204] 3.4, class matching degree

[0205] Firstly, it is assumed that the first classes obtained by performing the first target detection on the target image are respectively: pedestrian, car, truck, motorcycle, bicycle, traffic pyramid, and trailer, and the second classes obtained by performing the second target detection on the point cloud data are respectively: pedestrian, car, truck, motorcycle, bicycle, traffic pyramid, and other vehicles. Then, the similarity table based on experience is constructed, as shown in Table 1 above.

[0206] Then, after obtaining the class of the first target (i.e., the class corresponding to the first detection frame) by performing the first target detection on the target image, and obtaining the class of the second target (i.e., the class corresponding to the second detection frame) by performing the second target detection on the point cloud data corresponding to the target image, the class similarity is directly queried in Table 1 based on the class corresponding to the first detection frame and the class corresponding to the projection detection frame, as the class matching degree (simClass) between the first detection frame and the projection detection frame. For example, if the class corresponding to the first detection frame is pedestrian, and the class corresponding to the projection detection frame is also pedestrian, then the class matching degree between the first detection frame and the projection detection frame is 1.

[0207] 3.5, matching degree fusion

[0208] In the calculation of the region matching degree (simIOU), shape matching degree (simShape), location matching degree (simLoc), and class matching degree (simClass) of the first detection frame and the projected detection frame, the final matching needs to comprehensively calculate the comprehensive matching degree of the above four matching degrees. For example, the comprehensive matching degree is calculated by formula (11): sim = (simIOU + (simLoc + simShape + simClass) / 3) / 2 (11)

[0209] 3.6, matching judgment

[0210] Suppose that K (positive integer) second detection frames are detected after the second target detection on the point cloud data, the K second detection frames are projected into the target image to obtain K projected detection frames; then, the comprehensive matching degree between each projected detection frame and the first detection frame obtained by the first target detection on the target image is calculated in the above manner. If the comprehensive matching degree exceeds the matching degree threshold (which can be set according to actual needs, such as 0.2), it is considered that the projected detection frame matches the first detection frame (i.e., the second target corresponding to the projected detection frame and the first target corresponding to the first detection frame are the same target). If the comprehensive matching degree between one projected detection frame and multiple first detection frames exceeds the matching degree threshold, it is considered that the projected detection frame only matches the first detection frame with the largest comprehensive matching degree. If the comprehensive matching degree is lower than the matching degree threshold, it is considered that the projected detection frame does not match the first detection frame (i.e., the second target corresponding to the projected detection frame and the first target corresponding to the first detection frame are not the same target).

[0211] By the above method, target detection is performed on the target image and the point cloud data corresponding to the target image, the first detection frame obtained by detecting the target image and the projected detection frame obtained based on the point cloud data are matched in multiple dimensions, and the final target detection result is determined in combination with the matching results in multiple dimensions. In this way, various types of information in the target image and the point cloud data can be fully utilized, and the accuracy and reliability of the final detection result can be improved by matching in multiple dimensions.

[0212] So far, the target detection method provided by the embodiments of the present application has been described in combination with the exemplary application and implementation of the electronic device provided by the embodiments of the present application. The cooperation of each module in the target detection device 555 provided by the embodiments of the present application to implement the target detection scheme will be described below.

[0213] The detection module 5551 is configured to perform first target detection on a target image to obtain a first target and a first detection box corresponding to the first target, and perform second target detection on point cloud data corresponding to the target image to obtain a second target and a second detection box corresponding to the second target. The projection module 5552 is configured to project the second detection box onto the target image to obtain a projection detection box of the second detection box on the target image. The matching module 5553 is configured to perform matching in N dimensions on the first detection box and the projection detection box, to obtain matching degrees of the first detection box and the projection detection box in the N dimensions, where N is an integer greater than or equal to 2. The determination module 5554 is configured to determine, based on the matching degrees of the first detection box and the projection detection box in the N dimensions, a target detection result of whether the first target and the second target are the same.

[0214] In some embodiments, the matching module is further configured to perform at least two of the following matching operations on the first detection box and the projection detection box: performing matching in a region dimension on the first detection box and the projection detection box to obtain a region matching degree between the first detection box and the projection detection box; performing matching in a shape dimension on the first detection box and the projection detection box to obtain a shape matching degree between the first detection box and the projection detection box; performing matching in a position dimension on the first detection box and the projection detection box to obtain a position matching degree between the first detection box and the projection detection box; performing matching in a category dimension on the first target and the second target to obtain a category matching degree between the first detection box and the projection detection box; performing matching in a color dimension on the first target and the second target to obtain a color matching degree between the first detection box and the projection detection box; and performing matching in a texture dimension on the first target and the second target to obtain a texture matching degree between the first detection box and the projection detection box.

[0215] In some embodiments, the matching module is further configured to obtain a first intersection region and a first union region between the first detection box and the projection detection box; and determine, as a region matching degree between the first detection box and the projection detection box, a ratio between the first intersection region and the first union region.

[0216] In some embodiments, the matching module is further configured to perform center alignment on the first detection box and the projection detection box; obtain a second intersection region between the first detection box and the projection detection box after the center alignment, and obtain a second union region between the first detection box and the projection detection box after the center alignment; and determine, as a shape matching degree between the first detection box and the projection detection box, a ratio between the second intersection region and the second union region.

[0217] In some embodiments, the matching module is further configured to determine a distance between the first bounding box and the projected bounding box, and a proportion of an average of a width and a height of the first bounding box and the projected bounding box in the target image; take a product of the distance and the proportion as a reference value for a chi-square distribution table query, and query a target probability value from a constructed chi-square distribution table based on the reference value; obtain a mapping relationship between the target probability value and the position matching degree, and determine the position matching degree between the first bounding box and the projected bounding box based on the target probability value and the mapping relationship.

[0218] In some embodiments, the matching module is further configured to determine a positioning coordinate of the first bounding box, and determine a coordinate difference value between the positioning coordinate and an angle point coordinate of each angle point of the projected bounding box; average the coordinate difference value corresponding to each angle point to obtain an average coordinate difference value; normalize the average coordinate difference value based on the positioning coordinate to obtain the distance between the first bounding box and the projected bounding box.

[0219] In some embodiments, the average coordinate difference value includes an average horizontal coordinate difference value and an average vertical coordinate difference value; the matching module is further configured to obtain a first difference value between a maximum horizontal coordinate and a minimum horizontal coordinate in the positioning coordinate, and a second difference value between a maximum vertical coordinate and a minimum vertical coordinate; take a ratio of the average horizontal coordinate difference value and the first difference value as a horizontal coordinate normalization result, and take a ratio of the average vertical coordinate difference value and the second difference value as a vertical coordinate normalization result; obtain a first weight of the horizontal coordinate normalization result and a second weight of the vertical coordinate normalization result, and fuse the horizontal coordinate normalization result and the vertical coordinate normalization result based on the first weight and the second weight to obtain the distance between the first bounding box and the projected bounding box.

[0220] In some embodiments, the matching module is further configured to obtain a corresponding relationship among a first category of a target in an image, a second category of the target in point cloud data, and a target similarity, wherein the target similarity is a similarity of the first category and the second category; determine the category of the first target and the category of the second target, and determine the category matching degree between the first bounding box and the projected bounding box based on the category of the first target, the category of the second target, and the corresponding relationship.

[0221] In some embodiments, the matching module is further configured to obtain color information in the second target, map the color information to a two-dimensional color space to obtain a mapped feature, perform color coding on the mapped feature to obtain a coded feature, and perform image conversion on the coded feature to obtain a first converted image corresponding to the second target; perform color extraction on the first target to obtain a first color feature, and perform color extraction on the first converted image to obtain a second color feature; and perform matching on the first color feature and the second color feature to obtain a color matching degree between the first detection box and the projection detection box.

[0222] In some embodiments, the matching module is further configured to obtain texture information in the second target, and perform image conversion on the texture information to obtain a second converted image corresponding to the second target; perform texture extraction on the first target to obtain a first texture feature, and perform texture extraction on the second converted image to obtain a second texture feature; and perform matching on the first texture feature and the second texture feature to obtain a texture matching degree between the first detection box and the projection detection box.

[0223] In some embodiments, the determining module is further configured to determine a weight of the matching degree of each of the N dimensions, and perform weighted summation on the matching degrees of the first detection box and the projection detection box in the N dimensions based on the weight of the matching degree of each of the N dimensions to obtain a target detection result of whether the first target and the second target are the same.

[0224] In some embodiments, the determining module is further configured to, for the matching degree of each of the N dimensions, compare the matching degree with a preset matching degree threshold, and determine an initial detection result of whether the first target and the second target are the same in the corresponding dimension based on a comparison result; count a first number of the initial detection results indicating that the first target and the second target are the same in the corresponding dimension, and a second number of the initial detection results indicating that the first target and the second target are different in the corresponding dimension; and determine a target detection result of whether the first target and the second target are the same based on the first number and the second number.

[0225] In some embodiments, the detection module is further configured to perform sparse conversion on point cloud data corresponding to the target image to obtain a plurality of voxels corresponding to the point cloud data, perform mapping on each voxel to obtain a mapping image corresponding to each voxel, and perform second target detection on the mapping image corresponding to each voxel to obtain a second target and a second detection box corresponding to the second target.

[0226] In some embodiments, the projection module is further configured to determine a conversion relationship between a radar coordinate system and a camera coordinate system, wherein the radar coordinate system is a coordinate system of a radar that collects the point cloud data, and the camera coordinate system is a coordinate system of a camera that collects the target image; and project the second detection box onto the target image based on the conversion relationship to obtain a projection detection box of the second detection box on the target image.

[0227] In some embodiments, the second detection box includes a plurality of corner points, and the projection module is further configured to obtain position information of the second detection box in the radar coordinate, and perform projection transformation on each of the corner points based on the position information and the conversion relationship to obtain pixel coordinates of each of the corner points on the target image; filter a minimum horizontal coordinate, a maximum horizontal coordinate, a minimum vertical coordinate, and a maximum vertical coordinate from the pixel coordinates of each of the corner points on the target image; and determine the projection detection box of the second detection box on the target image based on a detection box composed of the minimum horizontal coordinate, the maximum horizontal coordinate, the minimum vertical coordinate, and the maximum vertical coordinate.

[0228] Embodiments of the present application provide a computer program product, which includes a computer program or computer executable instructions stored in a computer readable storage medium. A processor of an electronic device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions to cause the electronic device to perform the target detection method described above.

[0229] Embodiments of the present application provide a computer readable storage medium storing computer executable instructions, wherein the computer executable instructions or computer programs are stored in the computer readable storage medium. When the computer executable instructions or computer programs are executed by a processor, the processor will execute the target detection method provided by the embodiments of the present application, for example, the target detection method shown in FIG. 3A.

[0230] In some embodiments, the computer readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or various devices including one or any combination of the above memories.

[0231] In some embodiments, the computer executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or being deployed as modules, components, subroutines or other units suitable for use in a computing environment.

[0232] By way of example, computer-executable instructions can include but are not limited to: a) program modules that are stored internally, b) program modules that are filed in a storage device such as a zip drive, c) program modules that include scripts and other code material, d) program modules that are accessible via a communications network, and e) documents created by another program module and used as input to an executing program, etc.

[0233] By way of example, computer-executable instructions can be deployed to be executed on one electronic device or on multiple electronic devices that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0234] The above description is only for the preferred embodiments of the present application, and not intended to limit the scope of the present application. Any modification, equivalent replacement and improvement made within the spirit and scope of the present application shall fall into the scope of the present application.

Claims

1. A target detection method characterized by, The method comprises: performing first target detection on a target image to obtain a first target and a first detection box corresponding to the first target, and performing second target detection on point cloud data corresponding to the target image to obtain a second target and a second detection box corresponding to the second target; projecting the second detection box onto the target image to obtain a projection detection box of the second detection box on the target image; performing N-dimensional matching on the first detection box and the projection detection box to obtain matching degrees of the first detection box and the projection detection box in N dimensions, N being an integer greater than or equal to 2; determining, based on the matching degrees of the first detection box and the projection detection box in N dimensions, a target detection result of whether the first target and the second target are the same.

2. The method of claim 1, wherein, The N-dimensional matching of the first detection box and the projection detection box comprises: performing at least two of the following matching operations on the first detection box and the projection detection box: performing region dimension matching on the first detection box and the projection detection box to obtain a region matching degree between the first detection box and the projection detection box; performing shape dimension matching on the first detection box and the projection detection box to obtain a shape matching degree between the first detection box and the projection detection box; performing position dimension matching on the first detection box and the projection detection box to obtain a position matching degree between the first detection box and the projection detection box; performing category dimension matching on the first target and the second target to obtain a category matching degree between the first detection box and the projection detection box; performing color dimension matching on the first target and the second target to obtain a color matching degree between the first detection box and the projection detection box; performing texture dimension matching on the first target and the second target to obtain a texture matching degree between the first detection box and the projection detection box.

3. The method of claim 2, wherein, The region dimension matching of the first detection box and the projection detection box comprises: obtaining a first intersection region and a first union region between the first detection box and the projection detection box; determining, as a region matching degree between the first detection box and the projection detection box, a ratio between the first intersection region and the first union region.

4. The method of claim 2, wherein, The shape dimension matching of the first detection box and the projection detection box comprises: performing center alignment on the first detection box and the projection detection box; obtaining a second intersection region between the first detection box and the projection detection box after center alignment, and obtaining a second union region between the first detection box and the projection detection box after center alignment; determining, as a shape matching degree between the first detection box and the projection detection box, a ratio between the second intersection region and the second union region.

5. The method of claim 2, wherein, The position dimension matching of the first detection frame and the projection detection frame is performed to obtain a position matching degree between the first detection frame and the projection detection frame, including: determining a distance between the first detection frame and the projection detection frame, and a proportion of an average value of a width and a height of the first detection frame and the projection detection frame in the target image; multiplying the distance and the proportion as a reference value for chi-square distribution table query, and querying a target probability value from a constructed chi-square distribution table based on the reference value; obtaining a mapping relationship between the target probability value and the position matching degree, and determining the position matching degree between the first detection frame and the projection detection frame based on the target probability value and the mapping relationship.

6. The method of claim 5, wherein, The distance between the first detection frame and the projection detection frame is determined, including: determining a positioning coordinate of the first detection frame, and determining a coordinate difference value between each corner point coordinate of the projection detection frame and the positioning coordinate; averaging each coordinate difference value corresponding to each corner point to obtain an average coordinate difference value; normalizing the average coordinate difference value based on the positioning coordinate to obtain the distance between the first detection frame and the projection detection frame.

7. The method of claim 6, wherein, The average coordinate difference value includes an average horizontal coordinate difference value and an average vertical coordinate difference value; The distance between the first detection frame and the projection detection frame is determined based on the positioning coordinate, including: obtaining a first difference value between a maximum horizontal coordinate and a minimum horizontal coordinate in the positioning coordinate, and a second difference value between a maximum vertical coordinate and a minimum vertical coordinate; taking a ratio of the average horizontal coordinate difference value and the first difference value as a horizontal coordinate normalization result, and taking a ratio of the average vertical coordinate difference value and the second difference value as a vertical coordinate normalization result; obtaining a first weight of the horizontal coordinate normalization result and a second weight of the vertical coordinate normalization result, and fusing the horizontal coordinate normalization result and the vertical coordinate normalization result based on the first weight and the second weight to obtain the distance between the first detection frame and the projection detection frame.

8. The method of claim 2, wherein, The class dimension matching of the first target and the second target is performed to obtain a class matching degree between the first detection frame and the projection detection frame, including: obtaining a corresponding relationship among a first class of a target in an image, a second class of the target in point cloud data, and a target similarity, wherein the target similarity is a similarity of the first class and the second class; determining a class of the first target and a class of the second target, and determining the class matching degree between the first detection frame and the projection detection frame based on the class of the first target, the class of the second target, and the corresponding relationship.

9. The method of claim 2, wherein, The color dimension matching of the first target and the second target is performed to obtain a color matching degree between the first detection frame and the projection detection frame, including: obtaining color information in the second target, and mapping the color information to a two-dimensional color space to obtain a mapping feature; Color coding is performed on the mapping feature to obtain an encoded feature, and image conversion is performed on the encoded feature to obtain a first converted image corresponding to the second target; Color extraction is performed on the first target to obtain a first color feature, and color extraction is performed on the first converted image to obtain a second color feature; The first color feature and the second color feature are matched to obtain a color matching degree between the first detection box and the projection detection box.

10. The method of claim 2, wherein, The texture dimension matching of the first target and the second target is performed to obtain a texture matching degree between the first detection box and the projection detection box, including: Texture information in the second target is obtained, and image conversion is performed on the texture information to obtain a second converted image corresponding to the second target; Texture extraction is performed on the first target to obtain a first texture feature, and texture extraction is performed on the second converted image to obtain a second texture feature; The first texture feature and the second texture feature are matched to obtain a texture matching degree between the first detection box and the projection detection box.

11. The method of claim 1, wherein, The target detection result of whether the first target and the second target are the same is determined based on the matching degrees of the first detection box and the projection detection box in N dimensions, including: The weight of the matching degree of each dimension in N dimensions is determined; The matching degrees of the first detection box and the projection detection box in N dimensions are weighted and summed based on the weights of the matching degrees of each dimension in N dimensions to obtain the target detection result of whether the first target and the second target are the same.

12. The method of claim 1, wherein, The target detection result is determined based on the matching degrees of the first detection box and the projection detection box in N dimensions, including: For the matching degree of each dimension in N dimensions, the matching degree is compared with a preset matching degree threshold, and an initial detection result of whether the first target and the second target are the same in the corresponding dimension is determined based on the comparison result; The first number of the initial detection result indicating that the first target and the second target are the same in the corresponding dimension is counted, and the second number of the initial detection result indicating that the first target and the second target are different in the corresponding dimension is counted; Based on the first number and the second number, the target detection result of whether the first target and the second target are the same is determined.

13. The method of claim 1, wherein, The second target detection is performed on the point cloud data corresponding to the target image to obtain a second target and a second detection box corresponding to the second target, including: Sparse conversion is performed on the point cloud data corresponding to the target image to obtain a plurality of voxels corresponding to the point cloud data; Mapping is performed on each voxel to obtain a mapping image corresponding to each voxel; Second target detection is performed on the mapping image corresponding to each voxel to obtain a second target and a second detection box corresponding to the second target.

14. The method of claim 1, wherein, The second detection box is projected onto the target image to obtain a projection detection box of the second detection box on the target image, including: determine a conversion relationship between a radar coordinate system and a camera coordinate system, wherein the radar coordinate system is a coordinate system of a radar collecting the point cloud data, and the camera coordinate system is a coordinate system of a camera collecting the target image; project the second detection box onto the target image based on the conversion relationship to obtain a projection detection box of the second detection box on the target image.

15. The method of claim 14, wherein, The second detection box includes a plurality of corner points, and the projection of the second detection box onto the target image based on the conversion relationship to obtain a projection detection box of the second detection box on the target image includes: obtaining position information of the second detection box in the radar coordinate, and performing projection transformation on each of the corner points based on the position information and the conversion relationship to obtain pixel coordinates of each of the corner points on the target image; filtering out a minimum horizontal coordinate, a maximum horizontal coordinate, a minimum vertical coordinate, and a maximum vertical coordinate from the pixel coordinates of each of the corner points on the target image; determining the projection detection box of the second detection box on the target image based on a detection box composed of the minimum horizontal coordinate, the maximum horizontal coordinate, the minimum vertical coordinate, and the maximum vertical coordinate.

16. A target detection apparatus characterized by comprising: The device includes: a detection module configured to perform first target detection on a target image to obtain a first target and a first detection box corresponding to the first target, and perform second target detection on point cloud data corresponding to the target image to obtain a second target and a second detection box corresponding to the second target; a projection module configured to project the second detection box onto the target image to obtain a projection detection box of the second detection box on the target image; a matching module configured to perform N-dimensional matching on the first detection box and the projection detection box to obtain matching degrees of the first detection box and the projection detection box in N dimensions, N being an integer greater than or equal to 2; a determination module configured to determine a target detection result of whether the first target and the second target are the same based on the matching degrees of the first detection box and the projection detection box in N dimensions.

17. An electronic device, comprising: includes: a memory configured to store computer executable instructions or computer programs; a processor configured to execute the computer executable instructions or computer programs stored in the memory to implement the target detection method of any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that, The computer executable instructions or computer programs are stored in the memory and executed by the processor to implement the target detection method of any one of claims 1 to 15.

19. A computer program product comprising computer programs or computer executable instructions, characterized in that, The computer executable instructions or computer programs are executed by the processor to implement the target detection method of any one of claims 1 to 15.

Images