Object recognition method, device, storage medium and processor

Abstract

The present invention discloses an object recognition method, device, storage medium and processor. The method comprises: obtaining multiple detection frames of an object in a video, wherein each detection frame is used to indicate the position of the object in the video; determining the time frame number of the multiple detection frames in the video respectively, wherein the time frame number is used to characterize the time length of each detection frame appearing continuously in the video; determining a target detection frame in the multiple detection frames based on the time frame number and the frame number difference threshold; and recognizing the object in the video based on the target detection frame. The present invention solves the technical problem of low accuracy of object recognition.

Description

Technical Field

[0001] The present invention relates to the field of vehicles, and in particular to an object recognition method, device, storage medium and processor. Background Art

[0002] At present, in the process of autonomous driving, based on the results of detecting objects on traffic roads, the position, size, direction, category and other information of the objects in space can be accurately determined, thereby achieving the purposes of three-dimensional modeling and path planning.

[0003] In the related art, for the problem of object detection, a detection method of non-maximum suppression (NMS) is usually adopted. However, this method is only applicable to detection in two-dimensional scenes. Therefore, for spatial detection, there is a technical problem of low accuracy in object recognition.

[0004] With respect to the problem of low accuracy of object recognition in the above-mentioned related technologies, no effective solution has been proposed so far. Summary of the invention

[0005] The embodiments of the present invention provide a method, device, storage medium and processor for identifying an object, so as to at least solve the technical problem of low accuracy in target identification.

[0006] According to one aspect of an embodiment of the present invention, a method for identifying an object is provided, comprising: obtaining multiple detection frames of an object in a video, wherein each detection frame is used to indicate a position of the object in the video; respectively determining the number of time frames of the multiple detection frames in the video, wherein the number of time frames is used to characterize the length of time each detection frame appears continuously in the video; determining a target detection frame among the multiple detection frames based on the number of time frames and a frame difference threshold; and identifying the object in the video based on the target detection frame.

[0007] Optionally, based on the number of time frames and a frame number difference threshold, determining a target detection frame in multiple detection frames, including: determining an intersection-and-union ratio of a first detection frame in the multiple detection frames and a second detection frame in the multiple detection frames, wherein the first detection frame and the second detection frame are any two detection frames in the multiple detection frames at the same time; in response to the intersection-and-union ratio being greater than the intersection-and-union ratio threshold, obtaining the number of time frames of the first detection frame and the number of time frames of the second detection frame; based on the number of time frames of the first detection frame, the number of time frames of the second detection frame and the frame number difference threshold, determining the target detection frame in the first detection frame and the second detection frame.

[0008] Optionally, based on the time frame number of the first detection frame, the time frame number of the second detection frame and a frame difference threshold, determining a target detection frame from the first detection frame and the second detection frame includes: determining a first frame difference between the time frame number of the first detection frame and the time frame number of the second detection frame; based on the first frame difference and the frame difference threshold, determining the target detection frame from the first detection frame and the second detection frame.

[0009] Optionally, based on the first frame number difference and the frame number difference threshold, a target detection frame is determined from the first detection frame and the second detection frame, including: in response to the absolute value of the first frame number difference being greater than the frame number difference threshold, determining the detection frame with the longer time frame number between the first detection frame and the second detection frame as the target detection frame.

[0010] Optionally, based on the first frame number difference and the frame number difference threshold, a target detection frame is determined from the first detection frame and the second detection frame, including: in response to the absolute value of the first frame number difference being not greater than the frame number difference threshold, a detection frame with a large matching degree between the first detection frame and the second detection frame is determined as the target detection frame, wherein the matching degree is used to characterize the degree of matching between the corresponding detection frame and the object.

[0011] Optionally, the matching degree corresponding to each detection frame at each moment in the video is obtained to obtain at least one matching degree of each detection frame; and the quotient between the sum of at least one matching degree of the detection frame and the number of at least one matching degree is determined as the target matching degree of the detection frame.

[0012] Optionally, the frame number difference threshold is determined based on historical detection data of object processing.

[0013] According to another aspect of an embodiment of the present invention, there is also provided an object recognition device, comprising: an acquisition unit, configured to acquire multiple detection frames of an object in a video, wherein each detection frame is used to represent the position of the object in the video; a first determination unit, configured to respectively determine the number of time frames of the multiple detection frames in the video, wherein the number of time frames is used to characterize the length of time that each detection frame appears continuously in the video; a second determination unit, configured to determine a target detection frame among the multiple detection frames based on the number of time frames and a frame difference threshold; and a recognition unit, configured to recognize an object in the video based on the target detection frame.

[0014] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the object recognition method of the embodiment of the present invention.

[0015] According to another aspect of an embodiment of the present invention, a processor is provided, which is used to run a program, wherein the object recognition method of the embodiment of the present invention is executed when the program is run.

[0016] In an embodiment of the present invention, multiple detection frames of an object in a video are obtained, wherein each detection frame is used to indicate the position of the object in the video; the time frame numbers of the multiple detection frames in the video are determined respectively, wherein the time frame number is used to characterize the length of time each detection frame appears continuously in the video; based on the time frame number and the frame difference threshold, a target detection frame is determined in the multiple detection frames; and the object is identified in the video based on the target detection frame. In other words, the embodiment of the present invention achieves the purpose of filtering out overlapping objects that appear for a shorter time by comparing the time frame numbers between multiple detection frames, thereby achieving the technical effect of improving the accuracy of object recognition and solving the technical problem of low accuracy of object recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 is a flow chart of a method for identifying an object according to an embodiment of the present invention;

[0019] Figure 2 is a flow chart of a method for obtaining a detection frame according to an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of a non-maximum suppression recognition process according to the related art;

[0021] FIG4(a) is a schematic diagram of a processing result without non-maximum suppression according to an embodiment of the present invention.

[0022] FIG4( b ) is a schematic diagram of a processing result without non-maximum suppression according to an embodiment of the present invention;

[0023] Figure 5 is a flowchart of a processing method for non-maximum suppression according to an embodiment of the present invention;

[0024] Figure 6 is a schematic diagram of an object recognition device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] Example 1

[0028] According to an embodiment of the present invention, an embodiment of a method for identifying an object is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] Figure 1 is a flow chart of a method for identifying an object according to an embodiment of the present invention. Figure 1 The method for identifying an object shown in FIG. 1 comprises the following steps:

[0030] Step S102: obtaining multiple detection frames of the object in the video, wherein each detection frame is used to represent the position of the object in the video.

[0031] In the technical solution provided in the above step S102 of the present invention, images of a period of time can be acquired by means of multiple sensors or the like to obtain a video to be processed, and the video can be processed to achieve the purpose of acquiring multiple detection frames of the object in the video, wherein the video can be images of a period of time, or images of a period of time acquired by an acquisition device such as a camera, for example, it can be a picture of a continuous period of time taken by a vehicle-mounted camera during driving, etc.; the object can be an object existing in the video, for example, it can be a truck on a highway, or a pedestrian at an intersection, etc., and no specific restrictions are imposed on the video acquisition method and the types of objects in the video; the detection frame can be a tilted frame (rotate box, abbreviated as rbox), can be a three-dimensional detection frame (3D rbox), for example, it can be a cube, a cuboid, or can be a two-dimensional detection frame (2D rbox) on a two-dimensional plane, for example, it can be a rectangle.

[0032] Optionally, multiple detection frames of objects in the video can be obtained through sensors installed at different positions. For example, a three-dimensional detection frame can be obtained through lidar and vision (camera) sensors, and a two-dimensional detection frame can be obtained through millimeter wave radar.

[0033] For example, when a vehicle is traveling on roundabouts, construction sections, downtown areas, and straight roads, a video of the vehicle traveling for a period of time is obtained, and each frame in the video can be processed through devices such as lidar, millimeter-wave radar, and vision to obtain detection frames corresponding to objects in each frame of the video, thereby obtaining multiple detection frames.

[0034] Optionally, a three-dimensional detection frame can be acquired through vision (camera) and laser radar (lidar), and the detected three-dimensional detection frame is projected onto the ground to obtain a two-dimensional detection frame, thereby obtaining information such as the center point, size, and orientation of the detection frame, wherein the information of the detection frame can be used to indicate the position of an object in the video.

[0035] In the related art, the detection of two-dimensional detection frames can only be achieved through the detection method of non-maximum suppression, but this method has limitations in filtering objects in three-dimensional space in the scene. In the embodiment of the present invention, a three-dimensional detection frame of each object in the video in each frame can be obtained, and the three-dimensional detection frame can be projected onto a plane, thereby achieving the purpose of obtaining multiple two-dimensional detection frames of objects in the video.

[0036] Step S104, determining the time frame numbers of the multiple detection frames in the video respectively, wherein the time frame number is used to represent the time length that each detection frame appears continuously in the video.

[0037] In the technical solution provided in the above step S104 of the present invention, the number of time frames in which each detection frame appears continuously in the video can be determined respectively, wherein the number of time frames can be represented by frame_cnt, which can be used to characterize the length of time that each detection frame appears continuously in the video, for example, it can be 1 second, 2 seconds, etc. This is only for illustration and no specific limitation is made.

[0038] For example, if the detection frame corresponding to the object 1 appears in three frames continuously from the second frame of the video, the number of time frames in which the detection frame corresponding to the object 1 appears in the video may be three frames.

[0039] Step S106, determining a target detection frame from multiple detection frames based on the time frame number and the frame number difference threshold.

[0040] In the technical solution provided in the above step S106 of the present invention, based on the number of time frames corresponding to each detection frame, the frame difference between the two detection frames is determined, and the time frame number and the frame difference threshold are compared. The comparison result can be used to determine the target detection frame from multiple detection frames, wherein the target detection frame can be the detection frame with the highest degree of match with the object, and the frame difference threshold can be called the time threshold, which can be represented by time_threshold, and can be a value set according to experience or a value set according to actual conditions.

[0041] For example, starting from the second frame of the video, the detection frame corresponding to object one appears in the video for three consecutive frames, and the detection frame corresponding to object two appears in the video for five consecutive frames, and the time frame number of object two in the video is five frames. Assuming that the frame difference threshold is one frame, object one and object two can be filtered based on the time frame number of object one, the time frame number of object two and the set frame difference threshold, so as to achieve the purpose of determining the target detection frame from multiple detection frames.

[0042] In the related art, by detecting images with a single time frame number, the influence of other time frame information on the perception results during a period of time is ignored. In the embodiment of the present invention, the remaining time frame information is introduced, and a time threshold is set according to the time frame number in which the object appears, so as to filter out overlapping objects with a shorter appearance time, thereby improving the accuracy of object recognition.

[0043] Step S108: identifying an object in the video based on the target detection frame.

[0044] In the technical solution of step S108 of the present invention, based on the target detection frame, the position, direction and other information of the object in each frame of the video are identified.

[0045] In the above steps S102 to S108 of the present application, multiple detection frames of an object in a video are obtained, wherein each detection frame is used to indicate the position of the object in the video; the time frame numbers of the multiple detection frames in the video are respectively determined, wherein the time frame number is used to characterize the length of time each detection frame continuously appears in the video; based on the time frame number and the frame difference threshold, a target detection frame is determined among the multiple detection frames; and the object is identified in the video based on the target detection frame, thereby achieving the technical effect of improving the accuracy of target recognition and solving the technical problem of low accuracy of target recognition.

[0046] The above method of this embodiment is further introduced below.

[0047] As an optional embodiment, based on the time frame number and the frame number difference threshold, a target detection frame is determined in multiple detection frames, including: determining the intersection-and-union ratio of a first detection frame in the multiple detection frames and a second detection frame in the multiple detection frames, wherein the first detection frame and the second detection frame are any two detection frames in the multiple detection frames at the same time; in response to the intersection-and-union ratio being greater than the intersection-and-union ratio threshold, obtaining the time frame number of the first detection frame and the time frame number of the second detection frame; based on the time frame number of the first detection frame, the time frame number of the second detection frame and the frame number difference threshold, determining the target detection frame in the first detection frame and the second detection frame.

[0048] In this embodiment, any two detection frames at the same time can be selected from multiple detection frames to obtain a first detection frame and a second detection frame, determine the intersection-over-union ratio between the first detection frame and the second detection frame, and judge whether the intersection-over-union ratio between the first detection frame and the second detection frame is greater than the intersection-over-union ratio threshold. In response to the intersection-over-union ratio being greater than the intersection-over-union ratio threshold, it is indicated that the first detection frame and the second detection frame overlap, and the time frame numbers of the first detection frame and the second detection frame can be determined respectively. Based on the time frame number of the first detection frame, the time frame number of the second detection frame, and the frame number difference threshold, the target detection frame is determined in the first detection frame and the second detection frame, wherein the intersection-over-union ratio can be used to characterize the degree of overlap between the detection frames, which can be represented by IoU; the intersection-over-union ratio threshold can be a value set according to actual conditions, or a value obtained based on experience, which can be represented by T; the first detection frame and the second detection frame can be any two detection frames in the same frame.

[0049] Optionally, a first detection frame and a second detection frame are selected from multiple detection frames, the intersection of the first detection frame and the second detection frame is calculated, and the intersection ratio between the two detection frames is determined based on the area of ​​the convex polygon composed of the set of intersection points. If the intersection ratio is greater than the intersection ratio threshold, it means that the two detection frames overlap, then the detection frames are screened by the number of time frames to determine the time frame number of the first detection frame and the time frame number of the second detection frame, and the target detection frame is determined in the first detection frame and the second detection frame based on the time frame number of the first detection frame, the time frame number of the second detection frame and the frame difference threshold.

[0050] Optionally, a first detection frame and a second detection frame are selected from multiple detection frames, an intersection operation is performed on the first detection frame and the second detection frame to obtain a corresponding intersection point set, and an intersection-and-union ratio between the two detection frames is obtained by judging the area of ​​a convex polygon composed of the intersection point set. That is, the intersection-and-union ratio can be calculated by the following formula:

[0051] IoU=area_r1r2 / (area_r1+area_r2-arear1r2)

[0052] Among them, area_r1 and area_r2 are the areas of the first detection box and the second detection box respectively, and area_r1r2 is the area of ​​the convex shape formed by the intersection of the two detection boxes.

[0053] Optionally, if the IoU is greater than a threshold T, the time frame number of the first detection frame and the time frame number of the second detection frame are determined, and the target detection frame is determined in the first detection frame and the second detection frame based on the time frame number of the first detection frame, the time frame number of the second detection frame and a frame difference threshold.

[0054] As an optional embodiment, based on the time frame number of the first detection frame, the time frame number of the second detection frame and a frame difference threshold, determining a target detection frame from the first detection frame and the second detection frame, including: determining a first frame difference between the time frame number of the first detection frame and the time frame number of the second detection frame; based on the first frame difference and the frame difference threshold, determining the target detection frame from the first detection frame and the second detection frame.

[0055] In this embodiment, a first detection frame and a second detection frame are selected from multiple detection frames, the time frame number of the first detection frame and the time frame number of the second detection frame are determined, the time frame number difference between the first detection frame and the second detection frame is determined, a first frame number difference is obtained, and the first frame number difference is judged based on a frame number difference threshold, thereby achieving the purpose of determining the target detection frame from the first detection frame and the second detection frame.

[0056] As an optional embodiment, based on the first frame number difference and the frame number difference threshold, a target detection frame is determined from the first detection frame and the second detection frame, including: in response to the absolute value of the first frame number difference being greater than the frame number difference threshold, determining the detection frame with a longer time frame number between the first detection frame and the second detection frame as the target detection frame.

[0057] In this embodiment, a first detection frame and a second detection frame are selected from multiple detection frames, the time frame number of the first detection frame and the time frame number of the second detection frame are determined, the time frame number difference between the first detection frame and the second detection frame is determined, and a first frame number difference is obtained. It is judged whether the absolute value of the first frame number difference is greater than a frame number difference threshold. In response to the absolute value of the first frame number difference being greater than the frame number difference threshold, the detection frame with the longer time frame number between the first detection frame and the second detection frame is determined as the target detection frame.

[0058] Optionally, a first detection frame and a second detection frame are selected from multiple detection frames, the intersection of the first detection frame and the second detection frame is calculated, and the intersection ratio between the two detection frames is determined based on the area of ​​the convex polygon composed of the set of intersection points. If the intersection ratio is greater than the intersection ratio threshold, it means that the two detection frames overlap, and the detection frames can be screened by the number of time frames; further, the number of time frames of the first detection frame and the number of time frames of the second detection frame are determined, and the frame number difference between the two detection frames is determined to obtain a first frame number difference. If the absolute value of the first frame number difference is greater than the frame number difference threshold, it means that the two detection frames not only overlap but also overlap for a long time, and the detection frame with a longer time frame number in the two detection frames is determined as the target detection frame. Optionally, the above description can be used to iterate between multiple detection frames to achieve the purpose of determining the target detection frame from multiple detection frames.

[0059] As an optional embodiment, based on the first frame number difference and the frame number difference threshold, a target detection frame is determined from the first detection frame and the second detection frame, including: in response to the absolute value of the first frame number difference being not greater than the frame number difference threshold, a detection frame with a large target matching degree between the first detection frame and the second detection frame is determined as the target detection frame, wherein the target matching degree is used to characterize the degree of matching between the corresponding detection frame and the object.

[0060] In this embodiment, a first detection frame and a second detection frame are selected from multiple detection frames, the time frame number of the first detection frame and the time frame number of the second detection frame are determined, the time frame number difference between the first detection frame and the second detection frame is determined, and a first frame number difference is obtained. It is judged whether the absolute value of the first frame number difference is greater than a frame number difference threshold. In response to the absolute value of the first frame number difference being not greater than the frame number difference threshold, a detection frame with a large target matching degree between the first detection frame and the second detection frame is determined as a target detection frame, wherein the target matching degree can be used to characterize the matching degree between the corresponding detection frame and the object, and can be the matching mean of the detection frame and the object over a period of time, for example, it can be the confidence mean, which can be represented by the score mean.

[0061] Optionally, a first detection frame and a second detection frame are selected from multiple detection frames, and the intersection of the first detection frame and the second detection frame is calculated, and the intersection ratio between the two detection frames is determined based on the area of ​​the convex polygon composed of the set of intersection points. If the intersection ratio is greater than the intersection ratio threshold, it means that the two detection frames overlap, and the detection frames can be screened by the number of time frames; further, the number of time frames of the first detection frame and the number of time frames of the second detection frame are determined, and the frame number difference between the two detection frames is determined to obtain the first frame number difference. If the absolute value of the first frame number difference is not greater than the frame number difference threshold, it means that the two detection frames overlap but the overlapping time is not long, and the detection frame with a large target matching degree between the first detection frame and the second detection frame is determined as the target detection frame. Optionally, the above description is iterated between multiple detection frames to achieve the purpose of determining the target detection frame from multiple detection frames.

[0062] In an embodiment of the present invention, the overlapping relationship between detection frames is determined, and detection frames with low target matching are filtered out, thereby filtering the corresponding three-dimensional space objects, which can effectively improve the problem of repetition, jump, and association of detection objects, which leads to unstable detection results, thereby improving the fusion performance of multi-sensor three-dimensional space targets and improving the accuracy of object detection.

[0063] As an optional embodiment, the matching degree corresponding to each detection frame at each moment in the video is obtained to obtain at least one matching degree of each detection frame; the quotient between the sum of at least one matching degree of the detection frame and the number of at least one matching degree is determined as the target matching degree of the detection frame.

[0064] In this embodiment, the matching degree corresponding to each detection frame at each moment in the video is obtained to obtain at least one matching degree of each detection frame, all matching degrees corresponding to each detection frame are added together, and the quotient between the sum of all matching degrees corresponding to the detection frame and the number of matching degrees is determined as the target matching degree of the detection frame, wherein the matching degree can be used to indicate the degree of matching between the detection frame and the object to be identified in each frame, which can be represented by a score.

[0065] In the related art, non-maximum suppression is a post-processing module in the target detection framework, which is mainly used to delete highly redundant detection frames. Optionally, multiple detection frames will be generated for each object in a frame of an image during detection. Non-maximum suppression is used to remove redundancy in multiple detection frames of each target to obtain the final target detection frame. In general, the detection frame is generally represented by a horizontal rectangular frame. Since the objects in the three-dimensional space output by the sensor have directional uncertainty, in an embodiment of the present invention, Incline Non-Maximum Suppression (INMS) is used for the directional rectangular frame (rotate box, rbox).

[0066] Optionally, in an embodiment of the present invention, multiple detection frames corresponding to the object can be obtained through multiple sensors, the detection library obtained by the millimeter wave radar can be filtered using a speed threshold, the three-dimensional detection frame obtained by the visual sensor and the lidar detection is projected to the ground to obtain a two-dimensional detection frame, the center point, size, orientation and other information of the object are obtained, the number of frames (frame_cnt) in which each detection frame appears and the matching degree (score) of each frame are determined respectively, the matching degrees of multiple detection frames are obtained, all the matching degrees corresponding to each detection frame are added, and the quotient between the sum of all the matching degrees corresponding to the detection frame and the number of matching degrees is determined as the target matching degree of the detection frame, and the matching degree of the same detection frame in each frame is changed to the target matching degree.

[0067] Optionally, the detection frames of each frame can be sorted by target matching degree to obtain a detection frame sequence (rbox_list), so as to compare the detection frames in the detection frame sequence to obtain the target detection frame. The detection frames are sorted here only to ensure that the detection frames can be processed in an orderly manner to avoid omissions. Therefore, no specific restrictions are imposed on the sorting method.

[0068] In an embodiment of the present invention, the target matching degree of the detection frame is set to the average matching degree of all frames in which it appears, which effectively improves the problems of associated target jumps and unstable associated results in fusion, thereby improving the fusion of results in the multi-sensor object detection process.

[0069] As an optional embodiment, the frame number difference threshold is determined based on historical detection data of object processing.

[0070] In this embodiment, selection can be performed through multiple experiments to achieve the purpose of determining a suitable frame number difference threshold.

[0071] The embodiment of the present invention achieves the purpose of filtering out overlapping objects that appear for a shorter time by comparing the number of time frames between multiple detection frames, thereby achieving the technical effect of improving the accuracy of target recognition and solving the technical problem of low accuracy of target recognition.

[0072] Example 2

[0073] The technical solution of the embodiment of the present invention is illustrated below in conjunction with preferred implementation modes.

[0074] The core functional modules in the field of autonomous driving can be divided into perception, decision-making and control modules. Among them, the perception module needs to achieve accurate perception of the surrounding environment to provide a basis for analysis for other subsequent modules. Therefore, improving the accuracy of object detection plays an important role. At the same time, different sensors perceive different object properties and perception ranges. If different sensors of the same type are installed in different positions, the perception areas will also be different. Therefore, it is necessary to fuse the perception results of each sensor to obtain more accurate information about surrounding objects.

[0075] At present, three-dimensional object detection, as one of the key technologies of the autonomous driving perception module, can provide information such as the position, size, direction and category of the object in the three-dimensional space, thereby completing three-dimensional space modeling, path planning, etc. However, due to certain limitations of the perception of a single sensor, the processing and fusion of multi-sensor (such as lidar, millimeter-wave radar and vision) data has become a trend in object detection technology in the field of autonomous driving.

[0076] In the related art, when sensors detect three-dimensional objects, there is a problem of overlapping frames, which leads to false detection and missed detection, thus having a great impact on the correlation accuracy and fusion robustness of multi-sensor fusion. To address the problem of overlapping frames in the detection of three-dimensional objects by sensors, non-maximum suppression is a commonly used detection filtering method, which is used to extract object detection frames with high confidence and suppress false detection frames with low confidence. Among them, standard non-maximum suppression is suitable for the detection of horizontal two-dimensional rectangular frames, and smooth non-maximum suppression (soft NMS) and adaptive non-maximum suppression (adaptive NMS) can be used to optimize non-maximum suppression, which is prone to missed detection when objects are densely packed and there is occlusion.

[0077] To solve the above problem, an embodiment of the present invention proposes a detection method based on Incline Non-Maximum Suppression (INMS), which is based on the inclined frame, and performs an intersection operation on the currently traversed detection frame and the remaining other detection frames to obtain a corresponding set of intersection points, and calculates the intersection ratio of each two detection frames based on the area of ​​the convex polygon composed of the intersection point set. However, this method mostly performs detection and filtering based on single-frame observation information, ignoring the role of other timestamp information on the perception results, and thus has the problem of low accuracy in detecting objects.

[0078] In a related technology, a sensor data filtering and fusion method for autonomous driving is provided. The method is divided into a spatial filtering and fusion method and a subsequent temporal filtering and fusion method. The spatial filtering and fusion method performs spatial clustering processing on the data points after the autonomous driving perception system obtains a frame of raw data from the sensor, identifies the effective clustering results and excludes noise; tracks the correlation of the clustering results, so as to obtain the position speed information, historical information, and prediction information of a certain object; then estimates the feature information, calculates the length, width and orientation information of the object after clustering; and then performs a temporal filtering and fusion method to distinguish the original target into a confirmed target, a suspicious target and a pseudo target. This method does not need to consider the type and number of sensors. Although it can be used for any sensor data with large noise and small noise, it has a small amount of calculation, is simple, flexible and effective, but most of the methods are based on single-frame observation information for detection and filtering, ignoring the role of other timestamp information on the perception results, and still has the problem of low accuracy in object detection.

[0079] In another related technology, a non-maximum suppression method for multi-target detection is provided, which obtains the category and confidence score of each detected target, and sorts each target according to the confidence score to obtain a detection queue; obtains the target with the highest confidence score in the detection queue, recorded as target A, and determines whether the confidence score of target A meets the category preset conditions; if the confidence score of target A meets the category preset conditions of the category to which it belongs, determines whether the detection frame of target A overlaps with the detection frames of other targets; if it is determined that the detection frames of target A overlap with the detection frames of other targets, the target whose detection frame overlaps with the detection frame of target A is recorded as target B, and it is determined whether target A and target B are of the same category, and a corresponding suppression algorithm is used according to the judgment result to determine whether target B is suppressed. Since most of these methods are based on single-frame observation information for detection filtering, the effect of other timestamp information on the perception results is ignored, and thus there is still a problem of low accuracy in object detection.

[0080] In another related technology, a 3D target detection method based on multi-information fusion is proposed, which collects driving images and point cloud information respectively and performs preprocessing; based on the driving image, a 2D target detector is used to obtain a 2D target frame and its score; based on the point cloud information, a 3D target detector is used to obtain a candidate 3D target frame; using the correspondence between the 2D target frame and the candidate 3D target frame, as well as the 2D target frame score, the candidate 3D target frame is screened to obtain the 3D target frame. In other words, this method makes up for the deficiency of sparse point cloud by introducing visual information, constrains the 3D target frame by the 2D detection result, improves the recall rate of the 3D target frame, and reduces the probability of false detection and missed detection. However, this method does not consider the fusion of multi-sensor data, and therefore there is still a technical problem of low accuracy in object detection.

[0081] To solve the above problems, the embodiments of the present invention aim at the problem that overlapping frames are prone to occur in the process of multi-sensor 3D target detection, resulting in reduced fusion association accuracy. A multi-sensor 3D target detection filtering and fusion method based on INMS is proposed. The detection frame is obtained from multiple sensors including lidar, millimeter-wave radar and visual sensor, and the obtained detection frame is pre-processed and filtered. The detection results of the visual sensor and the millimeter-wave radar are filtered according to certain score thresholds and speed thresholds, respectively, wherein the score threshold and the speed threshold can be values ​​set in advance according to actual conditions or experience; and the detected 3D rbox is projected onto the ground to obtain a 2D rbox, thereby obtaining the center point, size, and orientation information of the detection box; in view of the related technologies, most of them are based on single-frame observation information for detection and filtering, ignoring the effect of other timestamp information on the perception results. In INMS, the information of other timestamps is introduced into the processing of a single detection box, and the time threshold is set according to the duration of the detection box, and some overlapping targets with a shorter appearance time are filtered out, and the matching degree (score) corresponding to the detection box is set to the mean matching degree of all its appearing frames, which can effectively improve the problems of overlapping, jumping, and unstable association results of detected targets, and enhance the fusion performance of multi-sensor target detection.

[0082] The embodiments of the present invention are further described below.

[0083] The embodiment of the present invention is mainly divided into two aspects: acquisition and processing of the detection frame.

[0084] Figure 2 is a flow chart of a method for obtaining a detection frame according to an embodiment of the present invention. Figure 2 As shown, the acquisition of the detection frame in the embodiment of the present invention may include the following steps:

[0085] Step S201, acquiring data from each sensor.

[0086] In this embodiment, the multi-sensor may include a laser radar (lidar), a millimeter-wave radar (radar), and a visual sensor (camera), and the detection data is acquired by using the laser radar, the millimeter-wave radar, and the visual sensor.

[0087] Step S202: filtering the acquired detection data based on a score threshold or a speed threshold.

[0088] In this embodiment, the acquired targets are pre-processed and filtered.

[0089] Optionally, the three-dimensional detection result (3Dbox) obtained by lidar can be projected onto the ground to obtain a two-dimensional detection box (2D rbox), and information such as the center point, size, and orientation of the rbox can be obtained.

[0090] Optionally, the detection results obtained by the camera can be filtered based on a score threshold. For example, the three-dimensional detection box (3D rbox) can be filtered by the score threshold, and the filtered detection results are projected onto a plane to obtain a corresponding plane detection box, and the center point, size, orientation information and other information of the detection box are obtained. The score threshold can be an empirical value, or a value obtained according to actual conditions or experimental tests. No specific restrictions are imposed on the determination of the score threshold.

[0091] Optionally, for the detection data acquired by radar, the acquired detection results can be filtered by a speed threshold to obtain a detection box that needs to be processed.

[0092] Step S203, input the data after all sensor screening into the tilt non-maximum suppression, and process the data.

[0093] The filtered detection frame is input into the improved tilt non-maximum suppression, and the data is processed using the improved tilt non-maximum suppression.

[0094] In related technologies, non-maximum suppression is a post-processing module in the target detection framework, which can be used to delete highly redundant detection boxes. Figure 3 is a schematic diagram of a non-maximum suppression recognition process according to related technology, such as Figure 3 As shown in FIG. 4 , multiple detection frames will be generated when detecting object A and object B. The essence of non-maximum suppression is to remove redundancy from multiple detection frames of each object to obtain the final detection result, which can be used to obtain the target detection frame with the maximum final matching degree. However, target detection generally uses a horizontal rectangular frame, and the three-dimensional target output by the sensor has directional uncertainty. Therefore, the embodiment of the present invention adopts tilted non-maximum suppression for the directional rectangular frame, and introduces the information of the remaining timestamps for a single target in the tilted non-maximum suppression. The time threshold is set according to the duration of its appearance, and some overlapping targets with a shorter appearance time are filtered out, and the target score is set to the average score of all its appearance frames. FIG. 4(a) is a schematic diagram of a processing result without non-maximum suppression according to an embodiment of the present invention, and FIG. 4(b) is a schematic diagram of a processing result without non-maximum suppression according to an embodiment of the present invention. As shown in FIG. 4(a) and FIG. 4(b), the embodiment of the present invention can improve the jump of associated targets and the instability of associated results in fusion. At the same time, the overlapping parts are filtered out by using the number of time frames, thereby effectively improving the fusion performance of multi-sensor three-dimensional targets.

[0095] Figure 5 is a flowchart of a processing method for non-maximum suppression according to an embodiment of the present invention. Figure 5 As shown, the processing of INSM proposed in the embodiment of the present invention may include the following steps:

[0096] Step S501, processing the detection frames filtered by each sensor.

[0097] Optionally, the number of time frames (frame_cnt) in which each detection frame appears and the confidence of each frame are obtained, the mean confidence of each detection frame is determined, and the score of each frame of the detection frame is changed to the corresponding mean confidence.

[0098] Optionally, all detection boxes in the same frame are sorted according to the confidence threshold to obtain a sorting table (rbox_list). It should be noted that the sorting can be in ascending order or descending order. The main purpose of the sorting is to ensure that all detection boxes can be processed to prevent the detection boxes from being missed during the processing. Therefore, the sorting method is not specifically described here. Any method for obtaining the sorting order should be within the protection scope of the embodiments of the present invention.

[0099] Step S502: Filter the detection frame using the intersection-over-union ratio threshold.

[0100] Optionally, starting from the first detection box in rbox_list, the intersection-and-union ratio between each detection box is determined in turn. The intersection operation can be performed on the detection box and other detection boxes to obtain the corresponding intersection point set. The intersection-and-union ratio between each two detection boxes is determined by judging the area of ​​the convex polygon composed of the intersection point set. The intersection-and-union ratio can be determined by the following formula:

[0101] IoU=area_r1r2 / (area_r1+area_r2-arear1r2),

[0102] Among them, area_r1 and area_r2 are the areas of two detection frames respectively, and area_r1r2 is the area of ​​the convex shape formed by the intersection of the two detection frames.

[0103] Optionally, if the IoU is greater than the IoU threshold (T), step S503 is implemented; if the IoU is not greater than the IoU threshold (T), the detection frame is retained.

[0104] Step S503, calculating the frame number difference between the detection frames, and filtering the overlapping detection frames using the frame number difference threshold.

[0105] Optionally, if the IoU is greater than the IoU threshold, it indicates that there is a possibility of overlap between the two detection frames, and the frame number difference between the two detection frames is judged and the frame number difference between the detection frames is calculated.

[0106] Optionally, the size of the frame difference threshold and the frame difference between the detection frames is determined. If the frame difference between the detection frames is not greater than the frame difference threshold (time_threshold), the detection frames with a larger matching degree are retained, and the detection frames with a smaller matching degree are removed; if the frame difference is greater than the frame difference threshold, the detection frames with a longer corresponding time frame number are retained, and the detection frames with a shorter corresponding time frame number are removed.

[0107] In the related art, by detecting images with a single time frame number, the influence of other time frame information on the perception results during a period of time is ignored. In the embodiment of the present invention, the remaining time frame information is introduced, and a time threshold is set according to the time frame number in which the object appears, so as to filter out overlapping objects with a shorter appearance time, thereby improving the accuracy of object recognition.

[0108] Optionally, multiple detection boxes are processed through step S502 and step S504, and the operation is iterated until all detection boxes in rbox_list are screened.

[0109] Step S504, obtaining the filtered target and performing multi-sensor data fusion.

[0110] The filtered target is obtained, multi-sensor data fusion is performed, and the detection frame that has been filtered multiple times is processed to obtain the object to be identified.

[0111] In the related art, the NMS algorithm can only realize the detection of two-dimensional horizontal rectangular frames, and there are limitations in the filtering of spatial objects in the scene. The embodiment of the present invention proposes a detection algorithm based on INMS to realize a multi-sensor three-dimensional space target detection filtering method. The method generates a three-dimensional detection frame for each target and projects it onto the ground to generate a corresponding two-dimensional detection frame. The detection frames with low matching degrees are filtered out based on the judgment of the overlapping relationship of the detection frames, and the corresponding overlapping objects are filtered out accordingly, thereby effectively improving the problems of overlapping, jumping, and unstable association results of the detection objects, thereby improving the fusion performance of multi-sensor spatial objects.

[0112] Furthermore, in view of the problem in the related art that detection and filtering are performed based on single-frame observation information, while the effect of other timestamp information on the perception results is ignored, the embodiments of the present invention improve the existing INMS by introducing other timestamp information for a single target in the INMS, setting a time threshold according to the duration of its appearance, filtering out some overlapping targets with a shorter appearance time, and setting the matching degree of the detection frame to the average matching degree of all its appearance frames, thereby improving the stability of filtering.

[0113] In an embodiment of the present invention, by comparing the number of time frames between multiple detection frames, the purpose of filtering out overlapping objects that appear for a shorter time is achieved, thereby achieving the technical effect of improving the accuracy of target recognition and solving the technical problem of low accuracy of target recognition.

[0114] Example 3

[0115] According to an embodiment of the present invention, a device for identifying an object is also provided. It should be noted that the device for identifying an object can be used to execute the method for identifying an object in the first embodiment.

[0116] Figure 6 FIG. 1 is a schematic diagram of an object recognition device according to an embodiment of the present invention. Figure 6 As shown, the object recognition device 600 may include: an acquisition unit 602 , a first determination unit 604 , a second determination unit 606 and a recognition unit 608 .

[0117] The acquisition unit 602 is used to acquire multiple detection frames of the object in the video, wherein each detection frame is used to represent the position of the object in the video.

[0118] The first determining unit 604 is used to respectively determine the time frame numbers of the multiple detection frames in the video, wherein the time frame number is used to represent the time length that each detection frame appears continuously in the video.

[0119] The second determination unit 606 is used to determine the target detection frame from the multiple detection frames based on the time frame number and the frame number difference threshold.

[0120] The recognition unit 608 is used to recognize an object in the video based on the target detection frame.

[0121] Optionally, the second determination unit 606 includes: a first determination module, used to determine the intersection-and-union ratio of a first detection frame among multiple detection frames and a second detection frame among multiple detection frames, wherein the first detection frame and the second detection frame are any two detection frames in the multiple detection frames at the same time; in response to the intersection-and-union ratio being greater than the intersection-and-union ratio threshold, obtaining the time frame number of the first detection frame and the time frame number of the second detection frame; based on the time frame number of the first detection frame, the time frame number of the second detection frame and the frame number difference threshold, determining the target detection frame in the first detection frame and the second detection frame.

[0122] Optionally, the first determination module includes: a first determination submodule, used to determine a first frame number difference between the time frame number of the first detection frame and the time frame number of the second detection frame; based on the first frame number difference and the frame number difference threshold, determine the target detection frame from the first detection frame and the second detection frame.

[0123] Optionally, the first determination submodule is further configured to determine, in response to an absolute value of the first frame number difference being greater than a frame number difference threshold, a detection frame with a longer time frame number between the first detection frame and the second detection frame as a target detection frame.

[0124] Optionally, the first determination submodule is also used to determine a detection frame with a large matching degree between the first detection frame and the second detection frame as a target detection frame in response to the absolute value of the first frame difference being not greater than a frame difference threshold, wherein the matching degree is used to characterize the degree of matching between the corresponding detection frame and the object.

[0125] Optionally, the device also includes: a third determination unit, used to obtain the matching degree corresponding to each detection frame at each moment in the video, and obtain at least one matching degree of each detection frame; and determine the quotient between the sum of at least one matching degree of the detection frame and the number of at least one matching degree as the target matching degree of the detection frame.

[0126] Optionally, the device further includes: a fourth determination unit, configured to determine a frame number difference threshold based on historical detection data processed on the object.

[0127] In an embodiment of the present invention, a plurality of detection frames of an object in a video are obtained by an acquisition unit, wherein each detection frame is used to indicate the position of the object in the video; a first determination unit is used to respectively determine the time frame numbers of the plurality of detection frames in the video, wherein the time frame number is used to characterize the length of time each detection frame continuously appears in the video; a second determination unit is used to determine a target detection frame in the plurality of detection frames based on the time frame number and the frame difference threshold; an identification unit is used to identify an object in the video based on the target detection frame. In other words, the embodiment of the present invention achieves the purpose of filtering out overlapping objects that appear for a shorter time by comparing the time frame numbers between the plurality of detection frames, thereby achieving the technical effect of improving the accuracy of target recognition and solving the technical problem of low accuracy of target recognition.

[0128] Example 4

[0129] According to an embodiment of the present invention, a computer-readable storage medium is further provided. The storage medium includes a stored program, wherein the program executes the object recognition method described in Embodiment 1.

[0130] Example 5

[0131] According to an embodiment of the present invention, a processor is further provided. The processor is used to run a program, wherein the object recognition method described in Embodiment 1 is executed when the program is run.

[0132] Example 6

[0133] According to an embodiment of the present invention, a vehicle is further provided. The vehicle is used to run a program, wherein the object recognition method described in Example 1 is executed when the program is run.

[0134] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0135] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0136] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0137] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0138] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0139] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0140] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for identifying an object, characterized in that: include: Acquire multiple detection frames of an object in a video, wherein each detection frame is used to represent a position of the object in the video; respectively determining the time frame numbers of the plurality of detection frames in the video, wherein the time frame numbers are used to represent the time length of continuous appearance of each detection frame in the video; Determine a target detection frame from the plurality of detection frames based on the time frame number and the frame number difference threshold; identifying the object in the video based on the target detection frame; Wherein, based on the time frame number and the frame number difference threshold, determining the target detection frame in the multiple detection frames, including: determining the intersection-and-union ratio of a first detection frame in the multiple detection frames and a second detection frame in the multiple detection frames, wherein the first detection frame and the second detection frame are any two detection frames in the multiple detection frames at the same time; in response to the intersection-and-union ratio being greater than the intersection-and-union ratio threshold, obtaining the time frame number of the first detection frame and the time frame number of the second detection frame; based on the time frame number of the first detection frame, the time frame number of the second detection frame and the frame number difference threshold, determining the target detection frame in the first detection frame and the second detection frame.

2. The method according to claim 1, characterized in that Determining a target detection frame from the first detection frame and the second detection frame based on the time frame number of the first detection frame, the time frame number of the second detection frame, and the frame number difference threshold includes: Determine a first frame number difference between the time frame number of the first detection frame and the time frame number of the second detection frame; Based on the first frame number difference and the frame number difference threshold, the target detection frame is determined from the first detection frame and the second detection frame.

3. The method according to claim 2, characterized in that Determining a target detection frame from the first detection frame and the second detection frame based on the first frame number difference and the frame number difference threshold includes: In response to the absolute value of the first frame number difference being greater than the frame number difference threshold, the detection frame having the longer time frame number between the first detection frame and the second detection frame is determined as the target detection frame.

4. The method according to claim 2, characterized in that: Determining a target detection frame from the first detection frame and the second detection frame based on the first frame number difference and the frame number difference threshold includes: In response to the absolute value of the first frame difference being not greater than the frame difference threshold, the detection frame with a greater target matching degree between the first detection frame and the second detection frame is determined as the target detection frame, wherein the target matching degree is used to characterize the degree of matching between the corresponding detection frame and the object.

5. The method according to claim 1, characterized in that: The method further comprises: Obtaining the matching degree corresponding to each detection frame at each moment in the video, and obtaining at least one matching degree of each detection frame; The quotient between the sum of at least one matching degree of the detection frame and the number of the at least one matching degree is determined as the target matching degree of the detection frame.

6. The method according to claim 1, characterized in that The method further comprises: The frame number difference threshold is determined based on historical detection data processed on the object.

7. A method for identifying an object, characterized in that: include: An acquisition unit, used to acquire multiple detection frames of an object in a video, wherein each of the detection frames is used to indicate a position of the object in the video; A first determining unit, configured to respectively determine the number of time frames of the plurality of detection frames in the video, wherein the number of time frames is used to represent the length of time that each detection frame appears continuously in the video; A second determining unit, configured to determine a target detection frame from the plurality of detection frames based on the time frame number and the frame number difference threshold; an identification unit, configured to identify the object in the video based on the target detection frame; Among them, the second determination unit is used to determine the target detection frame in the multiple detection frames based on the time frame number and the frame number difference threshold through the following steps: determine the intersection-and-union ratio of a first detection frame in the multiple detection frames and a second detection frame in the multiple detection frames, wherein the first detection frame and the second detection frame are any two detection frames in the multiple detection frames at the same time; in response to the intersection-and-union ratio being greater than the intersection-and-union ratio threshold, obtain the time frame number of the first detection frame and the time frame number of the second detection frame; based on the time frame number of the first detection frame, the time frame number of the second detection frame and the frame number difference threshold, determine the target detection frame in the first detection frame and the second detection frame.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 6.

9. A vehicle, characterized in that: Used to perform the method according to any one of claims 1 to 6.

Images