Dynamic object feature point detection method and device, electronic equipment and storage medium

Abstract

The application discloses a dynamic object feature point detection method and device, electronic equipment and a storage medium; the method comprises the following steps: acquiring a current frame image and a previous frame image, performing feature point extraction on the current frame image to obtain image feature points; performing object detection on the image feature points to determine a detection frame corresponding to an object, determining a first background feature point set and a first object feature point set corresponding to each detection frame according to the detection frames; if the total area of the detection frames meets a preset condition, determining a first displacement vector set corresponding to a background feature point according to the first background feature point set and the previous frame image; for each detection frame, determining a second displacement vector set corresponding to the object according to the first object feature point set corresponding to the detection frame and the previous frame image, and determining whether the object feature points in the detection frame are dynamic object feature points based on the first displacement vector set and the second displacement vector set, thereby solving the problem that the result is inaccurate when a monocular camera performs dynamic object detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a dynamic object feature point detection method and device, electronic equipment and a storage medium. BACKGROUND

[0002] Visual mapping technology is a technology based on data collected by sensors such as cameras, using algorithms to realize three-dimensional model reconstruction, positioning and navigation of the environment. Visual mapping positioning and mapping technology has wide application prospects in the fields of robots, autonomous driving, etc.

[0003] In traditional visual positioning and mapping, it is usually based on the assumption that all objects in the scene are in a stationary state. However, in scenarios such as logistics parks and open roads, a certain number of moving vehicles will appear during mapping, which will interfere with the correct interpretation and analysis of images, thereby causing interference to mapping and further affecting the positioning effect of autonomous vehicles based on the map.

[0004] In conventional visual mapping, dynamic object feature point removal usually relies on binocular, RGBD, etc. camera devices that can collect depth information. By measuring whether the depth information of the camera feature point changes at different times, the feature points from dynamic objects can be judged and removed. When mapping based on a monocular camera, it is difficult to remove dynamic objects because the depth information of the pixel points cannot be obtained. A mask of object detection is usually used to remove all pixels from prior dynamic objects (vehicles, pedestrians); or the algorithm identifies the detection box where the object is located, and the points in the detection box are directly regarded as dynamic points. However, the accuracy of the object detection mask is easily affected by false positives and false negatives of dynamic objects; the points in the detection box are not all dynamic points, and may also be static points, so directly regarding the points in the detection box as dynamic points is not accurate. The accuracy of the existing dynamic object detection method is low, which further affects the mapping quality. SUMMARY

[0005] The present application provides a dynamic object feature point detection method, device, electronic equipment and storage medium to improve the detection accuracy when detecting dynamic objects through a monocular camera, and further improve the quality of monocular camera visual mapping.

[0006] According to an aspect of the present application, a dynamic object feature point detection method is provided, comprising:

[0007] obtaining a current frame image and a previous frame image, performing feature point extraction on the current frame image to obtain image feature points, wherein the current frame image is collected by a monocular camera;

[0008] perform object detection on the image feature points, determine a detection box corresponding to at least one object, and determine a first background feature point set and a first object feature point set corresponding to each detection box according to the detection boxes;

[0009] if a total area of the detection boxes meets a preset condition, determine a first displacement vector set corresponding to background feature points according to the first background feature point set and the previous frame of image;

[0010] for each detection box, determine a second displacement vector set corresponding to an object according to the first object feature point set corresponding to the detection box and the previous frame of image, and determine whether an object feature point in the detection box is a dynamic object feature point based on the first displacement vector set and the second displacement vector set.

[0011] According to another aspect of the present application, a dynamic object feature point detection device is provided, comprising:

[0012] a feature point extraction module configured to acquire a current frame of image and a previous frame of image, perform feature point extraction on the current frame of image to obtain image feature points, wherein the current frame of image is acquired by a monocular camera;

[0013] an object detection module configured to perform object detection on the image feature points, determine a detection box corresponding to at least one object, and determine a first background feature point set and a first object feature point set corresponding to each detection box according to the detection boxes;

[0014] a first displacement set determination module configured to, if a total area of the detection boxes meets a preset condition, determine a first displacement vector set corresponding to background feature points according to the first background feature point set and the previous frame of image;

[0015] a first dynamic point detection module configured to, for each detection box, determine a second displacement vector set corresponding to an object according to the first object feature point set corresponding to the detection box and the previous frame of image, and determine whether an object feature point in the detection box is a dynamic object feature point based on the first displacement vector set and the second displacement vector set.

[0016] According to another aspect of the present application, an electronic device is provided, comprising:

[0017] at least one processor; and

[0018] a memory connected in communication with the at least one processor; wherein

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the dynamic object feature point detection method according to any one of the embodiments of the application.

[0020] According to another aspect of the application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement the dynamic object feature point detection method according to any one of the embodiments of the application when executed by the processor.

[0021] The technical scheme of the embodiment of the application, by acquiring a current frame image and a previous frame image, performing feature point extraction on the current frame image to obtain image feature points, wherein the current frame image is acquired by a monocular camera; performing object detection on the image feature points to determine at least one detection box corresponding to an object, and determining a first background feature point set and a first object feature point set corresponding to each detection box according to the detection boxes; if the total area of the detection boxes meets a preset condition, determining a first displacement vector set corresponding to a background feature point according to the first background feature point set and the previous frame image; for each detection box, determining a second displacement vector set corresponding to an object according to the first object feature point set corresponding to the detection box and the previous frame image, and determining whether the object feature points in the detection box are dynamic object feature points based on the first displacement vector set and the second displacement vector set, solves the problem of inaccurate detection results when a monocular camera performs dynamic object detection, and through object detection on the image feature points in the current frame image to obtain detection boxes, further determines the first background feature point set composed of background feature points and the first object feature point set corresponding to each detection box, and when the total area of the detection boxes meets the preset condition, performs dynamic object detection based on the first background feature point set, and through the first displacement vector set and the second displacement vector set, distinguishes the object feature points in the detection box as static or dynamic, improves the accuracy of the detection results, reduces the occurrence of false detection, and has low detection cost.

[0022] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the application, nor is it used to limit the scope of the application. Other features of the application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort.

[0024] Figure 1 is a flow chart of a dynamic object feature point detection method according to an embodiment of the present application;

[0025] Figure 2 is a flow chart of a dynamic object feature point detection method according to an embodiment of the present application;

[0026] Figure 3 is a structural schematic diagram of a dynamic object feature point detection device according to an embodiment of the present application;

[0027] Figure 4 is a structural schematic diagram of an electronic device implementing a dynamic object feature point detection method according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the personnel in the technical field better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0029] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment One

[0031] Figure 1 A flow chart of a dynamic object feature point detection method according to an embodiment of the present application is provided, and the present embodiment can be applicable to the case of identifying feature points in images dynamically and statically. The method can be executed by a dynamic object feature point detection device, which can be realized in the form of hardware and / or software, and can be configured in an electronic device. As shown in the figure, the method comprises: Figure 1 ​

[0032] S101, acquire a current frame image and a previous frame image, perform feature point extraction on the current frame image to obtain image feature points, wherein the current frame image is collected by a monocular camera.

[0033] In the embodiment, the current frame image can be specifically understood as an image currently performing dynamic object feature point detection, and the current frame image is collected by a monocular camera. The previous frame image can be specifically understood as an image collected before the current frame. The image feature points can be specifically understood as feature points in the current frame image.

[0034] Specifically, the monocular camera can be arranged on a vehicle, a robot or the like. The working frequency of the monocular camera is set in advance, and the monocular camera collects images periodically according to the working frequency. In the embodiment, when performing dynamic object feature point detection, the images can be processed at the same frequency to realize dynamic object detection, that is, each frame of image collected by the binocular camera can be sent to the execution device for processing to realize dynamic object feature point detection. Considering the performance of the execution device, the images can also be processed at different frequencies. For example, the collection frequency of the binocular camera is higher than the processing frequency of the execution device. At this time, if the execution device needs to ensure real-time performance, it cannot process all the images collected by the binocular camera. Therefore, the execution device can acquire the image collected at the latest time as the current frame image for processing when processing the images. The previous frame image can be an image collected in the last collection cycle of the current frame image, or the image detected at the time of the last dynamic object feature point detection of the current frame image.

[0035] The feature points of the current frame image are extracted by using a traditional feature extraction method such as Harris corner detector, SIFT feature point detector, or a deep learning-based feature extraction method such as SuperPoint, to obtain the image feature points in the current frame image.

[0036] S102, perform object detection on the image feature points, determine at least one detection box corresponding to an object, and determine a first background feature point set and a first object feature point set corresponding to each detection box according to the detection boxes.

[0037] In the embodiment, the detection frame can be a rectangle, a square, or the like; the first background feature point set can be specifically understood as a set composed of coordinates of background feature points in the current frame image, the background feature points being used to represent the background in the current frame image, and the first background feature point set including coordinates of at least one background feature point. The first object feature point set can be specifically understood as a set composed of coordinates of object feature points in a detection frame in the current frame image, the object feature points being used to represent objects in the current frame image, and the first object feature point set including coordinates of at least one object feature point. Each object corresponds to a detection frame, and each detection frame corresponds to a first object feature point set, that is, the number of first object feature point sets is the same as the number of detection frames.

[0038] It should be noted that, in the embodiment of the present application, the feature points are generally represented by coordinate values, and each feature point set refers to a set composed of coordinates of feature points.

[0039] Specifically, object detection on image feature points can be performed by using a deep learning algorithm, for example, a Yolo algorithm, an RCNN algorithm, or the like. In the embodiment of the present application, any target detection algorithm can be used when performing object detection. At least one detection frame corresponding to an object in the current frame image is obtained through object detection. When the current frame image includes multiple objects, the sizes of the objects in the image can be different due to factors such as the volume and distance of the objects. Therefore, the sizes of the detection frames corresponding to each object can be the same or different. The background feature points and the object feature points in the current frame image are distinguished according to the positions, sizes, and the like of all the detection frames in the current frame image. The first background feature point set is formed based on the background feature points representing the background, and the first object feature point set corresponding to each detection frame is formed based on the object feature points representing the objects.

[0040] In the embodiment, the preset condition can be set in advance according to requirements such as detection accuracy and processing speed; for example, the preset condition is that the total area of the detection frames is less than a certain threshold, or the preset condition is that the ratio of the total area of the detection frames to the total area of the current frame image is less than a certain threshold, or the like. The first displacement vector set can be specifically understood as a set composed of displacements of the background feature points relative to the previous frame image, and the first displacement vector set including at least one first displacement.

[0041] In the embodiment, the preset condition can be set in advance according to requirements such as detection accuracy and processing speed; for example, the preset condition is that the total area of the detection frames is less than a certain threshold, or the preset condition is that the ratio of the total area of the detection frames to the total area of the current frame image is less than a certain threshold, or the like. The first displacement vector set can be specifically understood as a set composed of displacements of the background feature points relative to the previous frame image, and the first displacement vector set including at least one first displacement.

[0042] Specifically, the area of each detection frame is calculated according to the height and width of each detection frame, and the total area of all detection frames is further calculated. The total area is compared with a preset condition. If the total area meets the preset condition, the first background feature point set and the previous frame image are detected to determine the corresponding background feature points in the two frames of images, and the displacement between the one-to-one corresponding background feature points is calculated to form the first displacement vector set.

[0043] In S104, for each detection frame, a second displacement vector set corresponding to the object is determined according to the first object feature point set corresponding to the detection frame and the previous frame image, and whether the object feature point in the detection frame is a dynamic object feature point is determined based on the first displacement vector set and the second displacement vector set.

[0044] In this embodiment, the second displacement vector set can be understood as a set composed of the displacements of each object feature point relative to the previous frame image. The second displacement vector set includes at least one second displacement.

[0045] For each detection frame, the same method can be used to determine whether the object feature point in the detection frame is a dynamic object feature point. For each detection frame, the first object feature point set corresponding to the detection frame is determined, the first object feature point set and the previous frame image are detected to determine the corresponding object feature points in the two frames of images, and the displacement between the one-to-one corresponding object feature points is calculated to form the second displacement vector set. The judgment condition for distinguishing dynamic and static object feature points is determined based on the first displacement vector set, for example, the mean, median, weighted value, etc. of each first displacement in the first displacement vector set is calculated as the judgment condition. Then, the second displacement in the second displacement vector set is judged according to the judgment condition, for example, the size of the second displacement is compared with the value in the judgment condition, or whether the difference between the second displacement and the value in the judgment condition is within a certain range is calculated, and then whether the object feature point is a dynamic object feature point is determined.

[0046] In the embodiments of the present application, the object feature points in the detection frame can be further judged to distinguish whether they are dynamic points or static points. Compared with the prior art in which all points in the detection frame are directly regarded as dynamic points, the dynamic point recognition method provided in the embodiments of the present application is more accurate.

[0047] The embodiment of the application provides a dynamic object feature point detection method, solves the problem of inaccurate detection results when a monocular camera detects a dynamic object, performs object detection on image feature points in a current frame image to obtain a detection frame, further determines a first background feature point set composed of background feature points and a first object feature point set corresponding to each detection frame, when the total area of each detection frame meets a preset condition, performs dynamic object detection based on the first background feature point set, and distinguishes object feature points in the detection frame into static and dynamic through a first displacement vector set and a second displacement vector set, thereby improving the accuracy of the detection result and reducing the occurrence of false detection.

[0048] Embodiment two

[0049] Figure 2 A flowchart of a dynamic object feature point detection method provided by the second embodiment of the application is provided, and the embodiment is optimized on the basis of the above-mentioned embodiment. As shown in the figure, the method comprises the following steps. Figure 2

[0050] S201, a current frame image and a previous frame image are acquired, feature points of the current frame image are extracted, and image feature points are obtained, wherein the current frame image is acquired through a monocular camera.

[0051] S202, object detection is performed on the image feature points, detection frames corresponding to at least one object are determined, and a first background feature point set and a first object feature point set corresponding to each detection frame are determined according to each detection frame.

[0052] As an optional embodiment of the embodiment, the first background feature point set and the first object feature point set corresponding to each detection frame are further optimized as follows.

[0053] A1, for each image feature point, whether the image feature point is in the detection frame is judged according to the coordinates of the image feature point and the size and position information of each detection frame, if yes, A2 is executed, otherwise, A3 is executed.

[0054] In the embodiment, the size of the detection frame can be the height and width of the detection frame, and the position information of the detection frame can be the coordinates of the center point, the top-left vertex and the top-right vertex of the detection frame. In the embodiment of the application, the size and position information of the detection frame can be obtained when the detection frame is determined. The coordinates of the image feature point are compared with the size and position information of the detection frame to judge whether the image feature point is in the detection frame, if yes, A2 is executed, otherwise, A3 is executed.

[0055] ​Exemplarily, the embodiment of the present application provides a method for judging whether an image feature point is in a detection frame: taking a horizontal right direction as an x axis and a vertical downward direction as a y axis as an example, the coordinates of the image feature point can be expressed as: [x1, y1], wherein x1 is a pixel horizontal coordinate of the image feature point, and y1 is a pixel vertical coordinate of the image feature point. The detection frame can be generally expressed as: [x, y, w, h], wherein [x, y] represents a top-left vertex of the detection frame, w is a width of the detection frame, and h is a height of the detection frame.

[0056] Based on the following formula, it can be judged whether the image feature point is in the detection frame:

[0057] x < x1 < x + w;

[0058] y < y1 < y + h;

[0059] In the embodiment of the present application, the image feature point is compared with all the detection frames in turn to determine whether the image feature point is in a certain detection frame. All the image feature points are traversed to judge whether each image feature point is a background feature point or an object feature point.

[0060] A2, the image feature point is added to the first object feature point set corresponding to the detection frame in which the image feature point is located as an object feature point.

[0061] If the image feature point is in the detection frame, it is determined that the image feature point is an object feature point, and the detection frame in which the image feature point is located is added to the first object feature point set corresponding to the detection frame in which the image feature point is located.

[0062] A3, the image feature point is added to the first background feature point set as a background feature point.

[0063] If the image feature point is not in any detection frame, it is determined that the image feature point is a background feature point, and the image feature point is added to the first background feature point set.

[0064] In the embodiment of the present application, when an object is detected, the object can also be numbered, that is, the detection frame is numbered. When the first object feature point set corresponding to the detection frame is determined, the object number can be added to the object feature point in the first object feature point set. After each image feature point is judged, the object feature point in each first object feature point set obtained at this time is added with a corresponding object number.

[0065] In the embodiment of the present application, the size and position information of the detection frame are used to judge whether the image feature point is an object feature point or a background feature point, and the image feature point is added to the first object feature point set or the first background feature point set, so that different ways are used for dynamic object feature point detection when the total area of the detection frame meets the preset condition and does not meet the preset condition.

[0066] S203, determine whether the total area of each detection frame meets a preset condition, if yes, execute S204; otherwise, execute S211.

[0067] The preset condition in the embodiment of the present application is related to the area of the detection frame. When setting the preset condition, if the total area of each detection frame meets the preset condition, it is determined that the number of background feature points is sufficient at this time, therefore, the steps of S204-S210 are used for distinguishing moving and static objects. If the total area of each detection frame does not meet the preset condition, it is determined that the number of background feature points is small at this time, which is not representative enough, therefore, the steps of S211-S214 are used for distinguishing moving and static objects. According to the total area of each detection frame, different ways are selected to distinguish moving and static feature points, which effectively improves the accuracy of the detection result.

[0068] S204, based on the sparse optical flow method, processing the first background feature point set and the previous frame image to obtain a second background feature point set and a third background feature point set.

[0069] Among them, the background feature points in the second background feature point set are background feature points in the current frame image, and the background feature points in the third background feature point set are background feature points in the previous frame image.

[0070] In the embodiment, the second background feature point set and the third background feature point set are both sets composed of image feature points representing the background, the number of background feature points in the second background feature point set and the third background feature point set is the same, and they are one-to-one corresponding.

[0071] The optical flow method is a method of finding the corresponding relationship between the previous frame and the current frame by using the change of pixels in the time domain and the correlation between adjacent frames, so as to calculate the motion information of objects between adjacent frames. The sparse optical flow method only calculates the motion of a small number of feature points, therefore, it has higher calculation efficiency and less memory requirement. In the embodiment of the present application, by extracting feature points from the current frame image, the number of feature points can be reduced, and then the data in two frames of images is processed by the sparse optical flow method to determine the motion change of the feature points.

[0072] The first background feature point set and the previous frame image are processed based on the sparse optical flow method, to determine background feature points in the first background feature point set that are one-to-one corresponding and matched with the previous frame image, to form a second background feature point set and a third background feature point set. When processing based on the sparse optical flow method, due to the movement of vehicles, robots and other equipment, there are background feature points in the first background feature point set that do not have corresponding feature points in the previous frame image. Therefore, the number of background feature points in the second background feature point set in the embodiment of the present application is less than or equal to the number of background feature points in the first background feature point set, that is, the second background feature point set is formed by the remaining background feature points after filtering out the background feature points in the first background feature point set that do not have matching background feature points in the previous frame image.

[0073] S205, subtracting the second background feature point set and the third background feature point set to obtain a first displacement vector set.

[0074] The coordinates of the one-to-one corresponding background feature points in the second background feature point set and the third background feature point set are subtracted to obtain a first displacement. In the embodiment of the present application, the coordinate subtraction can be the difference between the horizontal coordinates and the difference between the vertical coordinates, so the obtained first displacement includes the displacement in the horizontal direction and the displacement in the vertical direction. The first displacement vector set is formed based on the first displacement corresponding to each pair of matched background feature points.

[0075] S206, for each detection frame, the first object feature point set corresponding to the detection frame and the previous frame image are processed based on the sparse optical flow method to obtain a second object feature point set and a third object feature point set.

[0076] The object feature points in the second object feature point set are object feature points in the current frame image, and the object feature points in the third object feature point set are object feature points in the previous frame image.

[0077] In the embodiment, the second object feature point set and the third object feature point set are both sets composed of image feature points representing objects, and the number of object feature points in the second object feature point set and the third object feature point set is the same and one-to-one corresponding.

[0078] For each detection frame, the corresponding object feature point can be determined by S206-S210. The first object feature point set and the previous frame image are processed based on the sparse optical flow method to determine the object feature points in the first object feature point set that are one-to-one corresponding to the object feature points in the previous frame image, thereby forming a second object feature point set and a third object feature point set. Similarly, the number of object feature points in the second object feature point set in the embodiment of the application is less than or equal to the number of object feature points in the first object feature point set, that is, the second object feature point set is formed by the remaining object feature points after the object feature points that are not found to match in the previous frame image are filtered out from the first object feature point set.

[0079] S207, subtract the second object feature point set and the third object feature point set to obtain a second displacement vector set.

[0080] The coordinates of the one-to-one corresponding object feature points in the second object feature point set and the third object feature point set are subtracted to obtain a second displacement. Similarly, the coordinate subtraction in the embodiment of the application can be the difference between the horizontal coordinates and the difference between the vertical coordinates, and therefore the obtained second displacement includes the displacement in the horizontal coordinate direction and the displacement in the vertical coordinate direction. The second displacement vector set is formed based on each pair of matched object feature points.

[0081] S208, determine the displacement mean and the displacement threshold of the background region according to the first displacement vector set.

[0082] In this embodiment, the displacement mean can be specifically understood as the average value of the movement distances of all background feature points, and the displacement threshold can be specifically understood as the threshold for determining whether the movement distance of the background feature point is too large.

[0083] The movement distances of all background feature points in the background region are calculated based on the first displacements in the first displacement vector set. In the embodiment of the application, the movement distance of the background feature point can be calculated according to the L1 norm, the L2 norm, the L3 norm, etc. The average movement distance is further calculated according to the movement distance to obtain the displacement mean. The maximum difference, the minimum difference, the standard deviation, etc. are calculated according to the first displacements to obtain the displacement threshold.

[0084] As an optional embodiment of this embodiment, the displacement mean and the displacement threshold of the background region determined according to the first displacement vector set are further optimized as follows:

[0085] B1, for each first displacement in the first displacement vector set, the L2 norm of the first displacement is calculated.

[0086] For each first displacement in the first displacement vector set, the L2 norm of the first displacement is calculated, that is, the square root of the sum of the squares of the horizontal coordinate and the vertical coordinate of the first displacement, and the result is the L2 norm of the first displacement.

[0087] B2, the average of the L2 norms of the first displacements is determined as the displacement mean of the background region.

[0088] According to the L2 norms of the first displacements, the average of the L2 norms is calculated, and the result is taken as the displacement mean of the background region.

[0089] B3, the standard deviation of the L2 norms of the first displacements is determined as the displacement threshold of the background region.

[0090] According to the L2 norms of the first displacements, the standard deviation of the L2 norms is calculated, and the result is taken as the displacement threshold of the background region.

[0091] S209, the displacement of each object feature point corresponding to the second displacement vector set is determined.

[0092] The second displacement vector set includes at least one second displacement corresponding to an object feature point, and the distance moved by the object feature point is calculated based on the second displacement. In the embodiments of the present application, the distance moved by the object feature point can be calculated according to the L1 norm, the L2 norm, the L3 norm, etc.

[0093] As an optional embodiment of the present embodiment, the displacement of each object feature point included in the second displacement vector set is further optimized as follows:

[0094] C1, the second displacement corresponding to each object feature point in the second displacement vector set is determined, and the L2 norm of the second displacement is calculated.

[0095] The square root of the sum of the squares of the horizontal coordinate and the vertical coordinate of the second displacement is taken as the L2 norm of the second displacement, and the L2 norm of the second displacement corresponding to each object feature point is calculated according to the above method.

[0096] C2, the L2 norm of the second displacement is determined as the displacement of the object feature point.

[0097] S210, for each object feature point, the difference between the displacement of each object feature point and the displacement mean is calculated, and if the absolute value of the difference is not less than the displacement threshold, the object feature point is determined as a dynamic object feature point.

[0098] For each object feature point in each detection box, the following method can be used for dynamic and static distinction: the difference between the displacement of the object feature point and the average displacement is calculated, the absolute value of the difference is compared with the displacement threshold, if it is not less than the displacement threshold, the object feature point is determined as a dynamic object feature point; if it is less than the displacement threshold, the object feature point is determined as a static feature point.

[0099] In the step of judging the dynamic and static state of the object feature points, if the number of the object feature points in the second object feature point set is less than the number of the object feature points in the first object feature point set, only the object feature points matched in the last frame image can be detected for dynamic and static state, that is, only whether the object feature points in the second object feature point set are dynamic is judged. Alternatively, for the object feature point N1 not matched in the last frame image, the object feature point N2 closest to it in the current frame image can be determined according to the coordinates, and the dynamic and static state of the object feature point N1 is determined according to the dynamic and static state of the object feature point N2, that is, when the object feature point N2 is a dynamic object feature point, the object feature point N1 is also a dynamic object feature point, and when the object feature point N2 is a static object feature point, the object feature point N1 is also a static object feature point.

[0100] S211, determine the fundamental matrix between the current frame image and the last frame image.

[0101] The relative pose and the relative rotation between the current frame image and the last frame image are obtained, and the relative pose and the relative rotation can be obtained based on an RTK device, a wheel speed meter or other devices or sensors. The relative pose and the relative rotation between the two frames are determined based on the relative pose and the relative rotation between the two frames.

[0102] For example, the embodiment of the present application provides a formula for determining the fundamental matrix:

[0103] F=K -T ([t] x R) T K -1 ;

[0104] Wherein, K represents the camera intrinsic parameter, t represents the relative pose, [t] x represents the skew-symmetric matrix of the relative pose t between the two frames, and R represents the relative rotation between the two frames.

[0105] S212, for each detection box corresponding to an object, the object feature points in the detection box and the last frame image are processed based on the sparse optical flow method to obtain a matching point set corresponding to the last frame image.

[0106] In the embodiment, the matching point set can be understood as a set of object feature points in the last frame image which are matched one by one with the object feature points in the detection box.

[0107] For each detection frame, the corresponding matching point set can be determined in the following manner: based on the sparse optical flow method, the object feature points in the first object feature point set corresponding to the detection frame and the previous frame image are processed to determine the object feature points in the detection frame that are one-to-one corresponding to the matching object feature points in the previous frame image, thereby forming the matching point set. The object feature points in the matching point set obtained in this step have one-to-one corresponding object feature points in the detection frame.

[0108] S213, for each object feature point in the detection frame, determine the target matching feature point corresponding to the object feature point in the matching point set.

[0109] In this embodiment, the target matching feature point can be understood as an object feature point in the matching point set, which can be determined according to the object feature points in the detection frame. After determining an object feature point, the target matching feature point corresponding to the object feature point in the matching point set can be determined according to the corresponding relationship.

[0110] S214, determine the positional relationship of the object feature point in the two frames of images based on the coordinates of the object feature point and the target matching feature point and the fundamental matrix, and determine the object feature point as a dynamic object feature point if the positional relationship of the object feature point in the two frames of images does not satisfy the preset positional condition.

[0111] In this embodiment, the preset positional condition can be whether the positions of two points coincide or coincide within an error tolerance range. The coordinates of the object feature point and the target matching feature point and the fundamental matrix are brought into the calculation formula to obtain the positional relationship of the object feature point in the two frames of images. It is determined whether the positional relationship of the object feature point in the two frames of images satisfies the preset positional condition, if it satisfies, the object feature point is determined as a static feature point; if it does not satisfy, the object feature point is determined as a dynamic object feature point.

[0112] For example, the embodiment of the present application provides a formula for determining whether an object feature point is a dynamic object feature point:

[0113] pFp* < thre;

[0114] Where p is the coordinate of the object feature point, F is the fundamental matrix, p* is the coordinate of the target matching feature point, and pFp* represents the positional relationship of the object feature point in the two frames of images; thre is a preset threshold value, which can be set according to the speed of the vehicle or robot and the measurement accuracy of the RTK receiver. When pFp* < thre, it is determined that the preset positional condition is satisfied.

[0115] In the embodiment of the present application, the corresponding two object feature points can be substituted into the formula for calculation, and it is judged whether it is a dynamic point or a static point according to the calculation result. For a point on an object, it is collected at two times respectively, and the coordinates of two points can be collected. If the point is dynamic, its actual position has changed, so the coordinates have changed in the same coordinate system, and it is no longer the same point, so the calculated value is greater than the threshold.

[0116] The embodiment of the present application provides a dynamic object feature point detection method, which solves the problem of inaccurate detection result when a monocular camera detects a dynamic object. The object feature points in the current frame image are detected to obtain a detection frame. A first background feature point set composed of background feature points and a first object feature point set corresponding to each detection frame are determined according to the detection frame. Different processing modes are selected for dynamic and static differentiation according to whether the total area of each detection frame meets a preset condition. When the preset condition is met, the dynamic object is detected based on the first background feature point set. When the preset condition is not met, the dynamic and static differentiation is performed only according to the object feature points because the number of background feature points is small and not representative. When the dynamic and static feature points are detected, the detection result is more accurate, the object feature points in the detection frame can be judged again, and the false detection situation is reduced. The image is collected by a monocular camera, the detection cost is low, and the robustness is higher because the depth of the feature points is not dependent.

[0117] Embodiment three

[0118] Figure 3 A structural schematic diagram of a dynamic object feature point detection device provided for the embodiment three of the present application is shown in FIG. 3. As shown in the figure, the device comprises a feature point extraction module 31, an object detection module 32, a first displacement set determination module 33 and a first dynamic point detection module 34. Figure 3

[0119] The feature point extraction module 31 is configured to acquire a current frame image and a previous frame image, and extract feature points from the current frame image to obtain image feature points. The current frame image is collected by a monocular camera.

[0120] The object detection module 32 is configured to detect objects from the image feature points, determine a detection frame corresponding to at least one object, and determine a first background feature point set and a first object feature point set corresponding to each detection frame according to each detection frame.

[0121] The first displacement set determination module 33 is configured to determine a first displacement vector set corresponding to background feature points according to the first background feature point set and the previous frame image if the total area of each detection frame meets a preset condition. ​

[0122] The first dynamic point detection module 34 is configured to, for each detection frame, determine a second displacement vector set corresponding to the object according to the first object feature point set corresponding to the detection frame and the previous frame image, and determine whether the object feature point in the detection frame is a dynamic object feature point based on the first displacement vector set and the second displacement vector set.

[0123] The embodiment of the present application provides a dynamic object feature point detection device, which solves the problem of inaccurate detection result when a monocular camera detects a dynamic object, and through object detection on image feature points in a current frame image, a detection frame is obtained, a first background feature point set composed of background feature points is further determined according to the detection frame, and a first object feature point set corresponding to each detection frame is determined, when the total area of each detection frame meets a preset condition, dynamic object detection is performed based on the first background feature point set, and the object feature points in the detection frame are distinguished between dynamic and static through a first displacement vector set and a second displacement vector set, so that the accuracy of the detection result is improved, and the occurrence of false detection is reduced; image acquisition is performed through a monocular camera, and the detection cost is low.

[0124] Optionally, the object detection module 32 is specifically configured to: for each image feature point, determine whether the image feature point is in a detection frame according to the coordinates of the image feature point and the size and position information of each detection frame; if yes, the image feature point is added to the first object feature point set corresponding to the detection frame in which the image feature point is located as an object feature point; otherwise, the image feature point is added to the first background feature point set as a background feature point.

[0125] Optionally, the first displacement set determination module 33 comprises:

[0126] The first background set determination unit is configured to process the first background feature point set and the previous frame image based on a sparse optical flow method to obtain a second background feature point set and a third background feature point set, the background feature points in the second background feature point set are background feature points in the current frame image, and the background feature points in the third background feature point set are background feature points in the previous frame image.

[0127] The first displacement set determination unit is configured to subtract the second background feature point set and the third background feature point set to obtain the first displacement vector set.

[0128] Optionally, the first dynamic point detection module 34 comprises:

[0129] The first object set determination unit is configured to process the first object feature point set corresponding to the detection frame and the previous frame image based on the sparse optical flow method, to obtain a second object feature point set and a third object feature point set, wherein the object feature points in the second object feature point set are object feature points in the current frame image, and the object feature points in the third object feature point set are object feature points in the previous frame image.

[0130] The second displacement set determination unit is configured to subtract the second object feature point set from the third object feature point set, to obtain a second displacement vector set.

[0131] Optionally, the first dynamic point detection module 34 comprises:

[0132] The background displacement determination unit is configured to determine a displacement mean value and a displacement threshold value of a background region according to the first displacement vector set.

[0133] The object displacement determination unit is configured to determine a displacement of each object feature point in the second displacement vector set.

[0134] The feature point determination unit is configured to, for each object feature point, calculate a difference value between the displacement of the object feature point and the displacement mean value, and determine the object feature point as a dynamic object feature point if the absolute value of the difference value is not less than the displacement threshold value.

[0135] Optionally, the background displacement determination unit is specifically configured to: for each first displacement in the first displacement vector set, calculate an L2 norm of the first displacement; determine an average value of the L2 norms of the first displacements as the displacement mean value of the background region; and determine a standard deviation of the L2 norms of the first displacements as the displacement threshold value of the background region.

[0136] Optionally, the object displacement determination unit is specifically configured to determine a second displacement corresponding to each object feature point in the second displacement vector set, calculate an L2 norm of the second displacement, and determine the L2 norm of the second displacement as the displacement of the object feature point.

[0137] Optionally, the device further comprises:

[0138] The fundamental matrix determination module is configured to, if the total area of the detection frames does not satisfy a preset condition, determine a fundamental matrix between the current frame image and the previous frame image.

[0139] The matching point set determination module is configured to, for each detection frame corresponding to an object, process object feature points in the detection frame and a previous frame image based on the sparse optical flow method, to obtain a matching point set corresponding to the previous frame image.

[0140] A target point determination module is configured to determine, for each object feature point in the detection frame, a target matching feature point corresponding to the object feature point in the matching point set.

[0141] A second dynamic point detection module is configured to determine a positional relationship of the object feature point in two frames of images based on the coordinates of the object feature point and the target matching feature point and the fundamental matrix, and determine the object feature point as a dynamic object feature point if the positional relationship of the object feature point in the two frames of images does not satisfy a preset positional condition.

[0142] The dynamic object feature point detection device provided by the embodiment of the present application can execute the dynamic object feature point detection method provided by any embodiment of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0143] Embodiment four

[0144] Figure 4 A structural schematic diagram of an electronic device 40 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0145] As shown in Figure 4 The electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., which are communicatively connected to the at least one processor 41, wherein the memory stores a computer program executable by the at least one processor. The processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 into the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0146] A plurality of components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a magnetic disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0147] The processor 41 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 performs various methods and processes described above, such as the dynamic object feature point detection method.

[0148] In some embodiments, the dynamic object feature point detection method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 48. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded onto the RAM 43 and executed by the processor 41, one or more steps of the dynamic object feature point detection method described above can be performed. Alternatively, in other embodiments, the processor 41 can be configured to perform the dynamic object feature point detection method by any other appropriate means, such as by means of firmware.

[0149] The various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0150] Computer programs for implementing the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, enables the functions / acts specified in the flowcharts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine or entirely on a remote machine or server.

[0151] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0152] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0153] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0154] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0155] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, and the present disclosure is not limited herein as such.

[0156] The specific embodiments described above are not intended to limit the scope of the present disclosure. Those skilled in the art will understand that various modifications, combinations, sub-combinations, and alternatives can be made to the specific embodiments without departing from the spirit and principles of the present disclosure. Any further modifications, equivalents, and / or alternatives come within the scope of the present disclosure as recited by the claims.

Claims

1. A method for detecting feature points of a dynamic object, characterized in that, include: The current frame image and the previous frame image are acquired, and feature points are extracted from the current frame image to obtain image feature points. The current frame image is acquired by a monocular camera. Object detection is performed on the image feature points to determine at least one detection box corresponding to an object, and a first set of background feature points and a first set of object feature points corresponding to each detection box are determined based on each detection box. If the total area of ​​each detection box meets the preset condition, the first displacement vector set corresponding to the background feature points is determined according to the first background feature point set and the previous frame image; For each detection box, a second set of displacement vectors corresponding to the object is determined based on the first set of object feature points corresponding to the detection box and the previous frame image. Based on the first set of displacement vectors and the second set of displacement vectors, it is determined whether the object feature points in the detection box are dynamic object feature points.

2. The method according to claim 1, characterized in that, The first set of displacement vectors corresponding to the background feature points is determined based on the first set of background feature points and the previous frame image, including: Based on the sparse optical flow method, the first set of background feature points and the previous frame image are processed to obtain the second set of background feature points and the third set of background feature points. The background feature points in the second set of background feature points are the background feature points in the current frame image, and the background feature points in the third set of background feature points are the background feature points in the previous frame image. Subtract the second set of background feature points from the third set of background feature points to obtain the first set of displacement vectors.

3. The method according to claim 1, characterized in that, The second set of displacement vectors corresponding to the object is determined based on the first set of object feature points corresponding to the detection box and the previous frame image, including: The first set of object feature points corresponding to the detection box and the previous frame image are processed based on the sparse optical flow method to obtain the second set of object feature points and the third set of object feature points. The object feature points in the second set of object feature points are the object feature points in the current frame image, and the object feature points in the third set of object feature points are the object feature points in the previous frame image. Subtract the second set of object feature points from the third set of object feature points to obtain the second set of displacement vectors.

4. The method according to claim 1, characterized in that, The step of determining whether an object feature point in the detection frame is a dynamic object feature point based on the first set of displacement vectors and the second set of displacement vectors includes: The mean displacement and displacement threshold of the background region are determined based on the first set of displacement vectors. Determine the displacement of each object feature point corresponding to the second set of displacement vectors; For each object feature point, calculate the difference between the displacement of each object feature point and the mean displacement. If the absolute value of the difference is not less than the displacement threshold, the object feature point is determined to be a dynamic object feature point.

5. The method according to claim 4, characterized in that, Determining the mean displacement and displacement threshold of the background region based on the first set of displacement vectors includes: For each first displacement in the first set of displacement vectors, calculate the L2 norm of the first displacement; The average value of the L2 norm of each of the first displacements is determined as the average displacement value of the background region; The standard deviation of the L2 norm of each of the first displacements is determined as the displacement threshold of the background region.

6. The method according to claim 4, characterized in that, Determining the displacement of each object feature point included in the second set of displacement vectors includes: Determine the second displacement corresponding to each object feature point in the second displacement vector set, and calculate the L2 norm of the second displacement; The L2 norm of the second displacement is determined as the displacement of the object's feature point.

7. The method according to any one of claims 1-6, characterized in that, Also includes: If the total area of ​​all the detection boxes does not meet the preset condition, determine the fundamental matrix between the current frame image and the previous frame image; For each object's detection box, the object feature points in the detection box and the previous frame image are processed based on the sparse optical flow method to obtain a set of matching points corresponding to the previous frame image. For each object feature point in the detection frame, determine the target matching feature point corresponding to the object feature point in the matching point set; Based on the coordinates of the object feature points and the target matching feature points, and combined with the fundamental matrix, the positional relationship of the object feature points in the two frames of images is determined. If the positional relationship of the object feature points in the two frames of images does not meet the preset positional conditions, the object feature points are determined to be dynamic object feature points.

8. A dynamic object feature point detection device, characterized in that, include: The feature point extraction module is used to acquire the current frame image and the previous frame image, extract feature points from the current frame image to obtain image feature points, wherein the current frame image is acquired by a monocular camera; An object detection module is used to perform object detection on the image feature points, determine at least one detection box corresponding to an object, and determine a first set of background feature points and a first set of object feature points corresponding to each detection box based on each detection box. The first displacement set determination module is used to determine the first displacement vector set corresponding to the background feature points based on the first background feature point set and the previous frame image if the total area of ​​each detection box meets the preset condition. The first dynamic point detection module is used to determine, for each detection box, a second set of displacement vectors corresponding to the object based on the first set of object feature points corresponding to the detection box and the previous frame image, and determine whether the object feature points in the detection box are dynamic object feature points based on the first set of displacement vectors and the second set of displacement vectors.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the dynamic object feature point detection method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the dynamic object feature point detection method according to any one of claims 1-7.

Images