Moving object detection method and device, electronic equipment and storage medium

Abstract

The invention provides a moving object detection method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining the point cloud data and image data of an environment where a vehicle is located, and enabling the image data to comprise a plurality of continuous frames of images; feature recognition is performed according to the point cloud data, and object semantic information corresponding to a plurality of objects included in the environment where the vehicle is located is determined; performing optical flow tracking on the multi-frame image, and determining reference matching point pairs corresponding to the plurality of objects in the multi-frame image; classifying the reference matching point pairs corresponding to the plurality of objects according to the object semantic information corresponding to the plurality of objects, and determining a plurality of first matching point pairs and a plurality of second matching point pairs; and according to the wheel speedometer information of the vehicle and the plurality of first matching point pairs, screening out a target matching point pair corresponding to the moving object from the plurality of second matching point pairs. According to the invention, the detection accuracy of the dynamic object is ensured.

Description

Technical Field

[0001] This application relates to the field of image technology, and more particularly, to a method, apparatus, electronic device, and storage medium for detecting moving objects. Background Art

[0002] With the rapid development of technology, the application of intelligent transportation systems and vehicle autonomous driving technology in daily life has become increasingly widespread. Currently, various sensing technologies, such as lidar (LiDAR), vision sensors, wheel speed sensors, and ultrasonic sensors, are commonly used for vehicle environmental perception and dynamic object detection. However, each of these sensors has its own advantages and limitations. In the prior art, a perception system relying on a single sensor often has difficulty accurately distinguishing dynamic objects from static backgrounds in a dynamic environment, resulting in inaccurate identification of dynamic objects in the vehicle's environment. Therefore, how to accurately detect dynamic objects in the vehicle's environment has become an urgent problem to be solved. Summary of the Invention

[0003] In view of this, embodiments of this application propose a method, apparatus, electronic device, and storage medium for detecting moving objects to improve the above problems.

[0004] According to the first aspect of the embodiments of this application, a method for detecting a moving object is provided. The method includes: obtaining point cloud data and image data of the vehicle's environment, where the image data includes a plurality of consecutive frames of images; performing feature recognition on the point cloud data to determine the object semantic information corresponding to each of the plurality of objects included in the vehicle's environment; performing optical flow tracking on the plurality of frames of images to determine the reference matching point pairs corresponding to each of the plurality of objects in the plurality of frames of images; classifying the reference matching point pairs corresponding to each of the plurality of objects according to the object semantic information corresponding to each of the plurality of objects to determine a plurality of first matching point pairs and a plurality of second matching point pairs, where the first matching point pairs are the matching point pairs corresponding to the background objects among the plurality of objects, and the second matching point pairs are the matching point pairs corresponding to the objects other than the background objects among the plurality of objects; and screening out the target matching point pairs corresponding to the moving objects from the plurality of second matching point pairs according to the vehicle's wheel speed sensor information and the plurality of first matching point pairs.

[0005] According to a second aspect of the embodiments of the present application, a detection device for a moving object is provided. The device includes: an acquisition module configured to acquire point cloud data and image data of the environment where the vehicle is located, where the image data includes a plurality of consecutive frames of images; an identification module configured to perform feature identification on the point cloud data to determine the object semantic information corresponding to each of the plurality of objects included in the environment where the vehicle is located; a reference matching point pair determination module configured to perform optical flow tracking on the plurality of frames of images to determine the reference matching point pairs corresponding to each of the plurality of objects in the plurality of frames of images; a classification module configured to classify the reference matching point pairs corresponding to each of the plurality of objects according to the object semantic information corresponding to each of the plurality of objects to determine a plurality of first matching point pairs and a plurality of second matching point pairs, where the first matching point pairs are the matching point pairs corresponding to the background objects among the plurality of objects, and the second matching point pairs are the matching point pairs corresponding to the objects other than the background objects among the plurality of objects; and a moving object determination module configured to screen out the target matching point pairs corresponding to the moving objects from the plurality of second matching point pairs according to the vehicle's wheel speed meter information and the plurality of first matching point pairs.

[0006] According to a third aspect of the embodiments of the present application, an electronic device is provided, including: a processor; a memory storing computer-readable instructions thereon, and when the computer-readable instructions are executed by the processor, the detection method for a moving object as described above is implemented.

[0007] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, having computer-readable instructions stored thereon, and when the computer-readable instructions are executed by a processor, the detection method for a moving object as described above is implemented.

[0008] In the solution of the present application, the object semantic information corresponding to each of the plurality of objects in the measurement environment is determined by identifying the acquired point cloud data, and optical flow tracking is performed on the acquired plurality of consecutive frames of images to determine the reference matching points corresponding to each of the plurality of objects. Then, according to the object semantic information, the reference matching points are screened to obtain a plurality of first matching points and a plurality of second matching points. Then, the vehicle's wheel speed meter information and the plurality of first matching point pairs are used to screen the plurality of second matching points to determine the target matching points of the moving objects. The present application determines the detection of the moving objects in the environment where the vehicle is located through the multi-sensor data fusion of the point cloud data, the image data, and the wheel speed meter information, ensuring the detection accuracy of the dynamic objects, and further improving the detection accuracy of the dynamic objects by screening the plurality of second matching points that are non-background points through the wheel speed meter information.

[0009] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present invention. Description of the Drawings

[0010] The accompanying drawings here are incorporated into the description and form a part of this description, showing embodiments consistent with this application, and are used together with the description to explain the principles of this application. Obviously, the accompanying drawings in the following description are only some embodiments of this application, and those of ordinary skill in the art can obtain other accompanying drawings based on these drawings without creative efforts.

[0011] Figure 1 It is a schematic flowchart of a method for detecting a moving object shown according to an embodiment of this application.

[0012] Figure 2 It is a schematic diagram of object center positioning for point cloud data shown according to an embodiment of this application.

[0013] Figure 3 It is a schematic diagram of virtual optical flow tracking for image data shown according to an embodiment of this application.

[0014] Figure 4 It is a schematic flowchart of a method for detecting a moving object shown according to another embodiment of this application.

[0015] Figure 5 It is a schematic flowchart of a method for detecting a moving object shown according to yet another embodiment of this application.

[0016] Figure 6 It is a schematic flowchart of a method for detecting a moving object shown according to still another embodiment of this application.

[0017] Figure 7 It is a schematic diagram of a three-dimensional bounding box corresponding to a moving object in point cloud data shown according to an embodiment of this application.

[0018] Figure 8 It is a schematic flowchart of a method for detecting a moving object shown according to yet another embodiment of this application.

[0019] Figure 9 It is a schematic flowchart of a method for detecting a moving object shown according to yet another embodiment of this application.

[0020] Figure 10 It is a block diagram of a device for detecting a moving object shown according to an embodiment of this application.

[0021] Figure 11 It is a hardware structure diagram of an electronic device shown according to an embodiment of this application.

[0022] Through the above-mentioned drawings, specific embodiments of the present invention have been shown, and will be described in more detail hereinafter. These drawings and the written description are not intended to limit the scope of the inventive concept in any way, but to illustrate the concept of the present invention to those skilled in the art through specific embodiments. Detailed Description of Specific Embodiments

[0023] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more complete and comprehensive, and will fully convey the concept of the example embodiments to those skilled in the art.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and do not necessarily have to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order different from those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0025] In addition, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the embodiments of the present application. However, those skilled in the art will realize that the technical solutions of the present application can be practiced without one or more of the specific details, or other methods, devices, steps, etc. can be adopted. In other cases, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the present application.

[0026] The block diagrams shown in the drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices. The flowcharts shown in the drawings are only exemplary illustrations and do not necessarily include all the content and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined, so the actual execution order may change according to the actual situation.

[0027] Please refer toFigure 1 , Figure 1 shows the detection method of a moving object provided by an embodiment of the present application. In a specific embodiment, the detection method of the moving object can be applied to a moving object detection device 600 as shown in Figure 10 and an electronic device 700 equipped with the moving object detection device 600 ( Figure 11 ). The specific process of this embodiment will be described below. Of course, it can be understood that this method can be executed by an in-vehicle terminal with computing and processing capabilities. The following will elaborate in detail on the Figure 1 shown process. The detection method of the moving object may specifically include the following steps:

[0028] Step 110: Obtain point cloud data and image data of the vehicle's environment, where the image data includes multiple consecutive frames of images.

[0029] As a way, three-dimensional point cloud data of the vehicle's environment can be collected by a high-precision lidar device installed on the vehicle, and a continuous image sequence can be captured by the vehicle's image acquisition device, so as to obtain multiple consecutive frames of images. Optionally, the image acquisition device can be a camera with a high frame rate and high resolution, so that the collected continuous image sequence has the characteristics of high definition and temporal continuity.

[0030] Optionally, before obtaining the point cloud data and image data of the vehicle's environment, the lidar and image acquisition device of the vehicle can be synchronized and calibrated with the vehicle in advance to ensure the accuracy of the obtained point cloud data and image data, and thus ensure that accurate spatial position information can be obtained.

[0031] Step 120: Perform feature recognition on the point cloud data to determine the object semantic information corresponding to each of the multiple objects included in the vehicle's environment.

[0032] As a way, a neural network can be used to extract the key features corresponding to the point cloud data, and then perform feature recognition on the point cloud data based on the key features. Then, the neural network outputs an object point cloud with semantic labels (such as vehicles, pedestrians, trees, or street lights, etc.), and thus the object semantic information corresponding to each of the multiple objects included in the vehicle's environment can be determined based on the object point cloud with semantic labels. Optionally, the object semantic information can be an indication of the attributes or categories corresponding to the objects, etc.

[0033] Optionally, feature extraction can be performed on the point cloud data first to obtain the features (such as height, intensity, or reflectivity, etc.) corresponding to each of the multiple point clouds in the point cloud data. Then, a pre-trained deep learning model (such as CenterPoint) can be used to perform object center localization on the point cloud data based on the features corresponding to the determined multiple point clouds, as Figure 2 shown. Then, feature recognition is performed on the object located at the center to infer the semantic category of the multiple point clouds, thereby determining the object semantic information corresponding to each of the multiple objects in the vehicle's environment.

[0034] Optionally, before performing feature recognition on the point cloud data, preprocessing can be performed on the point cloud data. For example, filtering and noise reduction, ground segmentation, and / or data dimensionality reduction, etc. are performed on the point cloud data to improve the recognition accuracy rate when performing feature recognition on the point cloud data subsequently, and to avoid the problem of low recognition efficiency caused by data errors. Optionally, statistical analysis methods can also be used to detect and remove abnormal point cloud data in the point cloud data to ensure the quality of the point cloud data.

[0035] Step 130, perform optical flow tracking on the multiple frames of images to determine the reference matching point pairs corresponding to each of the multiple objects in the multiple frames of images.

[0036] As a method, feature point detection can be first performed on the multiple frames of images to determine the feature points in a certain frame of image. Then, the feature points in this frame of image are tracked according to the optical flow tracking algorithm to determine the positions of the same feature points in different frames of images, thereby obtaining the reference matching point pairs corresponding to each of the multiple objects in the multiple frames of images.

[0037] Optionally, the Shi-Tomasi corner detection method can be used to perform feature point detection on the multiple frames of images to determine the feature points in the images. Optionally, a certain frame of image in the multiple frames of images can be first converted into a grayscale image, and then the gradients in different directions (such as the directions corresponding to different coordinate axes of the image coordinate system) are determined for each pixel point in the grayscale image. Then, the autocorrelation matrix of each pixel point is determined based on the gradients of each pixel point in different directions and the coordinates of each pixel point in the image coordinate system, and the first eigenvalue and the second eigenvalue of the autocorrelation matrix are determined. Thereby, the minimum eigenvalue is selected from the first eigenvalue and the second eigenvalue to calculate the corner response degree, and finally the corner response degrees of all pixel points are sorted to determine the feature points.

[0038] Optionally, after determining the feature points, the determined feature points can be tracked by the Pyramid Lucas-Kanade method. Optionally, since the brightness values corresponding to the same object in different frame images can be considered invariant in the image, feature points with the same gray value as the feature points can be determined in other frame images, so as to determine the reference matching point pairs corresponding to each of the multiple objects. As Figure 3 shown, the determined multiple feature points can be tracked by the optical flow tracking method to determine multiple reference matching point pairs.

[0039] Step 140: Classify the reference matching point pairs corresponding to each of the multiple objects according to the object semantic information corresponding to each of the multiple objects, and determine multiple first matching point pairs and multiple second matching point pairs, where the first matching point pairs are the matching point pairs corresponding to the background objects among the multiple objects, and the second matching point pairs are the matching point pairs corresponding to the objects other than the background objects among the multiple objects.

[0040] As a way, after determining multiple reference matching point pairs, the object semantic information corresponding to each of the multiple objects determined according to the point cloud data and the objects corresponding to the multiple reference matching points in the image can be used to determine the object semantic information matched by the multiple reference matching point pairs, so as to classify the reference matching point pairs and obtain the first matching point pairs and the second matching point pairs.

[0041] Optionally, if the object semantic information indicates that there are point pairs corresponding to background objects such as trees, street lights, and road signs in the reference matching point pairs, then this point pair can be determined as the first matching point pair; if the object semantic information indicates that there are point pairs that are not background objects in the reference matching point pairs, then this point pair is determined as the second matching point pair.

[0042] Step 150: Screen out the target matching point pairs corresponding to the moving objects from the multiple second matching point pairs according to the wheel speedometer information of the vehicle and the multiple first matching point pairs.

[0043] As a way, in order to determine the moving objects in the environment where the vehicle is located, the pose and the pose of the vehicle can be inferred first according to the wheel speedometer information of the vehicle, and then the screening conditions for screening the multiple second matching point pairs can be determined based on the displacement and the pose of the vehicle and the first matching point pairs, and then the second matching point pairs are screened according to the determined screening conditions to obtain the target matching point pairs corresponding to the moving objects.

[0044] Optionally, the method of the FVB limit can be used to determine the range of the optical flow vectors in the restricted image based on the vehicle pose and the first matching point pair, so as to further screen the second matching point pair based on the range. Since the change of the point pairs of non-moving objects in the optical flow vectors in multiple frames is within a certain range, the point pairs that do not change in multiple frames in the second matching point pair can be deleted through this range, and the target matching point pair of the moving object can be obtained.

[0045] As a way, after determining the target matching point pair of the moving object, the corresponding point cloud in the point cloud data can be determined, and then the point cloud corresponding to the moving object can be deleted, so as to calculate the initial pose of the vehicle based on the deleted point cloud data of the moving object, image data and wheel speedometer information, thereby reducing the state estimation error and improving the overall performance of the system.

[0046] As a way, after determining the target matching point pair of the moving object, the target matching point pair can be projected into the three-dimensional space to obtain the position of the target matching point pair in the three-dimensional space, and then the point cloud corresponding to the position of the target matching point pair in the three-dimensional space can be determined in the point cloud data, and these point clouds can be deleted to obtain the target point cloud data. Then, the three-dimensional reconstruction and mapping of the scene are performed through the target point cloud data to obtain a static scene, so as to provide a stable reference through the reconstructed static scene for the vehicle navigation system and future path planning, and improve the navigation accuracy and safety of the entire system.

[0047] As another way, the detection method of the moving object can be integrated into the real-time system to ensure that the time efficiency of each step of the detection method of the moving object meets the real-time requirements, so as to improve the accuracy of navigation and self-state estimation in a complex dynamic environment. Optionally, the key hardware devices of the vehicle can be configured and calibrated in advance. Among them, the key hardware devices can include lidar, camera, computing unit, communication interface, etc. These devices need to work together, and they are configured and calibrated to ensure the time synchronization of all sensors, so as to facilitate the accurate collection and processing of data. Optionally, the algorithms required in the detection method of the moving object can be integrated into the computing module, such as lidar data processing, visual data processing, object recognition and state estimation algorithms, etc., so as to ensure that the computing module can work efficiently in cooperation, and the integrated computing module is optimized in performance after the algorithm integration, which can include reducing latency, increasing data processing speed and improving the accuracy of the algorithm, etc. And before the detection method of the moving object is applied offline, the system can be deployed in the actual environment, and the real-time operation of the system can be continuously monitored to timely discover and solve possible problems, and a user interface can be developed to enable the operator to monitor the system status, adjust the system settings and obtain key information in real time.

[0048] In an embodiment of the present application, by identifying the acquired point cloud data, the object semantic information corresponding to each of multiple objects in the measurement environment is determined, and optical flow tracking is performed on the acquired consecutive multiple frames of images to determine the reference matching points corresponding to each of the multiple objects. Then, according to the object semantic information, the reference matching points are filtered to obtain multiple first matching points and multiple second matching points. Then, the multiple second matching points are filtered by the vehicle's wheel speedometer information and the multiple first matching points to determine the target matching points of the moving object. The present application combines multi-sensor data fusion of point cloud data, image data, and wheel speedometer information to determine the detection of moving objects in the vehicle's environment, ensuring the detection accuracy of dynamic objects. And by filtering the multiple second matching points that are non-background points through the wheel speedometer information, the detection accuracy of dynamic objects is further improved.

[0049] Figure 4 shows a method for detecting a moving object provided by an embodiment of the present application. The following will be elaborated in detail for the Figure 4 process shown. The method for detecting a moving object may specifically include the following steps:

[0050] Step 210, acquire point cloud data and image data of the vehicle's environment, where the image data includes consecutive multiple frames of images.

[0051] Step 220, perform feature recognition on the point cloud data to determine the object semantic information corresponding to each of the multiple objects included in the vehicle's environment.

[0052] Step 230, perform optical flow tracking on the multiple frames of images to determine the reference matching point pairs corresponding to each of the multiple objects in the multiple frames of images; determine the displacement and pose of the vehicle according to the wheel speedometer information.

[0053] Step 240, classify the reference matching point pairs corresponding to each of the multiple objects according to the object semantic information corresponding to each of the multiple objects to determine multiple first matching point pairs and multiple second matching point pairs, where the first matching point pairs are the matching point pairs corresponding to the background objects among the multiple objects, and the second matching point pairs are the matching point pairs corresponding to the objects other than the background objects among the multiple objects.

[0054] Among them, for the specific step descriptions of steps 210 - 240, reference can be made to steps 110 - 140, and details will not be repeated here.

[0055] Step 250, determine the displacement and pose of the vehicle according to the wheel speedometer information.

[0056] As a means, a wheel speedometer, also known as a wheel speed odometer, is a commonly used device that directly obtains the vehicle speed and displacement from sensors installed on the wheels. The speed and angular velocity of the wheels can be determined through the wheel speedometer to obtain wheel speedometer information.

[0057] Optionally, the pose change can be performed according to the speed and angular velocity of the wheels in the wheel speedometer information to obtain the displacement and pose of the vehicle. Optionally, the displacement of the vehicle can be determined by the formula t 0 = vΔt, where Δt can be the duration corresponding to multiple consecutive frames of images collected, v is the speed provided by the wheel speedometer, and t 0 is the displacement of the vehicle. Optionally, the pose of the vehicle can be determined by the formula R 0 = exp(ωΔt), where ω is the angular velocity provided by the wheel speedometer, and exp(ωΔt) is to convert ωΔt into matrix form to obtain the pose of the vehicle, and this pose can indicate the rotation information of the wheel speedometer.

[0058] Step 260, determine the essential matrix according to the displacement and the pose, and determine the fundamental matrix according to the essential matrix and the multiple first matching point pairs.

[0059] As a means, after determining the displacement and pose of the vehicle, the essential matrix can be first converted according to the displacement and pose of the vehicle. Optionally, the displacement can be used as the translation vector and the pose as the rotation matrix. Therefore, the essential matrix can be determined by the formula E 0 = [t 0 × R 0 . The essential matrix can be used in the case of multi-view stereo vision to describe the corresponding relationship of the same point in space in different coordinate systems. Thus, the corresponding relationship between multiple matching points in consecutive frames of images in the image coordinate system and the world coordinate system can be determined through the essential matrix, and then the moving objects in the environment where the vehicle is located can be determined based on this corresponding relationship.

[0060] Optionally, after determining the essential matrix, the cost function of all matching point pairs can be determined through the essential matrix and multiple first matching point pairs, and the fundamental matrix can be determined by determining the optimal solution of this cost function, and then the screening of multiple second matching points can be determined based on this fundamental matrix. Optionally, its cost function can be cost = ∑d(x i+1 , Fx i ) + λ||K ―T FK ―1 ―E 0 || F , where cost is the cost function, F is the fundamental function, d is the epipolar geometric distance between the line from xi +1 to Fxi, and ||K​―TFK―1 -E 0 || F represents the Frobenius norm, λ is a weight factor, which can be a preset fixed value or determined according to the proportion of multiple first matching point pairs in the reference matching point pair. Optionally, after determining the optimal solution of the cost function, the basic function corresponding to the optimal solution is the required fundamental matrix.

[0061] Step 270, determine the signed limit distance corresponding to each of the multiple second matching point pairs according to the fundamental matrix, and screen out the target matching point pairs corresponding to the moving object from the multiple second matching point pairs according to the signed limit distance.

[0062] As a way, since the fundamental matrix is determined by multiple first matching point pairs as the background object, that is, a reliable fundamental matrix estimate F obtained by quickly converging after excluding the points of dynamic objects, therefore, the multiple second matching point pairs can be screened by this fundamental matrix, thereby ensuring the accuracy of the determined target matching point pairs.

[0063] Optionally, the corresponding epipolar line can be obtained through the first matching point pair and the estimated fundamental matrix Then simplify the coefficients of the epipolar line to obtain the coefficient matrix Then calculate the point to the signed epipolar line distance of the line l i′ The signed epipolar line distance can be determined by the formula to determine the signed epipolar line distance, where is the coordinate of any point in a pair of point pairs among multiple first matching point pairs in the image coordinate system. Optionally, the signed epipolar line distances corresponding to each of the multiple first matching point pairs can be determined first, and then the average value can be determined according to the signed epipolar line distances corresponding to each of the multiple first matching point pairs, and then the average value is used as the target signed epipolar line distance, so as to screen out the target matching point pairs corresponding to the moving object from the multiple second matching point pairs according to the target signed epipolar line distance.

[0064] In some embodiments, step 270 includes: respectively determining the magnitude relationship between the signed limit distance corresponding to each of the multiple second matching point pairs and a preset value; if the magnitude relationship indicates that there is a second matching point pair among the multiple second matching point pairs whose corresponding signed limit distance is greater than the preset value, then determine the second matching point pair whose corresponding signed limit distance is equal to the preset value as the target matching point pair.

[0065] As a way, since the signed limit distance corresponding to the static point is close to 0 affected by noise and the signed limit distance of the corresponding point in its dynamic counterpart is equal to the preset value, therefore, the target matching point pair can be determined among multiple second matching point pairs by determining the signed limit distance corresponding to each of the multiple second matching point pairs.

[0066] Optionally, if there is a point pair among the multiple second matching point pairs whose corresponding signed limit distance is equal to the preset value, it can be determined that the second matching point pair with the signed limit distance equal to the preset value is the target matching point pair of the moving object; if there is no point pair among the multiple second matching point pairs whose corresponding signed limit distance is equal to the preset value, it can be determined that there is no moving object in the image. At this time, multiple consecutive frames of images can be re-obtained, and then the reference matching point pairs corresponding to each of the multiple objects can be re-determined, and the reference matching point pairs can be classified to determine the first matching point pair and the second matching point pair.

[0067] Optionally, if there is a point pair among the multiple second matching point pairs whose corresponding signed limit distance is greater than the preset value, it can be determined that the second matching point pair with the signed limit distance greater than the preset value is an abnormal point pair of the moving object, and this point pair can be deleted.

[0068] In this embodiment, the fundamental matrix is determined according to the wheel speedometer information and multiple first matching point pairs, so that the signed limit distance corresponding to each of the multiple second matching point pairs can be determined according to the fundamental matrix, and thus the target matching point pair of the moving object can be screened out among the multiple second matching point pairs through the signed limit distance, effectively detecting moving objects in various complex traffic environments and reducing misjudgment.

[0069] Figure 5 The detection method of a moving object provided by an embodiment of the present application is shown. Next, the Figure 5 shown process will be elaborated in detail. The detection method of the moving object may specifically include the following steps:

[0070] Step 310, obtain the point cloud data and image data of the environment where the vehicle is located, where the image data includes multiple consecutive frames of images.

[0071] Step 320, perform feature recognition on the point cloud data to determine the object semantic information corresponding to each of the multiple objects included in the environment where the vehicle is located.

[0072] Step 330, perform optical flow tracking on the multiple frames of images to determine the reference matching point pairs corresponding to each of the multiple objects in the multiple frames of images; determine the displacement and pose of the vehicle according to the wheel speedometer information.

[0073] Step 340: Classify the reference matching point pairs corresponding to the multiple objects according to the object semantic information corresponding to each of the multiple objects, and determine a plurality of first matching point pairs and a plurality of second matching point pairs, where the first matching point pairs are the matching point pairs corresponding to the background objects among the multiple objects, and the second matching point pairs are the matching point pairs corresponding to the objects other than the background objects among the multiple objects.

[0074] Among them, for the specific step descriptions of steps 310 - 340, please refer to steps 110 - 140, which will not be elaborated here.

[0075] Step 350: Determine the pose of the vehicle according to the wheel speed information, and determine the camera internal parameters of the image acquisition device that acquires the image data.

[0076] As a method, the camera internal parameters are the calibration parameters of the image acquisition device, which are calibrated before the image acquisition device leaves the factory, and can be determined through the machine code corresponding to the image acquisition device. Optionally, before the image acquisition device leaves the factory, the calibrated parameters can be associated with the machine code, and then the machine code can be uploaded to the cloud server. Furthermore, the camera internal parameters of the image acquisition device that acquires the image data can be determined through the machine code.

[0077] Step 360: Determine the parallax range according to the camera internal parameters, the pose, and the plurality of first matching point pairs.

[0078] As a method, the parallax range can be determined by the coordinates of the first matching point pairs in the image coordinate system, the camera internal parameters, and the pose in the given consecutive images. Optionally, the formula where R is the pose of the vehicle, K is the camera internal parameter, is a pair of matching first matching point pairs, and then determine the maximum value and the minimum value, and determine the parallax range based on the maximum value and the minimum value

[0079] Optionally, the maximum parallax value and the minimum parallax value corresponding to each of the plurality of first matching point pairs can be calculated, and then the maximum value can be determined among the plurality of maximum parallax values to obtain the target maximum parallax value. Similarly, the minimum value can be determined among the plurality of minimum parallax values to obtain the target minimum parallax value. Furthermore, the parallax range can be determined according to the target maximum parallax value and the target minimum parallax value.

[0080] Optionally, the maximum parallax value and the minimum parallax value corresponding to each of the plurality of first matching point pairs can also be calculated, and then the average maximum parallax value and the average minimum parallax value can be calculated. Thus, the parallax range can be determined according to the average maximum parallax value and the average minimum parallax value.

[0081] Step 370: Determine the disparity values corresponding to each of the multiple second matching point pairs according to the camera internal parameters, the pose, and the multiple second matching point pairs.

[0082] As a way, it can be through the formula to respectively determine the disparity values corresponding to each of the multiple second matching point pairs, where R is the pose of the vehicle, K is the camera internal parameter, is a pair of matched second matching point pairs, is the disparity value.

[0083] Step 380: Screen out the target matching point pairs corresponding to the moving objects from the multiple second matching point pairs according to the disparity values corresponding to each of the multiple second matching point pairs and the disparity range.

[0084] As a way, it can be by respectively determining the relationship between the disparity values corresponding to each of the multiple second matching points and the disparity range, and based on this relationship, screening out the target matching point pairs of the moving objects from the multiple second matching point pairs.

[0085] In some embodiments, step 380 includes: if there are second matching point pairs among the multiple second matching point pairs whose corresponding disparity values are not within the disparity range, then determine the second matching point pairs whose disparity values are not within the disparity range as the target matching point pairs.

[0086] As a way, since the disparity range is determined according to multiple first matching point pairs, it can be determined that if the disparity value corresponding to a second matching point pair falls within the disparity range, then this matching point pair should be classified as a first matching point pair. Therefore, it can be by determining whether the disparity values corresponding to each of the multiple second matching point pairs are within the disparity range to screen the multiple second matching point pairs. Thus, when it is determined that there are second matching point pairs among the multiple second matching point pairs whose corresponding disparity values are not within the disparity range, then this matching point pair can be determined as the target matching point pair of the moving object.

[0087] Optionally, if there are no second matching point pairs among the multiple second matching point pairs whose corresponding disparity values are not within the disparity range, then multiple consecutive frames of images can be re-acquired, and then the reference matching point pairs corresponding to each of the multiple objects can be re-determined, and the reference matching point pairs can be classified to determine the first matching point pairs and the second matching point pairs, and then the second matching points can be screened here.

[0088] In this embodiment, the disparity range is determined by the camera internal parameters of the image acquisition device for acquiring image data, the wheel speedometer information of the vehicle, and multiple first matching point pairs, and the camera internal parameters of the image acquisition device for acquiring image data, the wheel speedometer information of the vehicle, and multiple second matching point pairs are used to determine the respective corresponding disparity values of the multiple second matching point pairs. Based on this, the target matching point pairs of the moving object are screened out from the multiple second matching point pairs according to the relationship between the disparity values and the disparity range, effectively detecting moving objects in various complex traffic environments and reducing misjudgment.

[0089] Figure 6 The detection method of a moving object provided by an embodiment of the present application is shown. The following will elaborate in detail on Figure 6 the process shown, and the detection method of the moving object may specifically include the following steps:

[0090] Step 410, obtain point cloud data and image data of the environment where the vehicle is located, where the image data includes multiple consecutive frames of images.

[0091] Step 420, perform feature recognition on the point cloud data to determine the object semantic information corresponding to each of the multiple objects included in the environment where the vehicle is located.

[0092] Step 430, perform optical flow tracking on the multiple frames of images to determine the reference matching point pairs corresponding to each of the multiple objects in the multiple frames of images.

[0093] Step 440, classify the reference matching point pairs corresponding to each of the multiple objects according to the object semantic information corresponding to each of the multiple objects, and determine multiple first matching point pairs and multiple second matching point pairs, where the first matching point pairs are the matching point pairs corresponding to the background objects among the multiple objects, and the second matching point pairs are the matching point pairs corresponding to the objects other than the background objects among the multiple objects.

[0094] Step 450, screen out the target matching point pairs corresponding to the moving object from the multiple second matching point pairs according to the wheel speedometer information of the vehicle and the multiple first matching point pairs.

[0095] Among them, for the specific step descriptions of steps 410 - 450, reference can be made to steps 110 - 150, and details will not be elaborated here.

[0096] Step 460, perform matching and fusion on the point cloud data and the image data to determine the target point cloud of the moving object in the point cloud data.

[0097] As a way, the target matching points of the determined moving object can be transformed into the world coordinate system to obtain multiple three-dimensional coordinates, and then the multiple three-dimensional coordinates are matched with the point cloud data to determine the target point cloud of the moving object in the point cloud data.

[0098] Step 470, determine the corner point cloud of the moving object in the target point cloud.

[0099] As a way, after determining the target point cloud, the three-dimensional bounding box of the moving object corresponding to the point cloud data can be determined according to the target point cloud, so that the corner point cloud of the moving object in the point cloud data can be determined based on the three-dimensional bounding box. Optionally, the corner point cloud can be the vertices of the three-dimensional bounding box. The three-dimensional bounding box can be an irregular three-dimensional bounding box determined based on the shape of the moving object, or a geometric three-dimensional bounding box determined according to the shape of the moving object. If it is a geometric three-dimensional bounding box, the corner point cloud can be the point cloud corresponding to the 8 vertices of the geometric three-dimensional bounding box, as Figure 7 shown.

[0100] Step 480, project the corner point cloud onto the image data, determine the two-dimensional bounding box of the moving object, and locate the moving object in the multiple frames of images according to the two-dimensional bounding box.

[0101] As a way, after determining the corner point cloud, the corner point cloud can be projected into the camera coordinate system first to obtain the corner point coordinates of the bounding box in the camera coordinate system, and then the corner point coordinates of the bounding box in the camera coordinate system are projected into the image coordinate system, so as to obtain the corner point coordinates of the moving object in the image coordinate system. Finally, the two-dimensional bounding box of the moving object is determined according to the corner point coordinates of the moving object in the image coordinate system.

[0102] Optionally, the positional relationship between the lidar and the image acquisition device of the vehicle can be determined first, that is, the conversion relationship between the radar coordinate system and the camera coordinate system is determined, so that the corner point coordinates of the bounding box of the corner point cloud in the camera coordinate system can be determined based on this conversion relationship. Optionally, the formula x 1 = C TL *x 0 can be used to determine the corner point coordinates of the bounding box of the corner point cloud in the camera coordinate system, where x 0 is the three-dimensional coordinate of the corner point cloud, and C TL is the rigid transformation matrix from the radar coordinate system to the camera coordinate system.

[0103] Optionally, after determining the corner point coordinates of the bounding box of the corner point cloud in the camera coordinate system, then through the formula F I (x 2 ) = I PC ·x 1To determine the coordinates of the corner point cloud projected in the image coordinate system, where I PC is the projection matrix of the camera. Based on this projection matrix, the corner point cloud can be projected onto the image coordinate system. F I (x 2 ) is the projection point of the corner point cloud on the two-dimensional image plane.

[0104] Optionally, after obtaining the projection of the corner point cloud in the image coordinates, the target projection points can be determined first according to the projection points in the image coordinates. The target projection points can be the projection points of the point cloud of the moving object at the lower left corner and the lower right corner in the image coordinates. Based on the projection points at the lower left corner and the lower right corner in the image coordinates, the two-dimensional border of the moving object can be determined, and then the moving object can be located according to the two-dimensional border. Optionally, the projection point at the lower left corner can be the minimum value on the horizontal and vertical coordinate axes among all the projection points of the moving object, and the projection point at the lower right corner can be the maximum value on the horizontal and vertical coordinate axes among all the projection points of the moving object.

[0105] In this embodiment, by fusing the point cloud data and the image data, the target point cloud of the moving object in the point cloud data is determined, and then the corner point cloud is determined in the target point cloud. Finally, by projecting the corner point cloud into the image data, the two-dimensional border of the moving object is determined, so that the moving object can be located in multiple frames of images according to the two-dimensional border, ensuring the positioning accuracy of the moving object in the image data.

[0106] Figure 8 Shows the detection method of a moving object provided by an embodiment of the present application. The following will be elaborated in detail for Figure 8 the process shown. The detection method of the moving object may specifically include the following steps:

[0107] Step 510, obtain the point cloud data and the image data of the environment where the vehicle is located, where the image data includes multiple consecutive frames of images.

[0108] Step 520, perform feature recognition on the point cloud data to determine the object semantic information corresponding to each of the multiple objects included in the environment where the vehicle is located.

[0109] Step 530, perform optical flow tracking on the multiple frames of images to determine the reference matching point pairs corresponding to each of the multiple objects in the multiple frames of images.

[0110] Step 540: Classify the reference matching point pairs corresponding to the multiple objects according to the object semantic information corresponding to each of the multiple objects, and determine a plurality of first matching point pairs and a plurality of second matching point pairs, where the first matching point pairs are the matching point pairs corresponding to the background objects among the multiple objects, and the second matching point pairs are the matching point pairs corresponding to the objects other than the background objects among the multiple objects.

[0111] Step 550: Screen out the target matching point pairs corresponding to the moving objects from the plurality of second matching point pairs according to the wheel speedometer information of the vehicle and the plurality of first matching point pairs.

[0112] Among them, for the specific step descriptions of steps 510 - 550, please refer to steps 110 - 150, which will not be elaborated here.

[0113] Step 560: Calculate the relative pose of the vehicle in adjacent frame images of multiple frames of images according to the wheel speedometer information, and determine the relative pose of the vehicle in the adjacent frame images.

[0114] As a way, first determine the change parameters of the wheel speedometer information in adjacent frame images, and then determine the relative pose of the vehicle in adjacent frame images of multiple frames of images according to the change parameters. It can be determined by the formula to determine the relative pose of the vehicle in adjacent frame images of multiple frames of images. Where θ is the angular change corresponding to the wheel speedometer information in adjacent frame images, and x and y are the position changes corresponding to the wheel speedometer information in adjacent frame images.

[0115] Step 570: Determine the relative pose as the initial value of the Iterative Closest Point (ICP) algorithm, and based on the ICP algorithm, determine the error function value between the matching point pairs in the adjacent frame images, and determine the current pose of the vehicle according to the error function value.

[0116] As a way, after determining the relative pose, the relative pose can be determined as the initial value of the ICP algorithm, and then perform ICP matching according to this initial value to determine the closest points corresponding to the feature points in adjacent frame images, so as to obtain the target feature point pairs. The Euclidean distance between the feature points and the surrounding feature points can be calculated, and the feature point pair corresponding to the minimum Euclidean distance is determined as the target feature point pair. Finally, the error function is determined according to the target feature point pair, and the current pose of the vehicle is determined by minimizing the error function. Optionally, it can be determined according to the formula E(R, t) = ∑ i ‖Rp i +t―q i ‖ 2To determine the error function, and thereby determine the corresponding R and t by determining the minimum value of the error function, and determine the determined R and t as the current pose of the vehicle, where p i and q i are target feature point pairs, and E is the error function value.

[0117] In this embodiment, pose estimation is performed on adjacent frame images in multiple frames of images according to the wheel speedometer information to obtain the relative pose of the vehicle in the adjacent frame images. Thereby, the error function value between the matching point pairs in the adjacent frame images is determined by the iterative closest point algorithm according to the relative pose of the vehicle in the adjacent frame images, and further, the current pose of the vehicle can be determined according to the error function value, improving the accuracy of the vehicle pose estimation.

[0118] Figure 9 is a schematic diagram of the working principle of a moving object detection method shown in an embodiment of the present application. As Figure 9 shown, obtain the lidar point cloud of the vehicle's environment and consecutive multiple frames of camera images of the vehicle's environment. First, perform optical flow tracking on the consecutive multiple frames of camera images to determine the reference matching point pairs corresponding to multiple objects, and identify the lidar point cloud to determine the object semantic information of multiple objects in the vehicle's environment. Then, classify the reference matching point pairs corresponding to multiple objects according to the object semantic information to obtain multiple background point pairs and multiple non-background point pairs. Then, determine the signed epipolar distance corresponding to each of the multiple non-background point pairs according to the multiple background point pairs and the wheel speedometer information of the vehicle. Screen the multiple non-background point pairs according to the signed epipolar distance to obtain multiple reference object point pairs. Then, further screen the multiple reference object point pairs according to the multiple background point pairs and the wheel speedometer information to obtain the target matching point pairs.

[0119] Meanwhile, identify the lidar point cloud to determine the three-dimensional regions corresponding to multiple objects in the lidar point cloud, and project the three-dimensional regions onto the image coordinates to obtain the two-dimensional regions corresponding to multiple objects. Then, determine the moving objects according to the two-dimensional regions and the target matching point pairs.

[0120] Then, the initial pose information can be estimated according to the wheel speedometer information and the adjacent frame images. Thereby, the two-dimensional border of the moving object is determined according to the initial position information and the target matching point pairs of the moving object in the image coordinate system, and the two-dimensional border is back-projected into the three-dimensional coordinate system to obtain the three-dimensional border point cloud. Thereby, the data within the three-dimensional border point cloud in the lidar point cloud is removed according to the three-dimensional border point cloud to obtain the target lidar point cloud. Finally, static scene reconstruction is performed according to the initial pose information and the target lidar point cloud to obtain the static scene.

[0121] Figure 10It is a block diagram of a detection device for a moving object shown according to an embodiment of the present application. As Figure 10 shown, the detection device 600 for the moving object includes an acquisition module 610, an identification module 620, a reference matching point pair determination module 630, a classification module 640, and a moving object determination module 650.

[0122] The acquisition module 610 is configured to acquire point cloud data and image data of the environment where the vehicle is located, where the image data includes a plurality of consecutive frames of images; the identification module 620 is configured to perform feature identification on the point cloud data to determine the object semantic information corresponding to each of the plurality of objects included in the environment where the vehicle is located; the reference matching point pair determination module 630 is configured to perform optical flow tracking on the plurality of frames of images to determine the reference matching point pairs corresponding to each of the plurality of objects in the plurality of frames of images; the classification module 640 is configured to classify the reference matching point pairs corresponding to each of the plurality of objects according to the object semantic information corresponding to each of the plurality of objects to determine a plurality of first matching point pairs and a plurality of second matching point pairs, where the first matching point pairs are the matching point pairs corresponding to the background objects among the plurality of objects, and the second matching point pairs are the matching point pairs corresponding to the objects other than the background objects among the plurality of objects; the moving object determination module 650 is configured to screen out the target matching point pairs corresponding to the moving objects from the plurality of second matching point pairs according to the vehicle's wheel speed information and the plurality of first matching point pairs.

[0123] In some embodiments, the moving object determination module 650 includes: a first determination sub-module, configured to determine the displacement and pose of the vehicle according to the wheel speed information; an essential matrix determination sub-module, configured to determine an essential matrix according to the displacement and pose, and determine a fundamental matrix according to the essential matrix and the plurality of first matching point pairs; a moving object first determination sub-module, configured to determine the signed limiting distance corresponding to each of the plurality of second matching point pairs according to the fundamental matrix, and screen out the target matching point pairs corresponding to the moving objects from the plurality of second matching point pairs according to the signed limiting distance.

[0124] In some embodiments, the moving object first determination sub-module includes: a size relationship determination unit, configured to respectively determine the size relationship between the signed limiting distance corresponding to each of the plurality of second matching point pairs and a preset value; a target matching point pair first determination unit, configured to, if the size relationship indicates that there is a second matching point pair among the plurality of second matching point pairs whose corresponding signed limiting distance is greater than the preset value, determine the second matching point pair whose corresponding signed limiting distance is equal to the preset value as the target matching point pair.

[0125] In some embodiments, the moving object determination module 650 further includes: a second determination sub-module, configured to determine the pose of the vehicle according to the wheel speedometer information and determine the internal camera parameters of the image acquisition device that acquires the image data; a parallax range determination sub-module, configured to determine the parallax range according to the internal camera parameters, the pose, and the multiple first matching point pairs; a parallax value determination sub-module, configured to determine the parallax values corresponding to the multiple second matching point pairs according to the internal camera parameters, the pose, and the multiple second matching point pairs; and a moving object second determination sub-module, configured to screen out the target matching point pairs corresponding to the moving object from the multiple second matching point pairs according to the parallax values corresponding to the multiple second matching point pairs and the parallax range.

[0126] In some embodiments, the moving object second determination sub-module includes: a target matching point pair second determination unit, configured to, if there is a second matching point pair among the multiple second matching point pairs whose corresponding parallax value is not within the parallax range, determine the second matching point pair whose parallax value is not within the parallax range as the target matching point pair.

[0127] In some implementations, the detection device 600 of the moving object further includes: a target point cloud determination module, configured to perform matching and fusion on the point cloud data and the image data to determine the target point cloud of the moving object in the point cloud data; a corner point cloud determination module, configured to determine the corner point cloud of the moving object in the target point cloud; and a positioning module, configured to project the corner point cloud onto the image data to determine the two-dimensional border of the moving object and position the moving object in the multiple frames of images according to the two-dimensional border.

[0128] In other implementations, the detection device 600 of the moving object further includes: a relative pose determination module, configured to perform pose calculation on adjacent frame images in multiple frames of images according to the wheel speedometer information to determine the relative pose of the vehicle in the adjacent frame images; and a current pose determination module, configured to use the relative pose as the initial value of the iterative closest point algorithm, determine the error function value between the matching point pairs in the adjacent frame images based on the iterative closest point algorithm, and determine the current pose of the vehicle according to the error function value.

[0129] According to one aspect of the embodiments of the present application, there is also provided an electronic device, as Figure 11 shown. The electronic device 700 includes a processor 710 and one or more memories 720. The one or more memories 720 are used to store program instructions executed by the processor 710. When the processor 710 executes the program instructions, the above-mentioned detection method of the moving object is implemented.

[0130] Further, the processor 710 may include one or more processing cores. The processor 710 runs or executes instructions, programs, code sets, or instruction sets stored in the memory 720, and calls data stored in the memory 720. Optionally, the processor 710 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 710 may integrate one or a combination of several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, application programs, etc.; the GPU is responsible for rendering and drawing the displayed content; the modem is used to process wireless communication. It can be understood that the above modem may not be integrated into the processor and may be implemented separately through a communication chip.

[0131] According to one aspect of the present application, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or may exist separately without being assembled into the electronic device. The above computer-readable storage medium carries computer-readable instructions, and when the computer-readable storage instructions are executed by a processor, the method in any of the above embodiments is implemented.

[0132] It should be noted that the computer-readable medium shown in the embodiments of the present application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, and this computer-readable medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0133] The units described in the embodiments of the present application can be implemented in software or in hardware, and the described units can also be provided in a processor. Among them, the names of these units do not, in some cases, limit the units themselves.

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. Among them, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the above-mentioned module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order from that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, as well as the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0135] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include the known common knowledge or conventional technical means in the technical field not disclosed in the present application.

[0136] It should be understood that the present application is not limited to the exact structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A method for detecting a moving object, characterized in that: The method comprises: Acquire point cloud data and image data of the environment where the vehicle is located, wherein the image data includes multiple continuous frames of images; Performing feature recognition on the point cloud data to determine object semantic information corresponding to each of a plurality of objects included in the environment where the vehicle is located; Performing optical flow tracking on the multiple frames of images to determine reference matching point pairs corresponding to the multiple objects in the multiple frames of images; Classifying the reference matching point pairs corresponding to the multiple objects according to the object semantic information corresponding to the multiple objects, and determining a plurality of first matching point pairs and a plurality of second matching point pairs, wherein the first matching point pairs are matching point pairs corresponding to background objects among the multiple objects, and the second matching point pairs are matching point pairs corresponding to objects among the multiple objects except the background objects; According to the wheel speed meter information of the vehicle and the plurality of first matching point pairs, a target matching point pair corresponding to the moving object is screened out from the plurality of second matching point pairs.

2. The method according to claim 1, characterized in that: The step of selecting target matching point pairs corresponding to the moving object from the plurality of second matching point pairs according to the wheel speed meter information of the vehicle and the plurality of first matching point pairs comprises: Determining the displacement of the vehicle and the position and posture of the vehicle according to the wheel speed meter information; Determine an essential matrix according to the displacement and the posture, and determine a basic matrix according to the essential matrix and the plurality of first matching point pairs; The signed limit distances corresponding to each of the plurality of second matching point pairs are determined according to the basic matrix, and the target matching point pairs corresponding to the moving object are screened out from the plurality of second matching point pairs according to the signed limit distances.

3. The method according to claim 2, characterized in that The step of selecting a target matching point pair corresponding to the moving object from the plurality of second matching point pairs according to the signed limit distance includes: Respectively determine the magnitude relationship between the signed limit distances corresponding to each of the plurality of second matching point pairs and a preset value; If the size relationship indicates that there is a second matching point pair whose corresponding signed limit distance is greater than the preset value among the plurality of second matching point pairs, the second matching point pair whose corresponding signed limit distance is equal to the preset value is determined as the target matching point pair.

4. The method according to claim 1, characterized in that: The method of selecting target matching point pairs corresponding to the moving object from the plurality of second matching point pairs according to the wheel speed meter information of the vehicle and the plurality of first matching point pairs further includes: Determine the position and posture of the vehicle according to the wheel speed meter information, and determine the camera intrinsic parameters of the image acquisition device that acquires the image data; Determining a parallax range according to the camera intrinsic parameters, the position and posture, and the plurality of first matching point pairs; Determine, according to the camera intrinsic parameters, the position and posture, and the plurality of second matching point pairs, the disparity values ​​corresponding to each of the plurality of second matching point pairs; According to the disparity values ​​corresponding to the plurality of second matching point pairs and the disparity range, a target matching point pair corresponding to the moving object is screened out from the plurality of second matching point pairs.

5. The method according to claim 4, characterized in that The step of selecting target matching point pairs corresponding to the moving object from the plurality of second matching point pairs according to the respective disparity values ​​corresponding to the plurality of second matching point pairs and the disparity range includes: If there is a second matching point pair whose corresponding disparity value is not within the disparity range among the plurality of second matching point pairs, the second matching point pair whose disparity value is not within the disparity range is determined as the target matching point pair.

6. The method according to any one of claims 1 to 5, characterized in that: After selecting target matching point pairs corresponding to the moving object from the plurality of second matching point pairs according to the wheel speed meter information of the vehicle and the plurality of first matching point pairs, the method further includes: Matching and fusing the point cloud data and the image data to determine a target point cloud of the moving object in the point cloud data; Determining a corner point cloud of the moving object in the target point cloud; The corner point cloud is projected into the image data to determine a two-dimensional bounding box of the moving object, and the moving object is positioned in the multiple frames of images according to the two-dimensional bounding box.

7. The method according to claim 6, characterized in that The method further comprises: Performing posture estimation on adjacent frame images in the plurality of frame images according to the wheel speed meter information to determine the relative posture of the vehicle in the adjacent frame images; The relative posture is determined as an initial value of an iterative closest point algorithm, and based on the iterative closest point algorithm, an error function value between matching point pairs in adjacent frame images is determined, and the current posture of the vehicle is determined according to the error function value.

8. A moving object detection device, characterized in that: The device comprises: An acquisition module, used to acquire point cloud data and image data of the environment where the vehicle is located, wherein the image data includes continuous multiple-frame images; A recognition module, used to perform feature recognition on the point cloud data to determine object semantic information corresponding to each of a plurality of objects included in the environment where the vehicle is located; A reference matching point pair determination module, used to perform optical flow tracking on the multiple frames of images, and determine reference matching point pairs corresponding to the multiple objects in the multiple frames of images; a classification module, configured to classify the reference matching point pairs corresponding to each of the multiple objects according to the object semantic information corresponding to each of the multiple objects, and determine a plurality of first matching point pairs and a plurality of second matching point pairs, wherein the first matching point pairs are matching point pairs corresponding to background objects among the multiple objects, and the second matching point pairs are matching point pairs corresponding to objects among the multiple objects excluding the background objects; The moving object determination module is used to select a target matching point pair corresponding to the moving object from the plurality of second matching point pairs according to the wheel speed meter information of the vehicle and the plurality of first matching point pairs.

9. An electronic device, characterized in that: The electronic device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images