Method and device for measuring motion state of target object, computer equipment and medium

By combining the camera and the bicycle sensor, the feature points in the video data are extracted and matched, and the motion state of the target object is calculated, which solves the problem of insufficient measurement accuracy in the prior art, and achieves high-precision distances for dynamic and static target objects. Speed ​​and acceleration measurements improve the detection capabilities of autonomous driving.

CN119022956APending Publication Date: 2024-11-26WUHAN JIMU INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411060555.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In the prior art, it is impossible to accurately measure the motion state of dynamic and static target objects, especially the measurement accuracy of distance and velocity is insufficient and the detection distance is limited.

Method used

Real-time video data is collected through an external camera, target information is extracted and feature matching processing is performed, and the distance, speed and acceleration of the target is calculated based on the bicycle sensor data, and its motion state is output.

Benefits of technology

It realizes accurate distance, velocity and acceleration measurement of dynamic and static target objects, improves detection distance and accuracy, and is suitable for providing key information for bicycle control in autonomous driving scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119022956A_ABST
    Figure CN119022956A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a method and device for measuring the motion state of a target object, computer equipment and a medium, and the method comprises the following steps: collecting real-time video data through an external camera, and obtaining the information of the target object from the real-time video data; performing feature extraction and matching processing on a first image of the ith frame in the real-time video data and a second image of the (i + 1) th frame in the real-time video data, and calculating the distance from the vehicle to the target object in the ith frame and the (i + 1) th frame according to the pixel coordinates of the target feature point pair; the displacement of the target object in the delta t time period is obtained through calculation, and the speed of the target object at the t moment is obtained through displacement calculation; calculating the acceleration of the target object at the t moment; and taking the distance of the target object, the speed of the target object at the t moment and the acceleration of the target object at the t moment as the motion state of the target object and outputting the motion state to the outside. According to the scheme, the motion state of the dynamic and / or static target object is measured through the collected video data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method, device, computer equipment and medium for measuring the motion state of a target object. Background Art

[0002] It is very important to measure the distance and speed of the target in autonomous driving. Reliable distance and speed measurement can provide key information for self-vehicle control. The current mainstream speed measurement method is based on image processing methods, such as optical flow, but the accuracy of distance and speed measurement needs to be improved, and the detection distance is limited. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a method for measuring the motion state of a target object to solve the technical problem that the motion state of a moving and / or static target object cannot be measured in the prior art. The method includes:

[0004] Collecting real-time video data through an external camera, and obtaining target information from the real-time video data, wherein the target information includes a category identifier of the target, a tracking identifier of the target, and a bounding box of the target;

[0005] Based on the target object information, feature extraction and matching processing are performed on the first image of the i-th frame in the real-time video data and the second image of the i+1-th frame in the real-time video data, and the feature points of the first image and the matching feature points in the second image located in the boundary frame of the target object are used as the target feature point pair, and the distance from the vehicle to the target object in the i-th frame and the i+1-th frame is calculated according to the pixel coordinates of the target feature point pair;

[0006] The vehicle speed and acceleration of the vehicle at time t+Δt are obtained through external sensors, and the travel distance of the vehicle in the x-axis direction from time t to time t+Δt is calculated based on the vehicle speed and the vehicle acceleration, and the first distance from the vehicle to the target object at time t and the second distance from the vehicle to the target object at time t+Δt are calculated, and the displacement of the target object in the Δt time period is calculated by the first distance, the second distance and the travel distance, and the speed of the target object at time t is calculated by the displacement, wherein Δt is the time interval;

[0007] Calculate the first velocity of the target object at time t+Δt and the second velocity at time t+2Δt respectively, and obtain the acceleration of the target object at time t by calculating the first velocity and the second velocity;

[0008] The distance of the target object, the speed of the target object at time t, and the acceleration of the target object at time t are taken as the motion state of the target object and output to the outside.

[0009] The embodiment of the present invention also provides a device for measuring the motion state of a target object to solve the technical problem that the motion state of a moving and / or static target object cannot be measured in the prior art. The device includes:

[0010] The target detection module is used to collect real-time video data through an external camera and obtain target information from the real-time video data, wherein the target information includes the category identification of the target, the tracking identification of the target and the bounding box of the target;

[0011] A distance measurement module is used to extract features and perform matching processing on the first image of the i-th frame in the real-time video data and the second image of the i+1-th frame in the real-time video data based on the target information, and use the feature points of the first image and the matching feature points in the second image located in the boundary frame of the target as the target feature point pair, and calculate the distance from the vehicle to the target in the i-th frame and the i+1-th frame according to the pixel coordinates of the target feature point pair;

[0012] a speed measurement module, for obtaining the speed and acceleration of the vehicle at time t+Δt through an external sensor, calculating the travel distance of the vehicle in the x-axis direction from time t to time t+Δt according to the speed and acceleration of the vehicle, calculating a first distance from the vehicle to the target object at time t and a second distance from the vehicle to the target object at time t+Δt, calculating the displacement of the target object in the Δt time period through the first distance, the second distance and the travel distance, and calculating the speed of the target object at time t through the displacement, wherein Δt is the time interval;

[0013] An acceleration measurement module is used to calculate a first velocity of the target object at time t+Δt and a second velocity at time t+2Δt, respectively, and obtain the acceleration of the target object at time t by calculating the first velocity and the second velocity;

[0014] The motion state acquisition module is used to take the distance of the target object, the speed of the target object at time t and the acceleration of the target object at time t as the motion state of the target object and output them to the outside.

[0015] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned methods for measuring the motion state of a target object when executing the computer program, so as to solve the technical problem in the prior art that the motion state of a dynamic and / or static target object cannot be measured.

[0016] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program for executing any of the above-mentioned methods for measuring the motion state of a target object, so as to solve the technical problem in the prior art that the motion state of a dynamic and / or static target object cannot be measured.

[0017] Compared with the prior art, the at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects:

[0018] The motion state of dynamic and / or static objects including distance, speed and acceleration is measured through the collected video data. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 is a flow chart of a method for measuring the motion state of a target object provided by an embodiment of the present invention;

[0021] Figure 2 is a structural block diagram of a computer device provided by an embodiment of the present invention;

[0022] Figure 3 It is a structural block diagram of a device for measuring the motion state of a target object provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0024] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0025] In the embodiment of the present invention, a forward-looking camera mounted on the front windshield of the vehicle is used to measure the distance, speed and acceleration of the target object. The x-axis of the vehicle coordinate system is defined as the forward direction of the vehicle, the y-axis is the direction perpendicular to the x-axis on the left side of the vehicle, and the z-axis is perpendicular to the xy plane to satisfy the right-hand rule. The installation height H, pitch angle Yaw and frame rate FPS of the camera are known. Other sensors used include speedometers and accelerometers. The speedometer provides the instantaneous speed v_x of the vehicle on the x-axis, and the accelerometer provides the acceleration acc_x of the vehicle on the x-axis.

[0026] In an embodiment of the present invention, a method for measuring the motion state of a target object is provided, such as Figure 1 As shown, the method includes:

[0027] Step S101: collecting real-time video data through an external camera, and obtaining target information from the real-time video data, wherein the target information includes a category identifier of the target, a tracking identifier of the target, and a bounding box of the target;

[0028] Step S102: performing feature extraction and matching processing on the first image of the i-th frame in the real-time video data and the second image of the i+1-th frame in the real-time video data according to the target object information, taking the feature points of the first image located in the boundary frame of the target object and the matching feature points in the second image as the target feature point pair, and calculating the distance from the vehicle to the target object in the i-th frame and the i+1-th frame according to the pixel coordinates of the target feature point pair;

[0029] Step S103: obtaining the vehicle speed and acceleration of the vehicle at time t+Δt through an external sensor, calculating the travel distance of the vehicle in the x-axis direction from time t to time t+Δt according to the vehicle speed and the vehicle acceleration, calculating a first distance from the vehicle to the target object at time t and a second distance from the vehicle to the target object at time t+Δt, calculating the displacement of the target object in the Δt time period through the first distance, the second distance and the travel distance, and calculating the speed of the target object at time t through the displacement, wherein Δt is the time interval;

[0030] Step S104: Calculate the first speed of the target object at time t+Δt and the second speed at time t+2Δt respectively, and obtain the acceleration of the target object at time t by calculating the first speed and the second speed;

[0031] Step S105: The distance of the target object, the speed of the target object at time t, and the acceleration of the target object at time t are taken as the motion state of the target object and output to the outside.

[0032] In specific implementation, the following steps are used to obtain target information from real-time video data:

[0033] The real-time video data is input into the small target detection model, and the category identification and bounding box of the target are obtained through the small target detection model; the category identification and bounding box of the target are input into the target tracking model, and the tracking identification of the target is obtained through the target tracking model.

[0034] In a specific implementation, in order to improve the detection distance and the detection effect of small targets, the following steps are performed to input the real-time video data into a small target detection model, and obtain the category identification of the target and the bounding box of the target through the small target detection model:

[0035] The input resolution is set, and the original image obtained from the real-time video data is converted into first image data of the input resolution size, and the first image data is input into the small target detection model, and the first detection result of the target object is obtained through the small target detection model; lane line detection is performed on the first image data, and lane line information is extracted, and the lane line vanishing point is calculated through the lane line information; the original image is cropped into second image data of the input resolution size with the lane line vanishing point as the center, and the second image data is input into the small target detection model, and the second detection result of the target object is obtained through the small target detection model; the first detection result and the second detection result are fused, and the generated target object category identification and target object boundary box are output.

[0036] Specifically, the target detection model is optimized. The input resolution of the target detection model is H×W. When doing the first reasoning, the full image is resized to H×W and then input into the target detection model to obtain the first detection result R1. When doing the second reasoning, an image of size H×W is cropped with the vanishing point obtained by the lane line detection as the center, and then input into the target detection model to obtain the second detection result R2. Then R1 and R2 are fused to obtain the final detection result. This method is conducive to improving the detection distance and improving the detection effect of the target detection model on small targets.

[0037] Using the above target detection strategy, a multi-target tracking algorithm (such as DeepSORT) is used to obtain stably tracked targets. The target information includes the target category identifier (target category ID {cid}), the target tracking identifier (tracking ID {tid}), the target bounding box {[x cid-tid -1,y cid-tid -1,x cid-tid -2,y cid-tid -2]}).

[0038] Since vehicles, pedestrians and cyclists are most likely to appear in autonomous driving scenarios, the target objects are identified as vehicles (including cars, buses and trucks), pedestrians and cyclists. When the target object is identified as a vehicle, the bounding box of the target object is the bounding box of the rear, front or side of the vehicle.

[0039] In a specific implementation, in order to obtain a pair of feature points in a bounding box of a target object, the following steps are performed to extract and match the first image of the i-th frame in the real-time video data and the second image of the i+1-th frame in the real-time video data through target information, and the feature points of the first image in the bounding box of the target object and the matching feature points in the second image are used as the target feature point pairs:

[0040] Based on the target object information, a feature extraction algorithm is used to obtain feature points in the first image as a first feature point set, and feature points in the second image as a second feature point set; a matching algorithm is used to perform feature matching on feature points in the first feature point set and in the second feature point set to generate feature point pairs; the pixel coordinates of the feature points in the feature point pair are obtained, and if the pixel coordinates are both within the bounding box of the target object, the feature pair is used as a target feature point pair.

[0041] Specifically, the pixel coordinates of the i-th frame in the matching feature pair of the previous and next two frames (the i-th frame and the i+1-th frame) are obtained by using the feature extraction and matching algorithm ({[x i ,y i ]},{[x i+1 ,y i+1 ]}), where i represents the frame number, and the pixel coordinates corresponding to the i+1th frame in the matching feature pair are obtained according to the obtained bounding box ({[x cid-tid-i-j ,y cid-tid-i-j ]},{[x cid-tid-i+1-j ,y cid-tid-i+1-j ]}), where j represents the category identifier (feature point ID) of the target object in the i+1th frame of the matching feature pair.

[0042] In the specific implementation, in order to directly index the depth value according to the pixel coordinates of the feature points and use the average depth value as the distance, the following steps are performed to calculate the distance from the vehicle to the target object in the i-th frame and the i+1-th frame according to the pixel coordinates of the target feature point pair:

[0043] Use a lightweight dense monocular depth estimation model to obtain the dense depth I of the first image. i and the dense depth I of the second image i+1 ; According to the dense depth I iThe first depth data is calculated by the pixel coordinates corresponding to the first image in the target feature point pair, and the first average value of the depth data of all feature points is calculated according to the first depth data of all feature points, and the first average value is used as the distance from the vehicle to the target object in the i-th frame; according to the dense depth I i+1 The second depth data is calculated based on the pixel coordinates corresponding to the second image in the target feature point pair, and the second average value of the depth data of all feature points is calculated based on the second depth data of all feature points, and the second average value is used as the distance from the vehicle to the target object in the i+1th frame.

[0044] Specifically, a lightweight dense monocular depth estimation model with good generalization ability in different outdoor scenes is used to obtain the dense depth I of the images of the previous and next two frames (the i-th frame and the i+1-th frame) i and I i+1 , according to the pixel coordinates of the feature point {[x cid-tid-i-j ,y cid-tid-i-j ]} and {[x cid-tid-i+1-j ,y cid-tid-i+1-j ]}Get the depth data of the feature points matched between the previous and next frames{d cid-tid-i-j} and {d cid-tid-i+1-j}, average the depth data of all feature points in the bounding box {d cid-tid-i} and {d cid-tid-i+1}, that is, the distance d from the vehicle to the target object with the category identifier cid and the tracking identifier tid in the i-th frame and i+1-th frame is obtained. Since the output of the lightweight dense monocular depth estimation model is a two-dimensional floating-point array with the same width and height as the input image, in meters, the depth value can be directly indexed according to the pixel coordinates of the feature points.

[0045] Specifically, because the target object is a three-dimensional object, and the distance d of the target object is the surface of the target object closest to the ego vehicle, the dense depth of this surface is a numerical range. Since no geometric point or feature point in this surface can represent the distance from the ego vehicle to the surface, the average value is taken to approximate this distance. The more feature points extracted on the target object, the more accurate the distance d of the ego vehicle from the target object.

[0046] In one embodiment, taking the simplest 2 farthest points and 1 closest point as an example:

[0047] First, obtain the bounding box of the target object. The principle of selecting the bounding box is that the surface of the target object is closest to the vehicle. For example, if the target object is a person, assume that the three feature points of the top of the head, chest and toes are obtained from the second step, and assume that the height of the person is 1.8m, the camera installation height on the vehicle is 1.5m, the person and the vehicle are on the same horizontal plane, the camera and the feature points of the chest of the person are on the same horizontal plane, and the distance is 6.4m. Then, according to the Pythagorean theorem, the distance from the camera to the top of the head feature point is d1 = sqrt((5+1.4)^2+(1.8-1.5)^2) = 6.407, the distance from the camera to the chest feature point is d2 = 6.4m, and the distance from the camera to the toe feature point is d3 = sqrt((5+1.4)^2+1.5^2) = 6.573, (d1+d2+d3) / 3 = 6.559. This distance is the distance between the camera and the target. It is known that the horizontal distance from the camera to the front of the vehicle is 1.4m, so the distance between the vehicle and the target object is 5.159m.

[0048] In specific implementation, the following steps are performed to calculate the displacement of the target object within the time period of Δt through the first distance, the second distance and the travel distance, and to calculate the speed of the target object at time t through the displacement:

[0049] The instantaneous speed v at time t is calculated by the first distance, the second distance and the travel distance cid-tid-t , where the displacement d of the target object in the time period Δt is cid-tidi =d t+Δt +d ego -d t , cid is the category identifier of the target, tid is the tracking identifier of the target, d t is the first distance, d t+Δt is the second distance, d ego is the travel distance; through displacement d cid-tidi Calculate the instantaneous velocity v at time t cid-tid-t , where the instantaneous velocity v cid-tid-t =v cid-tid-objavg =d cid-tidi / Δt,v cid-tid-objavg is the average speed of the target object in the time period Δt.

[0050] Specifically, from time t to time t+Δt, where Δt=1 / FPS (the relationship between Δt and FPS, Δt is the time increment, and FPS is the number of frames per second). Here, the movement of the vehicle in the z-axis direction during the Δt time is ignored, so the distance traveled by the vehicle in the x-direction (the y-direction is less used and not paid attention to) is According to the distances dt and dt+Δt from the vehicle to the target object at time t and time t+Δt obtained previously, the displacement d of the target object in the time period Δt is cid-tid =dt +Δt+d ego -d t , then the average speed of the target in the time period Δt is v cid-tid-avg =d cid-tid / Δt. In one embodiment, the camera frame rate FPS is 20, Δt is 50ms, and the average speed v cid-tid-avg It can be approximated as the instantaneous velocity v of the target at time t cid-tid-t (The camera frame rate is generally 20Hz. In this case, the interval between two frames is 50ms. The average speed in such a short time can be approximated as the instantaneous speed at time t. The higher the camera frame rate, the more accurate this approximation is).

[0051] In specific implementation, the following steps are performed to calculate the travel distance of the vehicle in the x-axis direction from time t to time t+Δt according to the vehicle speed and the vehicle acceleration:

[0052] The distance traveled by the vehicle in the x-axis direction Where v_x is the vehicle velocity in the x direction, acc_x is the vehicle acceleration in the x direction, and d ego is the distance traveled by the vehicle in the x-axis direction.

[0053] Specifically, according to the continuous calculation of the target object's velocity v within 2 Δt time periods t+Δt 、v t+2Δt , the acceleration acc of the target at time t can be approximately obtained obj =(v t+2Δt -v t+Δt ) / Δt.

[0054] In this embodiment, a computer device is provided, such as Figure 2 As shown, it includes a memory 201, a processor 202, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any of the above-mentioned methods for measuring the motion state of a target object is implemented.

[0055] Specifically, the computer device may be a computer terminal, a server or a similar computing device.

[0056] In this embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program for executing any of the above methods for measuring the motion state of a target object.

[0057] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0058] Based on the same inventive concept, a device for measuring the motion state of a target object is also provided in the embodiment of the present invention, as described in the following embodiments. Since the principle of solving the problem by the device for measuring the motion state of a target object is similar to the method for measuring the motion state of a target object, the implementation of the device for measuring the motion state of a target object can refer to the implementation of the method for measuring the motion state of a target object, and the repetitions are not repeated. As used below, the term "unit" or "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0059] Figure 3 is a structural block diagram of a device for measuring the motion state of a target object according to an embodiment of the present invention. Figure 3 As shown, it includes: a target detection module 301, a distance measurement module 302, a speed measurement module 303, an acceleration measurement module 304 and a motion state acquisition module 305. The structure is described below.

[0060] The target detection module 301 is used to collect real-time video data through an external camera and obtain target information from the real-time video data, wherein the target information includes a category identifier of the target, a tracking identifier of the target and a bounding box of the target;

[0061] The distance measurement module 302 is used to extract features and perform matching processing on the first image of the i-th frame in the real-time video data and the second image of the i+1-th frame in the real-time video data based on the target information, and use the feature points of the first image and the matching feature points in the second image located in the boundary frame of the target as the target feature point pair, and calculate the distance from the vehicle to the target in the i-th frame and the i+1-th frame according to the pixel coordinates of the target feature point pair;

[0062] The speed measurement module 303 is used to obtain the speed and acceleration of the vehicle at time t+Δt through an external sensor, calculate the travel distance of the vehicle in the x-axis direction from time t to time t+Δt according to the speed and acceleration of the vehicle, calculate a first distance from the vehicle to the target object at time t and a second distance from the vehicle to the target object at time t+Δt, calculate the displacement of the target object in the Δt time period through the first distance, the second distance and the travel distance, and calculate the speed of the target object at time t through the displacement, wherein Δt is the time interval;

[0063] The acceleration measurement module 304 is used to calculate the first velocity of the target object at time t+Δt and the second velocity at time t+2Δt, and obtain the acceleration of the target object at time t by calculating the first velocity and the second velocity;

[0064] The motion state acquisition module 305 is used to take the distance of the target object, the speed of the target object at time t and the acceleration of the target object at time t as the motion state of the target object and output it to the outside.

[0065] In one embodiment, the target detection module includes:

[0066] The target object recognition unit is used to input the real-time video data into the small target detection model, and obtain the category identification and the bounding box of the target object through the small target detection model;

[0067] The target object tracking unit is used to input the category identification of the target object and the bounding box of the target object into the target tracking model, and obtain the tracking identification of the target object through the target tracking model.

[0068] In one embodiment, the target object recognition unit is used to set the input resolution, convert the original image obtained from the real-time video data into first image data of the input resolution size, input the first image data into a small target detection model, and obtain a first detection result of the target object through the small target detection model; perform lane line detection on the first image data, extract lane line information, and calculate the lane line vanishing point through the lane line information; crop the original image into second image data of the input resolution size with the lane line vanishing point as the center, input the second image data into the small target detection model, and obtain a second detection result of the target object through the small target detection model; fuse the first detection result and the second detection result, and output the generated target category identification and target bounding box.

[0069] In one embodiment, the distance measurement module includes:

[0070] A feature point set generating unit, used to obtain feature points in the first image as a first feature point set and feature points in the second image as a second feature point set by using target object information and a feature extraction algorithm;

[0071] A feature point pair generating unit, configured to perform feature matching on feature points in the first feature point set and the second feature point set by using a matching algorithm to generate feature point pairs;

[0072] The target feature point pair determination unit is used to obtain the pixel coordinates of the feature points in the feature point pair. If the pixel coordinates are both within the bounding box of the target object, the feature pair is regarded as the target feature point pair.

[0073] In one embodiment, the distance measurement module further includes:

[0074] A dense depth acquisition unit, used to acquire a dense depth I of the first image using a lightweight dense monocular depth estimation model i and the dense depth I of the second image i+1 ;

[0075] The i-th frame distance calculation unit is used to calculate the distance between the dense depth I i The first depth data is calculated based on the pixel coordinates corresponding to the first image in the target feature point pair, and the first average value of the depth data of all feature points is calculated based on the first depth data of all feature points, and the first average value is used as the distance from the vehicle to the target object in the i-th frame;

[0076] The i+1th frame distance calculation unit is used to calculate the distance between the dense depth I i+1 The second depth data is calculated based on the pixel coordinates corresponding to the second image in the target feature point pair, and the second average value of the depth data of all feature points is calculated based on the second depth data of all feature points, and the second average value is used as the distance from the vehicle to the target object in the i+1th frame.

[0077] In one embodiment, the speed measurement module includes:

[0078] A driving distance calculation unit, used to calculate the instantaneous speed v at time t through the first distance, the second distance and the driving distance cid-tid-t , where the displacement d of the target object in the time period Δt is cid-tidi =d t+Δt +d ego -d t , cid is the category identifier of the target, tid is the tracking identifier of the target, d t is the first distance, d t+Δt is the second distance, d ego is the driving distance;

[0079] Average velocity calculation unit for displacement d cid-tidi Calculate the instantaneous velocity v at time t cid-tid-t , where the instantaneous velocity v cid-tid-t =v cid-tid-objavg =d cid-tidi / Δt,v cid-tid-objavg is the average speed of the target object in the time period Δt.

[0080] In one embodiment, the driving distance calculation unit is used to calculate the driving distance of the vehicle in the x-axis direction. Where v_x is the vehicle velocity in the x direction, acc_x is the vehicle acceleration in the x direction, and d ego is the distance traveled by the vehicle in the x-axis direction.

[0081] The embodiments of the present invention achieve the following technical effects:

[0082] A small target detection model is used to realize target detection, a target tracking model (such as deep sort algorithm) is used to realize stable tracking of the detection target, a feature extraction and matching algorithm is used to realize the extraction and matching of ORB (composed of key points and descriptors), and a lightweight monocular depth estimation model is used to obtain dense depth information frame by frame. The above models are all lightweight models, thereby ensuring that this embodiment can run in real time in edge devices; the lightweight dense monocular depth estimation model used makes a good trade-off between reasoning speed and detection depth, so that the detection distance of vehicles in static / dynamic scenes is more than 120m, and the detection distance of pedestrians is more than 80m. At the same time, by improving the detection accuracy of small targets at a long distance, the detection distance is greatly improved. The deep learning model has high generalization and can work stably for different scenes. The stable dense monocular depth estimation makes the measurement of the motion state of the target object highly robust and can improve the detection distance.

[0083] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0084] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for measuring the motion state of a target object, characterized in that: include: Collecting real-time video data through an external camera, and obtaining target information from the real-time video data, wherein the target information includes a category identifier of the target, a tracking identifier of the target, and a bounding box of the target; By using the target object information, feature extraction and matching processing are performed on the first image of the i-th frame in the real-time video data and the second image of the i+1-th frame in the real-time video data, feature points of the first image and matching feature points in the second image located in the boundary frame of the target object are used as target feature point pairs, and the distance from the vehicle to the target object in the i-th frame and the i+1-th frame is calculated according to the pixel coordinates of the target feature point pairs; Obtaining the vehicle speed and acceleration of the vehicle at time t+Δt through an external sensor, calculating the travel distance of the vehicle in the x-axis direction from time t to time t+Δt according to the vehicle speed and the vehicle acceleration, calculating a first distance from the vehicle to the target object at time t and a second distance from the vehicle to the target object at time t+Δt, calculating the displacement of the target object within a time period of Δt through the first distance, the second distance and the travel distance, and calculating the speed of the target object at time t through the displacement, wherein Δt is a time interval; Calculating a first speed of the target object at time t+Δt and a second speed of the target object at time t+2Δt respectively, and obtaining an acceleration of the target object at time t by calculating the first speed and the second speed; The distance of the target object, the speed of the target object at time t, and the acceleration of the target object at time t are taken as the motion state of the target object and output to the outside.

2. The method for measuring the motion state of a target object according to claim 1, characterized in that: Acquiring target object information from the real-time video data, including: Inputting the real-time video data into a small target detection model, and obtaining the category identification of the target object and the bounding box of the target object through the small target detection model; The category identifier of the target object and the bounding box of the target object are input into a target tracking model, and the tracking identifier of the target object is obtained through the target tracking model.

3. The method for measuring the motion state of a target object according to claim 2, characterized in that: Inputting the real-time video data into a small target detection model, and obtaining the category identifier of the target and the bounding box of the target through the small target detection model, including: Setting an input resolution, converting an original image obtained from the real-time video data into first image data of the input resolution, inputting the first image data into the small target detection model, and obtaining a first detection result of the target object through the small target detection model; Performing lane line detection on the first image data, extracting lane line information, and calculating a lane line vanishing point based on the lane line information; Taking the lane line vanishing point as the center, cropping the original image into second image data of the size of the input resolution, inputting the second image data into the small target detection model, and obtaining a second detection result of the target object through the small target detection model; The first detection result and the second detection result are fused, and the generated category identification of the target object and the bounding box of the target object are output.

4. The method for measuring the motion state of a target object according to claim 1, wherein: By using the target object information, feature extraction and matching processing are performed on the first image of the i-th frame in the real-time video data and the second image of the i+1-th frame in the real-time video data, and feature points of the first image and matching feature points in the second image located in the boundary frame of the target object are used as target feature point pairs, including: Using the target object information, using a feature extraction algorithm to obtain feature points in the first image as a first feature point set, and obtaining feature points in the second image as a second feature point set; Perform feature matching on the feature points in the first feature point set and the second feature point set by using a matching algorithm to generate feature point pairs; The pixel coordinates of the feature points in the feature point pair are obtained, and if the pixel coordinates are both within the bounding box of the target object, the feature pair is used as a target feature point pair.

5. The method for measuring the motion state of a target object according to claim 1, wherein: The distance between the vehicle and the target object at the i-th frame and the i+1-th frame is calculated according to the pixel coordinates of the target feature point pair, including: Use a lightweight dense monocular depth estimation model to obtain the dense depth I of the first image i and the dense depth I of the second image i+1 ; According to the dense depth I i The pixel coordinates of the first image corresponding to the target feature point are calculated to obtain first depth data, a first average value of the depth data of all feature points is calculated according to the first depth data of all feature points, and the first average value is used as the distance from the vehicle to the target object in the i-th frame; According to the dense depth I i+1 The second depth data is calculated based on the pixel coordinates corresponding to the second image in the target feature point pair, and the second average value of the depth data of all feature points is calculated based on the second depth data of all feature points, and the second average value is used as the distance from the vehicle to the target object in the i+1th frame.

6. The method for measuring the motion state of a target object according to claim 1, characterized in that: Calculating the displacement of the target object within a time period of Δt by using the first distance, the second distance, and the travel distance, and calculating the speed of the target object at time t by using the displacement, includes: The instantaneous speed v at time t is calculated by using the first distance, the second distance and the travel distance. cid-tid-t , where the displacement d of the target object in the time period Δt is cid-tidi =d t+Δt +d ego -d t , cid is the category identifier of the target object, tid is the tracking identifier of the target object, d t is the first distance, d t+Δt is the second distance, d ego is the driving distance; By the displacement d cid-tidi Calculate the instantaneous velocity v at time t cid-tid-t , where the instantaneous velocity v cid-tid-t =v cid-tid-objavg =d cid-tidi / Δt,v cid-tid-objavg is the average speed of the target object in the time period Δt.

7. The method for measuring the motion state of a target object according to any one of claims 1 to 6, characterized in that: Calculating the travel distance of the vehicle in the x-axis direction from time t to time t+Δt according to the vehicle speed and the vehicle acceleration includes: The distance traveled by the vehicle in the x-axis direction Where v_x is the vehicle velocity in the x direction, acc_x is the vehicle acceleration in the x direction, and d ego is the travel distance of the vehicle in the x-axis direction.

8. A device for measuring the motion state of a target, characterized in that: include: A target detection module, used to collect real-time video data through an external camera, and obtain target information from the real-time video data, wherein the target information includes a category identifier of the target, a tracking identifier of the target, and a bounding box of the target; a distance measurement module, configured to perform feature extraction and matching processing on a first image of an i-th frame in the real-time video data and a second image of an i+1-th frame in the real-time video data based on the target object information, taking feature points of the first image and matching feature points in the second image located in a boundary frame of the target object as a target feature point pair, and calculating the distance from the vehicle to the target object in the i-th frame and the i+1-th frame according to the pixel coordinates of the target feature point pair; a speed measurement module, configured to obtain the vehicle speed and acceleration of the vehicle at time t+Δt through an external sensor, calculate the travel distance of the vehicle in the x-axis direction from time t to time t+Δt according to the vehicle speed and the vehicle acceleration, calculate a first distance from the vehicle to the target object at time t and a second distance from the vehicle to the target object at time t+Δt, calculate the displacement of the target object within a time period of Δt through the first distance, the second distance and the travel distance, and calculate the speed of the target object at time t through the displacement, wherein Δt is a time interval; An acceleration measurement module, used to calculate a first velocity of the target object at time t+Δt and a second velocity at time t+2Δt, respectively, and obtain an acceleration of the target object at time t by calculating the first velocity and the second velocity; The motion state acquisition module is used to take the distance of the target object, the speed of the target object at time t and the acceleration of the target object at time t as the motion state of the target object and output it to the outside.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for measuring the motion state of a target object according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method for measuring the motion state of a target object according to any one of claims 1 to 7.