Target position frame prediction method and device in image, equipment and storage medium
By obtaining target pixels from reference frames and combining vehicle information and device calibration parameters, the bounding box of the target object in the image is predicted in real time, solving the problem of low prediction accuracy in untrained scenarios in existing technologies and achieving high-precision and real-time bounding box prediction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA AUTOMOTIVE INNOVATION CORP
- Filing Date
- 2022-08-25
- Publication Date
- 2026-06-09
AI Technical Summary
Existing technologies have low accuracy in predicting the position of target objects in image frames in untrained scenarios.
By obtaining the target pixel from the location box of the first reference frame, and combining it with the current vehicle position, yaw angle, equipment calibration parameters and world position, the location box of the target object in the current frame image is predicted in real time.
It achieves high-precision bounding box prediction in any scene without the need for pre-training of the prediction model and has good real-time performance.
Smart Images
Figure CN115471549B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of autonomous driving technology, and in particular to a method, apparatus, device, and storage medium for predicting the location bounding box of a target in an image. Background Technology
[0002] With advancements in technologies such as artificial intelligence, electronic information, automatic control, and intelligent manufacturing, autonomous driving technology has experienced rapid development. In autonomous driving matching and tracking technology, it is necessary to predict the bounding box position of the target object within the current image frame, especially for targets such as pedestrians and vehicles, in order to match the target object's current trajectory information.
[0003] Currently, a prediction model is typically trained using the bounding box information of the target object in historical image frames. This model is then used to predict the position of the target object's bounding box in the current image frame. However, this method suffers from low prediction accuracy in scenarios where the object has not been trained. Summary of the Invention
[0004] This application provides a method, apparatus, device, and storage medium for predicting the bounding box of a target in an image, which can improve the prediction accuracy of the bounding box of a target in an image.
[0005] To achieve the above objectives, this application adopts the following technical solution:
[0006] In a first aspect, this application provides a method for predicting the bounding box of a target in an image. The method includes: obtaining a first target pixel from a first bounding box of a first reference frame, wherein the first reference frame is the previous frame image at the current time, the image is an image captured by a capture device on a vehicle, the first bounding box is the bounding box of the target object after target detection in the first reference frame, and the first target pixel is any point on the bottom edge of the first bounding box.
[0007] Based on the first target pixel, and based on the current vehicle position and yaw angle at the current moment, the first vehicle position and yaw angle at the first moment corresponding to the first reference frame, the first target device position, the first height and the first width of the first position box, and based on the preset calibration parameters of the acquisition device and the preset second target world position, the predicted bounding box corresponding to the target object in the current frame image is obtained; wherein, the first target device position is the position of the first target pixel in the device coordinate system of the acquisition device, the second target world position is the position of the second target pixel in the world coordinate system, and the second target pixel is the pixel corresponding to the first target pixel in the previous frame image of the first reference frame.
[0008] In one embodiment, a predicted bounding box corresponding to the target object in the current frame image is obtained based on the first target pixel, the current vehicle position and yaw angle at the current moment, the first vehicle position and yaw angle at the first moment corresponding to the first reference frame, the position of the first target device, the first height and the first width of the first location box, and the preset calibration parameters of the acquisition device and the preset second target world position. This includes:
[0009] Based on the calibration parameters, the position of the first vehicle, the yaw angle of the first vehicle, and the world position of the second target, the position of the first target pixel in the world coordinate system at the current moment is determined, and the current world position of the target is obtained.
[0010] Based on the current target world position, current vehicle position, current vehicle yaw angle, first vehicle position, first vehicle yaw angle, first target equipment position, calibration parameters, first height, and first width, determine the prediction box corresponding to the target object in the current frame image.
[0011] In one embodiment, determining the position of the first target pixel in the world coordinate system at the current moment, based on calibration parameters, the first vehicle position, the first vehicle yaw angle, and the second target world position, to obtain the current target world position includes:
[0012] Based on the calibration parameters, the position of the first vehicle, and the yaw angle of the first vehicle, the position of the first target pixel in the world coordinate system is determined, and the world position of the first target is obtained.
[0013] Based on the world positions of the first and second targets, predict the position of the first target pixel in the world coordinate system at the current moment to obtain the current world position of the target.
[0014] In one embodiment, determining the position of the first target pixel in the world coordinate system based on calibration parameters, the position of the first vehicle, and the yaw angle of the first vehicle, to obtain the world position of the first target, includes:
[0015] Based on the position and yaw angle of the first vehicle, determine the transformation relationship between the vehicle's body coordinate system and the world coordinate system at the first moment;
[0016] Based on the calibration parameters and transformation relationships, the position of the first target pixel in the world coordinate system is determined, and the world position of the first target is obtained.
[0017] In one embodiment, determining the prediction bounding box corresponding to the target object in the current frame image based on the current target world position, current vehicle position, current vehicle yaw angle, first vehicle position, first vehicle yaw angle, first target device position, calibration parameters, first height, and first width includes:
[0018] Based on the position and yaw angle of the first vehicle, determine the transformation relationship between the vehicle's body coordinate system and the world coordinate system at the first moment;
[0019] Based on the transformation relationship, determine the current target world position in the vehicle coordinate system to obtain the current target vehicle position, and determine the current vehicle position in the vehicle coordinate system to obtain the current vehicle position.
[0020] Based on the current target vehicle body position, current vehicle body position, current vehicle yaw angle, first vehicle yaw angle, first target equipment position, calibration parameters, first height, and first width, determine the prediction box corresponding to the target object in the current frame image.
[0021] In one embodiment, determining the prediction bounding box corresponding to the target object in the current frame image based on the current target vehicle position, the current vehicle body position, the current vehicle yaw angle, the first vehicle yaw angle, the first target device position, calibration parameters, the first height, and the first width includes:
[0022] Based on the current target vehicle position and the current vehicle body position, determine the position of the first target pixel relative to the vehicle at the current moment, and obtain the current target vehicle reference position;
[0023] Determine the yaw angle difference based on the current vehicle yaw angle and the first vehicle yaw angle;
[0024] The current target vehicle's reference position is corrected based on the yaw angle difference to obtain the current target vehicle's position;
[0025] Based on the calibration parameters, determine the current target vehicle position in the equipment coordinate system;
[0026] Based on the current target device location, calibration parameters, first target device location, first height, and first width, determine the prediction box corresponding to the target object in the current frame image.
[0027] In one embodiment, determining the prediction bounding box corresponding to the target object in the current frame image based on the current target device location, calibration parameters, first target device location, first height, and first width includes:
[0028] Based on the calibration parameters, the position of the current target device in the image is determined, and the target position of the first target pixel in the current frame image at the current time is obtained;
[0029] Based on the first target device position, first height, and first width, determine the current height and current width of the first position box in the current frame image;
[0030] The predicted bounding box of the target object in the current frame image is obtained based on the current height, current width, and target position.
[0031] A second aspect of this application provides an apparatus for predicting the bounding box of a target in an image, the apparatus comprising:
[0032] The acquisition module is used to acquire a first target pixel from a first location box of a first reference frame, wherein the first reference frame is the previous frame image at the current moment, the image is the image acquired by the acquisition device on the vehicle, the first location box is the bounding box of the target object after target detection in the first reference frame, and the first target pixel is any point on the bottom edge of the first location box.
[0033] The determination module is used to obtain the prediction box corresponding to the target object in the current frame image based on the first target pixel, and based on the current vehicle position and current vehicle yaw angle at the current time, the first vehicle position and first vehicle yaw angle at the first time corresponding to the first reference frame, the first target device position, the first height and the first width of the first position box, and based on the preset calibration parameters of the acquisition device and the preset second target world position; wherein, the first target device position is the position of the first target pixel in the device coordinate system of the acquisition device, the second target world position is the position of the second target pixel in the world coordinate system, and the second target pixel is the pixel corresponding to the first target pixel in the previous frame image of the first reference frame.
[0034] A third aspect of this application provides an electronic device including a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the method for predicting the location bounding box of a target in an image as described in the first aspect of this application.
[0035] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for predicting the location bounding box of a target in an image as described in the first aspect of this application.
[0036] The beneficial effects of the technical solutions provided in this application include at least the following:
[0037] The method for predicting the bounding box of a target in an image provided in this application embodiment obtains a first target pixel from a first bounding box in a first reference frame. Then, based on the first target pixel, and according to the current vehicle position and yaw angle at the current moment, the first vehicle position and yaw angle at the first moment corresponding to the first reference frame, the first target device position, the first height and the first width of the first bounding box, and according to preset calibration parameters of the acquisition device and a preset second target world position, a predicted bounding box corresponding to the target object in the current frame image is obtained. Here, the first reference frame is the previous frame image at the current moment, the image is the image acquired by the acquisition device on the vehicle, the first bounding box is the bounding box of the target object after target detection in the first reference frame, the first target pixel is any point on the bottom edge of the first bounding box, the first target device position is the position of the first target pixel in the device coordinate system of the acquisition device, the second target world position is the position of the second target pixel in the world coordinate system, and the second target pixel is the pixel corresponding to the first target pixel in the previous frame image of the first reference frame. The prediction method of this application predicts the bounding box of the target in the image based on the target pixels in the historical reference image of the target image, the current vehicle position information, and the yaw angle data. Since this method predicts based on real-time acquired data, it has high prediction accuracy for any scene and does not require pre-training of the prediction model, thus providing good real-time performance. Attached Figure Description
[0038] Figure 1 This is a schematic diagram of the structure of a vehicle-mounted terminal provided in an embodiment of this application;
[0039] Figure 2 The flowchart of the method for predicting the location bounding box of a target in an image provided in the embodiments of this application Figure 1 ;
[0040] Figure 3 A schematic diagram of the predicted target location bounding box in an image provided in an embodiment of this application;
[0041] Figure 4 The flowchart of the method for predicting the location bounding box of a target in an image provided in the embodiments of this application Figure 2 ;
[0042] Figure 5 This is a structural diagram of a device for predicting the location bounding box of a target in an image, provided in an embodiment of this application. Detailed Implementation
[0043] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0044] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more.
[0045] In addition, the use of “based on” or “according to” implies openness and inclusivity, because processes, steps, calculations or other actions “based on” or “according to” one or more conditions or values can in practice be based on additional conditions or values beyond those conditions.
[0046] With advancements in technologies such as artificial intelligence, electronic information, automatic control, and intelligent manufacturing, autonomous driving technology for automobiles has experienced rapid development. In autonomous driving matching and tracking technology, it is necessary to predict the position of the target object's bounding box within the current image frame in order to match the target object's current trajectory information.
[0047] Currently, a prediction model is typically trained using bounding box information from historical image frames of the target image. This model is then used to predict the location of the target object's bounding box in the current image frame. However, this method suffers from low prediction accuracy in scenarios where the target object has not been trained.
[0048] To address the aforementioned issues, this application provides a method for predicting the bounding box of a target in an image. The method involves obtaining a first target pixel from a first bounding box in a first reference frame. Then, based on the first target pixel, and according to the current vehicle position and yaw angle at the current moment, the first vehicle position and yaw angle at the first moment corresponding to the first reference frame, the first target device position, the first height and the first width of the first bounding box, and preset calibration parameters of the acquisition device and a preset second target world position, a predicted bounding box corresponding to the target object in the current frame image is obtained. Here, the first reference frame is the previous frame image at the current moment, the image is acquired by the acquisition device on the vehicle, the first bounding box is the bounding box of the target object after target detection in the first reference frame, the first target pixel is any point on the bottom edge of the first bounding box, the first target device position is the position of the first target pixel in the device coordinate system of the acquisition device, the second target world position is the position of the second target pixel in the world coordinate system, and the second target pixel is the pixel corresponding to the first target pixel in the previous frame image of the first reference frame. The prediction method of this application predicts the bounding box of the target in the image based on the target pixels in the historical reference image of the target image, the current vehicle position information, and the yaw angle data. Since this method predicts based on real-time acquired data, it has high prediction accuracy for any scene and does not require pre-training of the prediction model, thus providing good real-time performance.
[0049] The execution subject of the method for predicting the location box of a target in an image provided in this application embodiment can be an electronic device. Specifically, the electronic device can be a computer device, a terminal device, or a server. The terminal device can be an in-vehicle terminal, various personal computers, laptops, smartphones, tablets, and portable wearable devices, etc. This application does not make any specific limitations.
[0050] Figure 1 Taking the vehicle-mounted terminal as an example, the execution entity is shown below. Figure 1 This is a schematic diagram of the internal structure of a vehicle-mounted terminal provided in an embodiment of this application. Figure 1 As shown, the vehicle-mounted terminal includes a processor and a memory connected via a system bus. The processor provides computing and control capabilities. The memory may include non-volatile storage media and internal memory. The non-volatile storage media stores an operating system and computer programs. These computer programs can be executed by the processor to implement the steps of a method for predicting the location bounding box of a target in an image, as provided in the various embodiments above. The internal memory provides a cached runtime environment for the operating system and computer programs in the non-volatile storage media.
[0051] Those skilled in the art will understand that Figure 1The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the vehicle terminal to which the present application is applied. A specific vehicle terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0052] Based on the aforementioned execution entity, embodiments of this application provide a method for predicting the bounding box of a target in an image. For example... Figure 2 As shown, the method includes the following steps:
[0053] Step 201: Obtain the first target pixel from the first position box of the first reference frame.
[0054] Wherein, the first reference frame is the previous frame image at the current moment, the image is the image captured by the acquisition device on the vehicle, the first location box is the bounding box of the target object after target detection in the first reference frame, and the first target pixel is any point on the bottom edge of the first location box.
[0055] Optionally, the first target pixel can be the midpoint of the bottom edge of the first location box.
[0056] It should be noted that, since this application includes transforming the first target pixel to the world coordinate system, such as... Figure 3 As shown, it can be seen that the bottom edge of the detection box of the target object is located on the ground. Therefore, the selected target pixel is the pixel located on the bottom edge of the first position box.
[0057] Step 202: Based on the first target pixel, and based on the current vehicle position and current vehicle yaw angle at the current moment, the first vehicle position and first vehicle yaw angle at the first moment corresponding to the first reference frame, the first target device position, the first height and the first width of the first position box, and based on the preset calibration parameters of the acquisition device and the preset second target world position, obtain the prediction box corresponding to the target object in the current frame image.
[0058] Wherein, the first target device position is the position of the first target pixel in the device coordinate system of the acquisition device, the second target world position is the position of the second target pixel in the world coordinate system, and the second target pixel is the pixel corresponding to the first target pixel in the previous frame of the first reference frame.
[0059] For example, if the first target pixel is the midpoint of the bottom edge of the first location box, then the second target pixel is the midpoint of the bottom edge of the second location box, and the second location box is the bounding box of the target object obtained after target detection of the previous frame image of the first reference frame.
[0060] The current vehicle position at the current moment and the first vehicle position at the first moment are obtained from the location information transmitted by the vehicle's Global Positioning System (GPS). The current vehicle yaw angle at the current moment and the first vehicle yaw angle at the first moment are obtained from the vehicle's inertial measurement unit (IMU) and transmitted from it.
[0061] The calibration parameters of the acquisition equipment include intrinsic parameters and extrinsic parameters. The intrinsic parameters are the calibration parameters of the acquisition equipment relative to the images captured by the acquisition equipment, while the extrinsic parameters are the calibration parameters of the acquisition equipment relative to the vehicle.
[0062] Optionally, the position of the first target pixel in the current frame image can be predicted first by obtaining the current vehicle position, current vehicle yaw angle, first vehicle position, first vehicle yaw angle, and according to preset calibration parameters and preset second target world position, to obtain the current target pixel. Then, the height and width of the first position box in the current frame image can be determined according to the obtained first height, first width, and first target device position. Finally, the prediction box corresponding to the target object in the current frame image can be obtained according to the current target pixel and the height and width of the first position box in the current frame image.
[0063] The method for predicting the bounding box of a target in an image provided in this application embodiment obtains a first target pixel from a first bounding box in a first reference frame. Then, based on the first target pixel, and according to the current vehicle position and yaw angle at the current moment, the first vehicle position and yaw angle at the first moment corresponding to the first reference frame, the first target device position, the first height and the first width of the first bounding box, and according to preset calibration parameters of the acquisition device and a preset second target world position, a predicted bounding box corresponding to the target object in the current frame image is obtained. Here, the first reference frame is the previous frame image at the current moment, the image is the image acquired by the acquisition device on the vehicle, the first bounding box is the bounding box of the target object after target detection in the first reference frame, the first target pixel is any point on the bottom edge of the first bounding box, the first target device position is the position of the first target pixel in the device coordinate system of the acquisition device, the second target world position is the position of the second target pixel in the world coordinate system, and the second target pixel is the pixel corresponding to the first target pixel in the previous frame image of the first reference frame. The prediction method of this application predicts the bounding box of the target in the image based on the target pixels in the historical reference image of the target image, the current vehicle position information, and the yaw angle data. Since this method predicts based on real-time acquired data, it has high prediction accuracy for any scene and does not require pre-training of the prediction model, thus providing good real-time performance.
[0064] Optional, such as Figure 4As shown, step 202 above, which involves obtaining the prediction box corresponding to the target object in the current frame image based on the first target pixel, the current vehicle position and yaw angle at the current moment, the first vehicle position and yaw angle at the first moment corresponding to the first reference frame, the first target device position, the first height and the first width of the first position box, and the preset calibration parameters of the acquisition device and the preset second target world position, can be described as follows:
[0065] Step 401: Based on the calibration parameters, the position of the first vehicle, the yaw angle of the first vehicle, and the world position of the second target, determine the position of the first target pixel in the world coordinate system at the current moment, and obtain the current world position of the target.
[0066] Step 402: Determine the prediction box corresponding to the target object in the current frame image based on the current target world position, current vehicle position, current vehicle yaw angle, first vehicle position, first vehicle yaw angle, first target equipment position, calibration parameters, first height, and first width.
[0067] Understandably, since what can be obtained in real time is the vehicle's current position and yaw angle in world coordinates, it is necessary to first predict the target pixels in the current frame image.
[0068] The position coordinates of the first target pixel in the first reference frame are transformed to world coordinates to obtain the current world position of the target. Then, based on the current world position of the target, the current vehicle position, the current vehicle yaw angle, the first vehicle position, the first vehicle yaw angle, the first target device position, calibration parameters, the first height, and the first width, the prediction box corresponding to the target object in the current frame image is determined.
[0069] Optionally, step 401 above, which determines the position of the first target pixel in the world coordinate system at the current moment based on the calibration parameters, the position of the first vehicle, the yaw angle of the first vehicle, and the world position of the second target, can be performed as follows:
[0070] Based on the calibration parameters, the position of the first vehicle, and the yaw angle of the first vehicle, the position of the first target pixel in the world coordinate system is determined, and the world position of the first target is obtained. Then, based on the world position of the first target and the world position of the second target, the position of the first target pixel in the world coordinate system at the current moment is predicted, and the current world position of the target is obtained.
[0071] Specifically, the process of determining the position of the first target pixel in the world coordinate system based on the calibration parameters, the position of the first vehicle, and the yaw angle of the first vehicle, and obtaining the world position of the first target, can be as follows: Based on the position of the first vehicle and the yaw angle of the first vehicle, determine the transformation relationship between the vehicle body coordinate system and the world coordinate system at the first moment, and then determine the position of the first target pixel in the world coordinate system based on the calibration parameters and the transformation relationship, thereby obtaining the world position of the first target.
[0072] Optionally, step 402 above, which involves determining the prediction box corresponding to the target object in the current frame image based on the current target world position, current vehicle position, current vehicle yaw angle, first vehicle position, first vehicle yaw angle, first target device position, calibration parameters, first height, and first width, can be as follows:
[0073] Based on the first vehicle position and the first vehicle yaw angle, the transformation relationship between the vehicle body coordinate system and the world coordinate system at the first moment is determined. Then, based on the transformation relationship, the position of the current target world position in the vehicle body coordinate system is determined, thus obtaining the current target vehicle position. The position of the current vehicle position in the vehicle body coordinate system is also determined, thus obtaining the current vehicle position. Finally, based on the current target vehicle position, the current vehicle body position, the current vehicle yaw angle, the first vehicle yaw angle, the first target device position, calibration parameters, the first height, and the first width, the prediction box corresponding to the target object in the current frame image is determined.
[0074] Specifically, the process of determining the prediction box corresponding to the target object in the current frame image based on the current target vehicle position, current vehicle position, current vehicle yaw angle, first vehicle yaw angle, first target device position, calibration parameters, first height, and first width can be as follows: Based on the current target vehicle position and current vehicle position, determine the position of the first target pixel relative to the vehicle at the current moment to obtain the current target vehicle reference position; determine the yaw angle difference based on the current vehicle yaw angle and the first vehicle yaw angle; correct the current target vehicle reference position based on the yaw angle difference to obtain the current target vehicle position; determine the current target device position in the device coordinate system based on the calibration parameters; and finally, determine the prediction box corresponding to the target object in the current frame image based on the current target device position, calibration parameters, first target device position, first height, and first width.
[0075] Furthermore, the process of determining the prediction box corresponding to the target object in the current frame image based on the current target device position, calibration parameters, first target device position, first height, and first width can be as follows: determine the position of the current target device in the image based on the calibration parameters, obtain the target position of the first target pixel in the current frame image at the current time, then determine the current height and current width of the first position box in the current frame image based on the first target device position, first height, and first width, and finally obtain the prediction box corresponding to the target object in the current frame image based on the current height, current width, and target position.
[0076] To facilitate understanding by those skilled in the art, this application provides a method for predicting the bounding box of a target in an image, using an in-vehicle terminal as an example. Specifically, the method includes:
[0077] (1) Obtain the first target pixel from the first position box of the first reference frame.
[0078] Wherein, the first reference frame is the previous frame image at the current moment, the image is the image captured by the acquisition device on the vehicle, the first location box is the bounding box of the target object after target detection in the first reference frame, and the first target pixel is any point on the bottom edge of the first location box.
[0079] In actual execution, for target object M, there is already historical trajectory information. The first target pixel is obtained from the first reference frame. The pixels of the first target pixel cb are (u, v).
[0080] (2) Based on the position of the first vehicle and the yaw angle of the first vehicle, determine the transformation relationship between the vehicle body coordinate system and the world coordinate system at the first moment.
[0081] Among them, the first vehicle position information is the position information of the vehicle in the world coordinate system at the first moment corresponding to the first reference frame, and the first vehicle yaw angle is the yaw angle of the vehicle at the first moment.
[0082] The first vehicle position at the first moment is the position information transmitted by the vehicle's Global Positioning System (GPS). This first vehicle position information is the vehicle's position information in the world coordinate system.
[0083] Optionally, this transformation relationship includes: the translation matrix T between the vehicle coordinate system and the world coordinate system. t-1 The rotation matrix R between the vehicle coordinate system and the world coordinate system t-1 For example, the rear axle center of the vehicle can be used as the origin of the vehicle body coordinate system (VCS), and then the translation matrix T between the vehicle body coordinate system and the world coordinate system at the first moment can be determined based on the first vehicle position. t-1And determine the rotation matrix R between the vehicle coordinate system and the world coordinate system at the first moment based on the first vehicle yaw angle. t-1 .
[0084] (3) Determine the position of the first target pixel in the world coordinate system based on the calibration parameters and transformation relationship, and obtain the world position of the first target.
[0085] The calibration parameters refer to the calibration parameters of the data acquisition device. Specifically, the calibration parameters include the intrinsic parameter K and the extrinsic parameter R. cam T cam Intrinsic parameters are the calibration parameters of the acquisition device relative to the image captured by the acquisition device, while extrinsic parameters are the calibration parameters of the acquisition device relative to the vehicle. The first target world position is the position of the first target pixel in the world coordinate system.
[0086] Optionally, the coordinates of the first target pixel in the vehicle coordinate system can be calculated using the following formula (1). At the same time, the value s of the first target pixel (u, v) in the Z-axis direction of the camera coordinate system can be calculated.
[0087]
[0088] Then, through the rotation matrix R at the first moment t -1 and translation matrix T t -1 and formula (2) can transform the point in the VCS coordinate system to the world coordinate system, obtaining the world position of the first target, which is: Formula (2) is:
[0089]
[0090] (4) Based on the world position of the first target and the world position of the second target, predict the position of the first target pixel in the world coordinate system at the current moment, and obtain the current world position of the target.
[0091] Wherein, the second target world position is the position of the second target pixel in the world coordinate system, and the second target pixel is the pixel corresponding to the first target pixel in the previous frame of the first reference frame.
[0092] For example, if the first target pixel is the midpoint of the bottom edge of the first location box, then the second target pixel is the midpoint of the bottom edge of the second location box, and the second location box is the bounding box of the target object obtained after target detection of the previous frame image of the first reference frame.
[0093] It is understandable that, since this application is for a continuous prediction process of multiple image frames in a video, the world position of the second target has been pre-calculated when the previous frame was predicted. Specifically, the process of determining the world position of the second target can be referred to the above steps (1)-(3).
[0094] Optionally, the above process can be: based on the first target world location Second target world position The displacement of the first target pixel is calculated. Then, based on the time interval between the first moment corresponding to the first reference frame and the second moment corresponding to the previous frame of the first reference frame, the velocity of the first target pixel is obtained according to the displacement and the time interval. Based on the velocity and the time interval between the current moment and the first moment, the displacement of the first target pixel from the first moment to the current moment is obtained. Adding the first target world position, the current target world position is obtained.
[0095] (5) Based on the transformation relationship, determine the position of the current target world position in the vehicle coordinate system to obtain the current target vehicle position, and determine the position of the current vehicle position in the vehicle coordinate system to obtain the current vehicle position.
[0096] Assume the vehicle's current position in the world coordinate system is... Through the rotation matrix R at the first moment t-1 Translation matrix T t-1 The current vehicle position at the current moment and current target world location Rotate to the VCS coordinate system to obtain the current position of the target vehicle. and the current vehicle body position
[0097] The specific conversion process can be shown in formula (2).
[0098] (6) Based on the current target vehicle position and the current vehicle body position, determine the position of the first target pixel relative to the vehicle at the current time, and obtain the current target vehicle reference position.
[0099] Optionally, by calculation Obtain the current reference position of the target vehicle.
[0100] (7) Determine the yaw angle difference based on the current vehicle yaw angle and the first vehicle yaw angle.
[0101] Calculate the current vehicle yaw angle. t and the first vehicle's yaw angle t-1 The difference is used to obtain the yaw angle difference ΔYaw.
[0102] (8) Correct the current target vehicle reference position based on the yaw angle difference to obtain the current target vehicle position.
[0103] Optionally, the position of the target vehicle after the rotational yaw angle difference ΔYaw from the current reference position can be calculated according to formula (3), thus obtaining the current target vehicle position.
[0104]
[0105] (9) Determine the current target equipment position in the equipment coordinate system based on the calibration parameters.
[0106] Optionally, the external parameters R of the acquisition device can be used as a reference. cam T cam The current location of the target vehicle Switch to the camera coordinate system to obtain the current position of the target device.
[0107] (10) Determine the position of the current target device in the image based on the calibration parameters, and obtain the target position of the first target pixel in the current frame image at the current time.
[0108] Optionally, the current target device location can be determined based on the intrinsic parameter K of the acquisition device. Transform to the image coordinate system to obtain the target position new_cb of the first target pixel in the current frame image at the current time.
[0109] (11) Determine the current height and current width of the first position box in the current frame image based on the first target device position, the first height and the first width.
[0110] Wherein, the first target device position is the position of the first target pixel in the device coordinate system of the acquisition device. Optionally, the first target device position can be obtained by transforming the first target pixel using the intrinsic parameter K.
[0111] Understandably, because the distance between the target object and the acquisition device varies at different times, the size of the target object in the image also varies at different times, and consequently, the size of the target object's bounding box in the image also varies at different times. The height and width of the target object's bounding box are proportional to the distance between the target object and the acquisition device. Therefore, based on the current position of the target device and the first height H of the first bounding box... t-1 and the first width W t-1 Calculate the current height H of the target object's bounding box in the current frame image. t and current width W t .
[0112]
[0113]
[0114] (12) Obtain the prediction box corresponding to the target object in the current frame image based on the current height, current width and target position.
[0115] Since the pixel coordinates of the first target pixel in the current frame image have been predicted, and the current height and width of the first location box in the current frame image have been calculated, the predicted box corresponding to the target object in the current frame image can be obtained.
[0116] It should be noted that steps (1)-(12) above illustrate the method for predicting the bounding box of a target in an image using one target object as an example. The method for predicting the bounding boxes of other target objects included in the image can be obtained based on the above method. Figure 3 The diagram shows the predicted bounding box of a target object according to the method for predicting the bounding box of a target in an image provided in this application. The dashed lines represent the bounding boxes of the first reference frame, and the solid lines represent the bounding boxes of the predicted bounding boxes of the current frame image.
[0117] It should be understood that while the steps in the flowchart of the above embodiments are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowchart may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the steps or stages in other steps.
[0118] like Figure 5 As shown, this application embodiment also provides a device for predicting the bounding box of a target in an image, the device comprising:
[0119] The acquisition module 11 is used to acquire a first target pixel from a first location box of a first reference frame, wherein the first reference frame is the previous frame image at the current moment, the image is the image acquired by the acquisition device on the vehicle, the first location box is the bounding box of the target object after target detection in the first reference frame, and the first target pixel is any point on the bottom edge of the first location box.
[0120] The determining module 12 is used to obtain the prediction box corresponding to the target object in the current frame image based on the first target pixel, and based on the current vehicle position and current vehicle yaw angle at the current time, the first vehicle position and first vehicle yaw angle at the first time corresponding to the first reference frame, the first target device position, the first height and the first width of the first position box, and based on the preset calibration parameters of the acquisition device and the preset second target world position; wherein, the first target device position is the position of the first target pixel in the device coordinate system of the acquisition device, the second target world position is the position of the second target pixel in the world coordinate system, and the second target pixel is the pixel corresponding to the first target pixel in the previous frame image of the first reference frame.
[0121] In one embodiment, the determining module 12 is specifically used for:
[0122] Based on the calibration parameters, the position of the first vehicle, the yaw angle of the first vehicle, and the world position of the second target, the position of the first target pixel in the world coordinate system at the current moment is determined, and the current world position of the target is obtained.
[0123] Based on the current target world position, current vehicle position, current vehicle yaw angle, first vehicle position, first vehicle yaw angle, first target equipment position, calibration parameters, first height, and first width, determine the prediction box corresponding to the target object in the current frame image.
[0124] In one embodiment, the determining module 12 is specifically used for:
[0125] Based on the calibration parameters, the position of the first vehicle, and the yaw angle of the first vehicle, the position of the first target pixel in the world coordinate system is determined, and the world position of the first target is obtained.
[0126] Based on the world positions of the first and second targets, predict the position of the first target pixel in the world coordinate system at the current moment to obtain the current world position of the target.
[0127] In one embodiment, the determining module 12 is specifically used for:
[0128] Based on the position and yaw angle of the first vehicle, determine the transformation relationship between the vehicle's body coordinate system and the world coordinate system at the first moment;
[0129] Based on the calibration parameters and transformation relationships, the position of the first target pixel in the world coordinate system is determined, and the world position of the first target is obtained.
[0130] In one embodiment, the determining module 12 is specifically used for:
[0131] Based on the position and yaw angle of the first vehicle, determine the transformation relationship between the vehicle's body coordinate system and the world coordinate system at the first moment;
[0132] Based on the transformation relationship, determine the current target world position in the vehicle coordinate system to obtain the current target vehicle position, and determine the current vehicle position in the vehicle coordinate system to obtain the current vehicle position.
[0133] Based on the current target vehicle body position, current vehicle body position, current vehicle yaw angle, first vehicle yaw angle, first target equipment position, calibration parameters, first height, and first width, determine the prediction box corresponding to the target object in the current frame image.
[0134] In one embodiment, the determining module 12 is specifically used for:
[0135] Based on the current target vehicle position and the current vehicle body position, determine the position of the first target pixel relative to the vehicle at the current moment, and obtain the current target vehicle reference position;
[0136] Determine the yaw angle difference based on the current vehicle yaw angle and the first vehicle yaw angle;
[0137] The current target vehicle's reference position is corrected based on the yaw angle difference to obtain the current target vehicle's position;
[0138] Based on the calibration parameters, determine the current target vehicle position in the equipment coordinate system;
[0139] Based on the current target device location, calibration parameters, first target device location, first height, and first width, determine the prediction box corresponding to the target object in the current frame image.
[0140] In one embodiment, the determining module 12 is specifically used for:
[0141] Based on the calibration parameters, the position of the current target device in the image is determined, and the target position of the first target pixel in the current frame image at the current time is obtained;
[0142] Based on the first target device position, first height, and first width, determine the current height and current width of the first position box in the current frame image;
[0143] The predicted bounding box of the target object in the current frame image is obtained based on the current height, current width, and target position.
[0144] The target location bounding box prediction device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described in detail here.
[0145] Specific limitations regarding the device for predicting the bounding box of a target in an image can be found in the limitations of the method for predicting the bounding box of a target in an image above, and will not be repeated here. Each module in the aforementioned device for predicting the bounding box of a target in an image can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor of an electronic device in hardware form, or stored in the memory of an electronic device in software form, so that the processor can call and execute the corresponding operations of each module.
[0146] In another embodiment of this application, an electronic device is also provided, including a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the steps of the method for predicting the location box of a target in an image as described in the embodiments of this application.
[0147] In another embodiment of this application, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method for predicting the location box of a target in an image as described in the embodiments of this application.
[0148] In another embodiment of this application, a computer program product is also provided, which includes computer instructions that, when executed on a device for predicting the location box of a target in an image, cause the device for predicting the location box of a target in an image to perform each step of the method flow shown in the above method embodiment for predicting the location box of a target in an image.
[0149] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks, SSDs).
[0150] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0151] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A method for predicting the bounding box of a target in an image, characterized in that, The method includes: The first target pixel is obtained from the first location box of the first reference frame, wherein the first reference frame is the previous frame image at the current time, the image is the image captured by the acquisition device on the vehicle, the first location box is the bounding box of the target object after target detection in the first reference frame, and the first target pixel is any point on the bottom edge of the first location box. Based on the first target pixel, and based on the current vehicle position and yaw angle at the current moment, the first vehicle position and yaw angle at the first moment corresponding to the first reference frame, the first target device position, the first height and the first width of the first location box, and based on the preset calibration parameters of the acquisition device and the preset second target world position, the prediction box corresponding to the target object in the current frame image is obtained, including: Determine the position of the first target pixel in the world coordinate system to obtain the world position of the first target; Based on the first target world position and the second target world position, predict the position of the first target pixel in the world coordinate system at the current moment to obtain the current target world position; wherein, the second target world position is the position of the second target pixel in the world coordinate system, and the second target pixel is the pixel corresponding to the first target pixel in the previous frame of the first reference frame. Based on the position of the first vehicle and the yaw angle of the first vehicle, determine the transformation relationship between the vehicle body coordinate system and the world coordinate system at the first moment; Based on the transformation relationship, the current target world position is determined in the vehicle coordinate system to obtain the current target vehicle position, and the current vehicle position is determined in the vehicle coordinate system to obtain the current vehicle position. Based on the current target vehicle position and the current vehicle body position, determine the position of the first target pixel relative to the vehicle at the current moment, and obtain the current target vehicle reference position; The yaw angle difference is determined based on the current vehicle yaw angle and the first vehicle yaw angle. The current target vehicle reference position is corrected based on the yaw angle difference to obtain the current target vehicle position; Determine the current target vehicle position in the device coordinate system of the acquisition device, and determine the position of the current target device in the image, to obtain the target position of the first target pixel in the current frame image at the current time; Based on the location of the first target device and the first height and first width of the first location box, the current height and current width of the first location box in the current frame image are determined, wherein the location of the first target device is the position of the first target pixel in the device coordinate system of the acquisition device; Based on the current height, the current width, and the target position, the prediction bounding box corresponding to the target object in the current frame image is obtained.
2. The method according to claim 1, characterized in that, The step of obtaining the prediction box corresponding to the target object in the current frame image based on the first target pixel, and based on the current vehicle position and current vehicle yaw angle at the current moment, the first vehicle position and first vehicle yaw angle at the first moment corresponding to the first reference frame, the first target device position, the first height and first width of the first position box, and based on the preset calibration parameters of the acquisition device and the preset second target world position, includes: Based on the calibration parameters, the position of the first vehicle, the yaw angle of the first vehicle, and the world position of the second target, the position of the first target pixel in the world coordinate system at the current moment is determined, and the current world position of the target is obtained. Based on the current target world position, the current vehicle position, the current vehicle yaw angle, the first vehicle position, the first vehicle yaw angle, the first target device position, the calibration parameters, the first height, and the first width, determine the prediction box corresponding to the target object in the current frame image.
3. The method according to claim 2, characterized in that, The step of determining the position of the first target pixel in the world coordinate system at the current moment, based on the calibration parameters, the position of the first vehicle, the yaw angle of the first vehicle, and the world position of the second target, to obtain the current world position of the target includes: Based on the calibration parameters, the position of the first vehicle, and the yaw angle of the first vehicle, the position of the first target pixel in the world coordinate system is determined, and the world position of the first target is obtained. Based on the first target world position and the second target world position, predict the position of the first target pixel in the world coordinate system at the current moment to obtain the current target world position.
4. The method according to claim 3, characterized in that, The step of determining the position of the first target pixel in the world coordinate system based on the calibration parameters, the position of the first vehicle, and the yaw angle of the first vehicle, to obtain the world position of the first target, includes: Based on the position of the first vehicle and the yaw angle of the first vehicle, determine the transformation relationship between the vehicle body coordinate system and the world coordinate system at the first moment; Based on the calibration parameters and the transformation relationship, the position of the first target pixel in the world coordinate system is determined, and the world position of the first target is obtained.
5. The method according to claim 2, characterized in that, The step of determining the prediction box corresponding to the target object in the current frame image based on the current target world position, the current vehicle position, the current vehicle yaw angle, the first vehicle position, the first vehicle yaw angle, the first target device position, the calibration parameters, the first height, and the first width includes: Based on the current target vehicle position, the current vehicle body position, the current vehicle yaw angle, the first vehicle yaw angle, the first target device position, the calibration parameters, the first height, and the first width, the prediction box corresponding to the target object in the current frame image is determined.
6. The method according to claim 5, characterized in that, The step of determining the prediction box corresponding to the target object in the current frame image based on the current target vehicle body position, the current vehicle yaw angle, the first vehicle yaw angle, the first target device position, the calibration parameters, the first height, and the first width includes: Based on the calibration parameters, determine the current target device position in the device coordinate system; Based on the current target device location, the calibration parameters, the first target device location, the first height, and the first width, the prediction box corresponding to the target object in the current frame image is determined.
7. The method according to claim 6, characterized in that, The step of determining the prediction box corresponding to the target object in the current frame image based on the current target device position, the calibration parameters, the first target device position, the first height, and the first width includes: Based on the calibration parameters, the position of the current target device in the image is determined, and the target position of the first target pixel in the current frame image at the current time is obtained; Based on the first target device position, the first height, and the first width, determine the current height and current width of the first position frame in the current frame image; The predicted bounding box of the target object in the current frame image is obtained based on the current height, current width, and target position.
8. A device for predicting the location bounding box of a target in an image, characterized in that, The device includes: The acquisition module is used to acquire a first target pixel from a first location box of a first reference frame, wherein the first reference frame is the previous frame image at the current moment, the image is an image acquired by the acquisition device on the vehicle, the first location box is the bounding box of the target object after target detection in the first reference frame, and the first target pixel is any point on the bottom edge of the first location box. The determination module is used to obtain a prediction box corresponding to the target object in the current frame image based on the first target pixel, and based on the current vehicle position and current vehicle yaw angle at the current time, the first vehicle position and first vehicle yaw angle at the first time corresponding to the first reference frame, the position of the first target device, the first height and the first width of the first position box, and based on the preset calibration parameters of the acquisition device and the preset second target world position, including: Determine the position of the first target pixel in the world coordinate system to obtain the world position of the first target; Based on the first target world position and the second target world position, predict the position of the first target pixel in the world coordinate system at the current moment to obtain the current target world position; wherein, the second target world position is the position of the second target pixel in the world coordinate system, and the second target pixel is the pixel corresponding to the first target pixel in the previous frame of the first reference frame. Based on the position of the first vehicle and the yaw angle of the first vehicle, determine the transformation relationship between the vehicle body coordinate system and the world coordinate system at the first moment; Based on the transformation relationship, the current target world position is determined in the vehicle coordinate system to obtain the current target vehicle position, and the current vehicle position is determined in the vehicle coordinate system to obtain the current vehicle position. Based on the current target vehicle position and the current vehicle body position, determine the position of the first target pixel relative to the vehicle at the current moment, and obtain the current target vehicle reference position; The yaw angle difference is determined based on the current vehicle yaw angle and the first vehicle yaw angle. The current target vehicle reference position is corrected based on the yaw angle difference to obtain the current target vehicle position; Determine the current target vehicle position in the device coordinate system of the acquisition device, and determine the position of the current target device in the image, to obtain the target position of the first target pixel in the current frame image at the current time; Based on the location of the first target device and the first height and first width of the first location box, the current height and current width of the first location box in the current frame image are determined, wherein the location of the first target device is the position of the first target pixel in the device coordinate system of the acquisition device; Based on the current height, the current width, and the target position, the prediction bounding box corresponding to the target object in the current frame image is obtained.
9. An electronic device, characterized in that, The device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the method for predicting the bounding box of a target in an image as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method for predicting the bounding box of a target in an image as described in any one of claims 1 to 7.