Target object positioning method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides a target object positioning method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: at a target synchronization moment in a point cloud scanning process, controlling an image sensor to carry out image acquisition to obtain a point cloud frame and an image frame; the point cloud frame comprises an initial position and an initial azimuth angle corresponding to the scanned target object; determining a compensation time coefficient corresponding to the target object based on the target synchronization moment, the scanning angle corresponding to the target synchronization moment and the initial position; updating the initial position and the initial azimuth angle based on the compensation time coefficient, and determining a target position and a target azimuth angle corresponding to the target object; and identifying the target object in the image frame according to the target position and the target azimuth angle. According to the invention, the accuracy of target object positioning can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a target object positioning method and device, electronic equipment, a computer readable storage medium and a computer program product. BACKGROUND

[0002] At present, when a vehicle equipped with a laser radar and a camera is driving, the point cloud data obtained by laser radar scanning and the image data collected by the camera are usually used for joint positioning of surrounding objects. However, due to the mechanical mechanism of the laser radar itself (such as a rotating laser radar), a certain time is needed to complete the scanning, which leads to the time required for the laser radar to obtain the point cloud data being longer than the time required for the camera to collect the image data. This time difference will cause the positions of objects to be misaligned in the joint positioning process (for example, in an image, the point cloud frame and the image frame of the same object cannot be aligned), especially in the joint positioning process of objects with relatively high speed. Therefore, how to improve the accuracy of joint positioning has become a problem to be solved. SUMMARY

[0003] The present application provides a target object positioning method, device, electronic equipment, computer readable storage medium and computer program product, which can improve the accuracy of target object positioning.

[0004] The technical solution of the present application is as follows:

[0005] The present application provides a target object positioning method, which comprises:

[0006] Data acquisition is performed at a target synchronization time to obtain a point cloud frame and an image frame; the point cloud frame contains an initial position and an initial azimuth angle corresponding to a target object;

[0007] Based on the scanning angle corresponding to the target synchronization time and the initial position, a compensation time coefficient corresponding to the target object is determined;

[0008] Based on the compensation time coefficient, the initial position and the initial azimuth angle are updated to determine a target position and a target azimuth angle corresponding to the target object.

[0009] The present application provides a target object positioning device, which comprises:

[0010] A control module is configured to perform data acquisition at a target synchronization time to obtain a point cloud frame and an image frame; the point cloud frame contains an initial position and an initial azimuth angle corresponding to a target object;

[0011] determining a compensation time coefficient corresponding to the target object based on the initial position and the scanning angle corresponding to the target synchronization moment;

[0012] updating the initial position and the initial azimuth angle based on the compensation time coefficient to determine a target position and a target azimuth angle corresponding to the target object.

[0013] Optionally, the determining module is further configured to determine a time difference between the target synchronization moment and a corresponding scanning start moment of the point cloud frame, determine a compensation angle based on a line connecting the initial position and a scanning center point and an included angle between scanning lines of the target synchronization moment, and determine the compensation time coefficient based on a ratio of the compensation angle to the scanning angle and the time difference.

[0014] Optionally, the initial position and the initial azimuth angle are in a first coordinate system, and the first coordinate system corresponds to the point cloud frame; the updating module is further configured to obtain a corresponding relationship between the first coordinate system and a second coordinate system, convert the initial position and the initial azimuth angle to the second coordinate system based on the corresponding relationship to obtain a position to be compensated and an azimuth angle to be compensated, determine a linear velocity and an angular velocity of the target object, update the position to be compensated based on the compensation time coefficient and the linear velocity to determine an updated position, update the azimuth angle to be compensated based on the compensation time coefficient and the angular velocity to determine an updated azimuth angle, and convert the updated position and the updated azimuth angle to the first coordinate system to determine the target position and the target azimuth angle.

[0015] Optionally, the updating module is further configured to determine a position compensation amount based on a product of the compensation time coefficient and the linear velocity, increase the position to be compensated by the position compensation amount to determine the updated position in a case where it is determined based on a point cloud rotation direction that the target object is within the scanning angle, and decrease the position to be compensated by the position compensation amount to determine the updated position in a case where it is determined based on the point cloud rotation direction that the target object is outside the scanning angle.

[0016] Optionally, the updating module is further configured to determine an azimuth angle compensation amount based on a product of the compensation time coefficient and the angular velocity, increase the azimuth angle to be compensated by the azimuth angle compensation amount to determine the updated azimuth angle in a case where it is determined based on a point cloud rotation direction that the target object is within the scanning angle, and decrease the azimuth angle to be compensated by the azimuth angle compensation amount to determine the updated azimuth angle in a case where it is determined based on the point cloud rotation direction that the target object is outside the scanning angle.

[0017] Optionally, the determining module is further configured to determine an initial compensation time coefficient according to a product of the ratio and the time difference; and determine the compensation time coefficient based on the initial compensation time coefficient and a target fluctuation value.

[0018] Optionally, the target object positioning apparatus further comprises a de-distortion module, configured to acquire original point cloud data of a point cloud scan; and perform de-distortion processing on the original point cloud data at the target synchronization moment to determine the point cloud frame.

[0019] Optionally, the determining module is further configured to determine at least one target synchronization moment in a point cloud scanning process according to a radar scanning frequency and an image acquisition frequency of the image sensor.

[0020] Optionally, the determining module is further configured to determine an object within a target distance threshold as the target object.

[0021] The present application provides an electronic device, which comprises:

[0022] a memory configured to store computer executable instructions;

[0023] a processor configured to execute the computer executable instructions stored in the memory to implement the target object positioning method provided by the embodiments of the present application.

[0024] The present application provides a vehicle, which comprises an image sensor, a radar for point cloud scanning, and the above-mentioned electronic device provided by the present application.

[0025] The present application provides a computer readable storage medium, which stores a computer program or computer executable instructions, and is configured to be executed by a processor to implement the target object positioning method provided by the present application.

[0026] The present application provides a computer program product, which comprises a computer program or computer executable instructions, and is configured to be executed by a processor to implement the target object positioning method provided by the present application.

[0027] The present application has the following beneficial effects:

[0028] By acquiring data at the target synchronization moment, point cloud frames and image frames are obtained. Based on the scanning angle corresponding to the target synchronization moment and the initial position of the target object in the point cloud frame, a compensation time coefficient is determined. Based on the compensation time coefficient, the initial position and initial azimuth of the target object are updated to determine the target position and target azimuth of the target object. Thus, the point cloud frame and image frame are synchronized, the initial position and initial azimuth of the target object in the point cloud frame are compensated, and the target position and target azimuth of the target object in the image frame are obtained. This achieves data alignment between the point cloud and image sensors, improving the accuracy of target object localization. Attached Figure Description

[0029] Figure 1A This is a schematic diagram illustrating the discrepancy between the current point cloud frame and the obstacle image;

[0030] Figure 1B This is Schematic 2, showing the current discrepancy between the point cloud frame and the obstacle image;

[0031] Figure 2A This illustrates the discrepancy between the current point cloud frame and the obstacle image. Figure 3 ;

[0032] Figure 2B This is diagram four illustrating the discrepancy between the current point cloud frame and the obstacle image;

[0033] Figure 3 This is an optional flowchart illustrating the target object location method provided in the embodiments of this application;

[0034] Figure 4A This is an optional schematic diagram of the target synchronization time provided in the embodiments of this application;

[0035] Figure 4B This is a schematic diagram illustrating one of the optional scanning angles and compensation angles provided in an embodiment of this application.

[0036] Figure 5 This is a schematic diagram illustrating the positioning error that can be compensated by the target object positioning method provided in the embodiments of this application;

[0037] Figure 6A This is an optional schematic diagram of the target synchronization time and corresponding scanning angle provided in the embodiments of this application;

[0038] Figure 6B This is an optional schematic diagram of the target synchronization time and corresponding scanning angle provided in the embodiments of this application;

[0039] Figure 7A This is a schematic diagram of the positioning effect without using the target object positioning method provided in the embodiments of this application;

[0040] Figure 7B This is a schematic diagram illustrating the positioning effect of the target object positioning method provided in the embodiments of this application;

[0041] Figure 8A This is a second schematic diagram illustrating the positioning effect without using the target object positioning method provided in the embodiments of this application;

[0042] Figure 8B This is a second schematic diagram illustrating the positioning effect of the target object positioning method provided in the embodiments of this application;

[0043] Figure 9A This is a schematic diagram of the positioning effect without using the target object positioning method provided in the embodiments of this application. Figure 3 ;

[0044] Figure 9B This is a schematic diagram illustrating the positioning effect of the target object positioning method provided in the embodiments of this application. Figure 3 ;

[0045] Figure 10 This is an optional structural schematic diagram of the target object positioning device provided in the embodiments of this application;

[0046] Figure 11 This is a schematic diagram of an optional structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0049] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0050] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0051] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0052] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0053] Currently, vehicles equipped with LiDAR and cameras typically use a combination of point cloud data obtained from LiDAR scanning and image data captured by the camera to locate surrounding obstacles during operation. Current technologies directly project the point cloud points or pre-drawn point cloud bounding boxes (e.g., eight-dot boxes) of obstacles onto the image captured by the camera. However, due to a delay (e.g., 100ms) in point cloud generation from the end of a frame to the beginning, a discrepancy arises between the point cloud bounding boxes of obstacles and the obstacle image in the camera's captured image. This is especially problematic when scanning an object moving at a relatively high speed relative to the vehicle; the object's point cloud bounding box may deviate, causing its projected bounding box in the image to lag or lead its actual position. Figure 1A and Figure 1B As shown, when this vehicle overtakes the obstacle (vehicle 1) on its left, the point cloud frame corresponding to vehicle 1 ( Figure 1A Point cloud frame 10, Figure 1B The point cloud frame 13) cannot contain the point cloud. Figure 1A or Figure 1B The vehicle image in the image, the projection frame corresponding to the point cloud frame ( Figure 1A Projection frame 11 in the middle, Figure 1B The projection frame 14 in the image lags behind the image frame of vehicle 1. Figure 1A Image frame 12 in the middle, Figure 1B Image frame 15). The image frame of vehicle 1 is used to indicate the position of vehicle 1 in the image.

[0054] like Figure 2A and Figure 2B As shown, when this vehicle overtakes the obstacle (vehicle 2) on its right, the point cloud frame corresponding to vehicle 2 ( Figure 2A Point cloud frame 20, Figure 2B The point cloud frame 23) cannot contain Figure 2A or Figure 2BThe vehicle image in the image, the projection frame corresponding to the point cloud frame ( Figure 2A Projection frame 21 in the middle, Figure 2B The projection frame 24 in the image is ahead of the image frame of vehicle 2. Figure 2A Image frame 22 in the middle, Figure 2B Image frame 25). The image frame of vehicle 2 is used to indicate the position of vehicle 2 in the image.

[0055] It can be seen that the current method of combining lidar and camera for positioning reduces the accuracy of positioning.

[0056] This application provides a target object positioning method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of target object positioning.

[0057] See Figure 3 , Figure 3 The flowchart illustrating the target object location method provided in this application embodiment includes S101 to S103, as follows:

[0058] S101. Data is acquired at the target synchronization time to obtain point cloud frames and image frames.

[0059] Specifically, during the target synchronization phase of the point cloud scanning process, the image sensor is controlled to acquire images, resulting in point cloud frames and image frames. The point cloud frame contains the initial position and initial azimuth angle of the scanned target object. The target synchronization time may include a preset time point for synchronizing radar data and image data.

[0060] This application embodiment can be applied to vehicles or equipment equipped with radar and image sensors. For example, this application embodiment can be applied to data acquisition equipment such as truth-checking vehicles or data acquisition vehicles, or to crowdsourcing vehicles, etc., depending on the specific circumstances; this application embodiment is not limited in its application. The radar is used for point cloud scanning. For example, the radar may include a lidar. For each scan, at the start of the scan, a rotational scan begins from the initial scanning angle (e.g., 0 degrees), completing a 360-degree scanning angle range to obtain a point cloud frame. Generating a point cloud frame through radar scanning takes a certain amount of time, such as 100ms. The image acquisition time of the image sensor, such as the exposure time, is on the order of milliseconds. To synchronize image data and point cloud data, within one scan cycle of the radar, the image sensor is controlled to simultaneously acquire images at the target synchronization moment to obtain an image frame; a point cloud frame is obtained when the point cloud scan is completed. During the scanning process, the point cloud frame will scan objects around the vehicle, such as moving objects like other vehicles. The target object, as well as its position and azimuth in the point cloud coordinate system, will be determined from the point cloud frame and used as the initial position and initial azimuth.

[0061] It should be noted that in some embodiments, at least one image sensor is deployed on the vehicle. For example, the vehicle may be equipped with six cameras, which are located at different positions on the vehicle. During the target synchronization moment in the point cloud scanning process, the six cameras are controlled to simultaneously acquire images.

[0062] In some embodiments, the image acquisition frequency (the number of image frames acquired per unit time) of the image sensor is typically higher than the radar scanning frequency. For example, the image acquisition frequency can be 30 Hz, and the radar scanning frequency can be 10 Hz. At least one target synchronization moment is determined during the point cloud scanning process based on the radar scanning frequency and the image acquisition frequency of the image sensor. Here, the radar scanning frequency is used to characterize the number of scans the radar performs per unit time; for example, a radar scanning frequency of 10 Hz represents 10 radar scans per second.

[0063] For example, the number of target synchronization moments can be determined based on the ratio of the image acquisition frequency to the radar scanning frequency, and this number of moments can be used as target synchronization moments during point cloud scanning. For example, taking an image acquisition frequency of 30 Hz and a LiDAR radar scanning frequency of 10 Hz as an example, the number of synchronization moments can be determined as follows: Figure 4A The three target synchronization times shown are t1, t2, and t3. Here, t1 is the time when the LiDAR scans to the left rear of the vehicle; for example, t1 = T - 83 (ms); T is the frame end time of the point cloud frame, i.e., the time when the point cloud frame is generated; t2 is the time when the LiDAR scans to the front of the vehicle; for example, t2 = T - 50 (ms); t3 is the time when the LiDAR scans to the right rear of the vehicle; for example, t3 = T - 17 (ms). Thus, by triggering at least one image sensor to expose at times t1', t2', and t3' respectively, image frames at times t1, t2, and t3 can be acquired, and the point cloud frame can be synchronized or aligned with the image frames acquired at times t1, t2, and t3 respectively. Alternatively, at least one image sensor can be triggered to expose at only one of these times, such as t2', to acquire the image frame at time t2, and the point cloud frame can be aligned with the image frame acquired at time t2. The target synchronization time represents the point in time at which the point cloud frame is synchronized with the image frame. In this way, the point cloud frame obtained by radar scanning is synchronized with the image frame acquired at each target synchronization moment in at least one target synchronization moment, thereby achieving at least one synchronization between radar data and image data.

[0064] In some embodiments, the raw point cloud data after point cloud scanning is completed is acquired; the raw point cloud data is subjected to distortion correction at the target synchronization time to determine the point cloud frame.

[0065] Here, since the points scanned by the scanning LiDAR do not share the same timestamp, it is necessary to align the timestamps of the points in the point cloud to the same point in time. This facilitates the synchronization of the point cloud and the image. Figure 4A For example, for different target synchronization times, the original point cloud data of radar scanning can be distorted in three directions: directly in front, left rear, and right rear to determine the corresponding point cloud frame.

[0066] In some embodiments, the method further includes: identifying objects within a target distance threshold as target objects.

[0067] In other words, in some embodiments, the target object localization method of this application can be applied only to objects within a target distance threshold in the point cloud frame. The target distance threshold includes a preset distance threshold. For example, the distance between a scanned object in the point cloud frame and the scan center point can be determined. If this distance is less than the target distance threshold, the object is identified as the target object. For example, the target distance threshold can be 60m, meaning the target object localization method of this application is applied only to obstacles within a 60m range. This is because obstacles within a 60-meter range have visible errors, especially when the obstacle is on the left or right side of the vehicle. Errors from obstacles beyond 60 meters are negligible. Error compensation is performed using the target object localization method of this application once the obstacle is within 60 meters again. The specific method is selected based on actual conditions, and this application does not limit the choice. This reduces the system's computational burden and resource consumption, improving system operating efficiency.

[0068] S102. Based on the scanning angle corresponding to the target synchronization time and the initial position of the target object, determine the compensation time coefficient corresponding to the target object.

[0069] In this embodiment, based on the target synchronization time, the duration between the target synchronization time and the scan start time corresponding to the point cloud frame can be determined. The scan angle corresponding to the target synchronization time represents the angle traversed from the scan start time to the target synchronization time, such as the angle rotated by the radar scan line. The initial position corresponding to the target object can be the coordinates of the target object in the point cloud frame. Since the radar scans from the scan start time, it takes a certain amount of time to scan to the scan angle corresponding to the target synchronization time. If the target object moves between the time it is scanned and the target synchronization time, a positioning error will occur. Therefore, in this embodiment, based on the target synchronization time, the scan angle corresponding to the target synchronization time, and the initial position corresponding to the target object, the duration between the time the radar scans the target object and the target synchronization time can be determined as a compensation time coefficient. Furthermore, positioning compensation for the target object can be performed based on the compensation time coefficient, improving positioning accuracy.

[0070] In some embodiments, S102 can be implemented by performing the following process, including:

[0071] Determine the time difference between the target synchronization time and the scan start time corresponding to the point cloud frame; determine the compensation angle based on the line connecting the initial position of the target object and the scan center point, and the angle between the scan lines at the target synchronization time; determine the compensation time coefficient based on the ratio of the compensation angle to the scan angle and the time difference.

[0072] The time difference between the target synchronization time and the scan start time corresponding to the point cloud frame represents the duration from the scan start time to the target synchronization time. For example, based on... Figure 4A When the end time of the point cloud frame is T, the start time of the corresponding scan of the point cloud frame can be T-100 (ms). Then, when the target synchronization time is t1, the time difference is (T-83)-(T-100)=17ms; when the target synchronization time is t2, the time difference is (T-50)-(T-100)=50ms; and when the target synchronization time is t3, the time difference is (T-17)-(T-100)=83ms.

[0073] The scan line at target synchronization refers to the scan line at which the radar reaches target synchronization during the scanning process. This can be understood as the scan line at target synchronization having rotated past the scanning angle corresponding to the target synchronization moment.

[0074] For example, based on Figure 4A The scan line at the target synchronization time can be represented as rays emanating from the scan center point (center of the circle) at times t1, t2, and t3, such as... Figure 4B The t1, t2, and t3 scan lines are shown in the diagram. The angle between the line connecting the initial position of the target object and the scan center point, and the scan line at the target synchronization time, is the compensation angle. The compensation angle represents the angle rotated by the radar scan line from the moment the target object is scanned to the moment of target synchronization. For example, using... Figure 4B Taking the initial position of the target object shown in the figure as an example, when the target synchronization time is t1, the scanning angle is α. t1 The compensation angle is θ t1 When the target synchronization time is t2, the scanning angle is α. t2 The compensation angle is θ t2 When the target synchronization time is t3, the scanning angle is α. t3 The compensation angle is θ t3Based on the ratio between the compensation angle and the scanning angle, and combined with the time required for the scanning angle corresponding to the target synchronization time (that is, the time difference between the target synchronization time and the scanning start time corresponding to the point cloud frame), the time between the moment the target object is scanned and the target synchronization time can be determined as the compensation time coefficient.

[0075] For example, the process of determining the compensation time coefficient based on the ratio of the compensation angle to the scanning angle and the time difference can be represented by formula (1) as follows:

[0076] time_diff=θ / α*T Formula (1)

[0077] In formula (1), θ represents the compensation angle, α represents the scanning angle corresponding to the target synchronization moment, T represents the time difference between the target synchronization moment and the scanning start moment corresponding to the point cloud frame, and time_diff is the compensation time coefficient.

[0078] For example, taking the target synchronization time as the moment when the radar scans directly in front of the point cloud, i.e., the time t2 mentioned above, as an example, the process of determining the compensation time coefficient based on the ratio of the compensation angle to the scanning angle and the time difference can be expressed by formula (2) as follows:

[0079] time_diff=θ / 180* 50ms formula (2)

[0080] In formula (2), θ is the compensation angle, 180 represents the scanning angle of 180 degrees at time t2; 50ms represents the time difference between time t2 and the start time of the scanning of the point cloud frame, and time_diff is the compensation time coefficient.

[0081] In some embodiments, the initial compensation time coefficient is determined based on the product of the ratio and the time difference; the compensation time coefficient is determined based on the initial compensation time coefficient and the target fluctuation value.

[0082] In other words, in some embodiments, the product of the ratio of the compensation angle to the scanning angle and the time difference can be used as the initial compensation time coefficient. The initial compensation time coefficient is then fine-tuned based on the target fluctuation value to determine the final compensation time coefficient. The target fluctuation value represents the amount of fluctuation in the preset compensation time coefficient. The target fluctuation value can be determined based on empirical values; for example, the target fluctuation value can be 15ms. The specific value can be selected based on actual conditions, and this application does not impose any limitations.

[0083] S103. Based on the compensation time coefficient, update the initial position and initial azimuth to determine the target position and target azimuth corresponding to the target object.

[0084] In this embodiment, based on the compensation time coefficient determined by the above steps, the initial position and initial azimuth can be updated based on the compensation time coefficient and the movement speed of the target moving object. The updated initial position is used as the target position, and the updated initial azimuth is used as the target azimuth. The movement speed of the target object can include linear velocity and angular velocity. The linear velocity is related to the real-time position of the target object, and the angular velocity is related to the real-time azimuth (or heading angle) of the target object. By compensating for the time coefficient and linear velocity, the initial position can be updated to obtain the actual target position of the target object. By compensating for the time coefficient and angular velocity, the initial azimuth can be updated to obtain the actual target azimuth of the target object.

[0085] In some embodiments, the process of S103 can be implemented through the following process:

[0086] Obtain the correspondence between the first coordinate system and the second coordinate system; the second coordinate system corresponds to the image frame; based on the correspondence, transform the initial position and initial azimuth to the second coordinate system to obtain the position to be compensated and the azimuth to be compensated; determine the linear velocity and angular velocity of the target object; based on the compensation time coefficient and linear velocity, update the position to be compensated and determine the updated position; based on the compensation time coefficient and angular velocity, update the azimuth to be compensated and determine the updated azimuth; transform the updated position and updated azimuth to the first coordinate system to determine the target position and target azimuth.

[0087] In some embodiments, the correspondence between the first coordinate system and the second coordinate system can be determined based on vehicle mileage information. This vehicle mileage information includes the linear velocity and angular velocity of the vehicle itself, calculated by a vehicle odometer. The vehicle odometer can be deployed in the vehicle provided in this embodiment. In some embodiments, after the vehicle starts, the vehicle odometer acquires sensor data from the vehicle itself to calculate the vehicle's linear velocity and angular velocity in real time, and then calculates the vehicle's real-time position based on these values. By mapping the real-time position of the vehicle calculated by the vehicle odometer to point cloud frames obtained from radar scanning and image frames acquired by image sensors, the correspondence between the coordinate system of the point cloud frames (referred to as the first coordinate system) and the coordinate system of the image frames (referred to as the second coordinate system) can be obtained. For example, the first coordinate system can be a local coordinate system, and the second coordinate system can be a global coordinate system.

[0088] In some embodiments, the correspondence between the first coordinate system and the second coordinate system can be predetermined through the above method or a pre-determined debugging process, and the correspondence can be stored in the form of calibration parameters. The correspondence between the first coordinate system and the second coordinate system can then be determined by obtaining the calibration parameters. The specific choice depends on the actual situation, and this application does not limit the specific choice.

[0089] In some embodiments, a target following algorithm can be applied to image frames in a continuous time series acquired by an image sensor to obtain the linear velocity and angular velocity of the target object. Since the linear velocity and angular velocity of the target object are calculated based on the image frames, it is necessary to transform the initial position and initial azimuth of the target object in the first coordinate system to the second coordinate system for subsequent update operations. Here, the position to be compensated is the position corresponding to the initial position transformed to the second coordinate system, and the azimuth to be compensated is the azimuth corresponding to the initial azimuth transformed to the second coordinate system. Furthermore, the position compensation amount can be determined based on the product of the compensation time coefficient and the linear velocity, and the position to be compensated is updated using the position compensation amount to determine the updated position (second coordinate system); the angular velocity compensation amount can be determined based on the product of the compensation time coefficient and the angular velocity, and the azimuth to be compensated is updated using the angular velocity compensation amount to determine the updated azimuth (second coordinate system). Finally, the updated position and updated azimuth are transformed back to the first coordinate system to determine the target position and target azimuth. In other words, the target position is the position corresponding to the first coordinate system after the update position is transformed; the target azimuth is the azimuth corresponding to the first coordinate system after the update azimuth is transformed.

[0090] In some embodiments, the above-mentioned updating of the position to be compensated based on the compensation time coefficient and linear velocity, and determining the update position, includes:

[0091] The position compensation amount is determined by multiplying the compensation time coefficient by the linear velocity. If the target object is determined to be within the scanning angle based on the rotation direction of the point cloud, the position to be compensated is increased by the position compensation amount to determine the updated position. If the target object is determined to be outside the scanning angle based on the rotation direction of the point cloud, the position to be compensated is decreased by the position compensation amount to determine the updated position.

[0092] The rotation direction of the point cloud represents the rotation direction of the radar scan; for example, it represents whether the radar scans clockwise or counterclockwise.

[0093] In some embodiments, if the target object is determined to be within the scanning angle corresponding to the target synchronization moment based on the point cloud rotation direction, it indicates that the target object was scanned first, and image acquisition was performed after the target synchronization moment. Therefore, the target object's position in the image frame may precede the position to be compensated determined based on the point cloud frame. Thus, a position compensation amount needs to be added to the position to be compensated to determine the updated position.

[0094] In some embodiments, the updated position may include the two-dimensional coordinates of the target object in a second coordinate system. The updated position can be determined by formulas (3) and (4), as follows:

[0095] pos_x'=pos_x+Vx*time_diff Formula (3)

[0096] pos_y'=pos_y+Vy*time_diff formula (4)

[0097] Where pos_x is the x-axis coordinate of the target object in the second coordinate system; Vx is the linear velocity of the target object on the x-axis in the second coordinate system; pos_y is the y-axis coordinate of the target object in the second coordinate system; Vy is the linear velocity of the target object on the y-axis in the second coordinate system; pos_x' and pos_y' are the coordinates of the target object on the x-axis and y-axis in the second coordinate system after compensation according to the position compensation amount (Vx*time_diff), which are the updated positions of the target object.

[0098] In some embodiments, if the target object is determined to be outside the scanning angle corresponding to the target synchronization moment based on the point cloud rotation direction, it indicates that the target synchronization moment was reached before the target object was scanned. The target object's position in the image frame may lag behind the compensation position determined based on the point cloud frame. Therefore, a position compensation amount needs to be subtracted from the compensation position to determine the updated position.

[0099] For example, the target object can be an obstacle vehicle, and the target synchronization time can be... Figure 5 The T-50 (ms) time in the text is equivalent to Figure 4A At time t2, when the obstacle vehicle is located to the left rear of the radar, a compensation angle θ1 is formed with the scan line at the target synchronization time. Since the point cloud rotation direction is clockwise, when the obstacle vehicle is within the scan angle at the target synchronization time, the point cloud scan position (initial position) of the obstacle vehicle will lag behind the vehicle position of the obstacle vehicle acquired by the image sensor, and the point cloud frame generated based on the point cloud scan position of the obstacle vehicle will lag behind the image of the obstacle vehicle in the image frame. When the obstacle vehicle is located to the right front of the radar, a compensation angle θ2 is formed with the scan line at the target synchronization time. At this time, the obstacle vehicle is outside the scan angle at the target synchronization time, and the point cloud scan position (initial position) of the obstacle vehicle will lead the vehicle position of the obstacle vehicle acquired by the image sensor, and the point cloud frame generated based on the point cloud scan position of the obstacle vehicle will lead the image of the obstacle vehicle in the image frame. It can be seen that, using the above method of the embodiments of this application, it is possible to... Figure 5 The two types of positioning errors are compensated to improve positioning accuracy.

[0100] Similarly, such asFigure 6A As shown, the target synchronization time is T-83ms, which is equivalent to Figure 4A At time t1, when the target object is in Figure 6A When the target object is within angle θ, the point cloud box corresponding to the initial position of the target object lags behind the image of the target object in the image frame, and a position compensation amount needs to be added to the position to be compensated. When the target object is within... Figure 6A When the angle is outside θ, the point cloud box corresponding to the initial position of the target object leads the image of the target object in the image frame, and the position compensation amount needs to be subtracted from the position to be compensated. For example... Figure 6B As shown, the target synchronization time is T-17ms, which is equivalent to... Figure 4A At time t3, when the target object is in Figure 6A When the target object is within angle θ, the point cloud box corresponding to the initial position of the target object lags behind the image of the target object in the image frame, and a position compensation amount needs to be added to the position to be compensated. When the target object is within... Figure 6A When the point cloud box corresponding to the initial position of the target object is outside the θ angle, it leads the image of the target object in the image frame, and the position compensation amount needs to be subtracted from the position to be compensated.

[0101] In some embodiments, the above-mentioned updating of the azimuth angle to be compensated based on the compensation time coefficient and angular velocity, and determining the updated azimuth angle, includes:

[0102] The azimuth compensation amount is determined by multiplying the compensation time coefficient and the angular velocity. If the target object is determined to be within the scanning angle based on the rotation direction of the point cloud, the azimuth to be compensated is increased by the azimuth compensation amount to determine the updated azimuth. If the target object is determined to be outside the scanning angle based on the rotation direction of the point cloud, the azimuth to be compensated is decreased by the azimuth compensation amount to determine the updated azimuth.

[0103] Among them, determining the azimuth compensation amount and compensating the azimuth to be compensated based on the azimuth compensation amount, and determining the updated azimuth are similar to the above-mentioned position update process, and will not be repeated here.

[0104] In some embodiments, after determining the target position and target azimuth angle corresponding to the target object, the target object can be identified in the image frame based on the target position and target azimuth angle. That is, the target position and target azimuth angle determined by the point cloud frame can be used to determine the identification of the target object in the image frame.

[0105] In this embodiment, since the image frame is acquired at the target synchronization moment, the point cloud data scanned by the image frame and the point cloud frame at the target synchronization moment are synchronized or aligned. The target position and azimuth angle of the target object are obtained based on compensation at the target synchronization moment. Therefore, identifying the target object in the image frame according to the target position and azimuth angle ensures that the identifier of the target object in the image frame matches the image of the target object, improving positioning accuracy. For example, the identifier of the target object in the image frame may include the point cloud bounding box of the target object. The specific selection depends on the actual situation, and this embodiment does not limit the choice.

[0106] For example, Figures 7A-9A The image shows the difference between the projection box (dashed box) of the point cloud generated by the target object localization method without using the embodiments of this application and the image of the target object (image within the solid box) in the image frame; Figures 7B-9B The image shows the point cloud bounding box positioning effect after positioning compensation using the target object positioning method of this application. It can be seen that the method of this application can accurately synchronize point cloud data and image data, resulting in a closer fit between the point cloud bounding box and the image, thereby improving the positioning accuracy of the target object.

[0107] It should be noted that, for vehicles equipped with at least one image sensor with at least one viewpoint, the image frame here can include an image frame acquired from any one of the at least one viewpoints, or an image frame obtained by image fusion of image frames acquired from at least one viewpoint. The specific choice depends on the actual situation, and this application embodiment does not limit it.

[0108] It should be noted that when multiple target synchronization times are determined in a single frame of point cloud, each target synchronization time is processed using the above process in S101-S104, and multiple target synchronization times are synchronized multiple times.

[0109] Understandably, by controlling the image sensor to acquire images during the target synchronization moment in the point cloud scanning process, point cloud frames and image frames are obtained. Based on the target synchronization moment, the corresponding scanning angle, and the initial position of the scanned target object, a compensation time coefficient is determined. Based on the compensation time coefficient, the initial position and initial azimuth of the target object are updated to determine the target position and target azimuth. Thus, the point cloud frames and image frames are synchronized, and the initial position and initial azimuth of the target object in the point cloud frame are compensated to obtain the target position and target azimuth of the target object in the image frame. Furthermore, the target object is marked in the image frame based on its position and target azimuth, ensuring accurate alignment between the marked area (e.g., the point cloud frame) and the image of the target object in the image frame. This achieves data alignment between the point cloud and the image sensor, improving the accuracy of target object localization.

[0110] This application provides a target object positioning device 1, such as... Figure 10 As shown, it includes:

[0111] Control module 11 is used to acquire data at the target synchronization time to obtain point cloud frames and image frames; the point cloud frame contains the initial position and initial azimuth angle corresponding to the target object;

[0112] The determining module 12 is used to determine the compensation time coefficient corresponding to the target object based on the scanning angle corresponding to the target synchronization time and the initial position;

[0113] The update module 13 is used to update the initial position and the initial azimuth angle based on the compensation time coefficient, and to determine the target position and target azimuth angle corresponding to the target object.

[0114] In some embodiments, the determining module 12 is further configured to determine the time difference between the target synchronization time and the scanning start time corresponding to the point cloud frame; determine the compensation angle based on the line connecting the initial position and the scanning center point, and the angle between the scanning lines at the target synchronization time; and determine the compensation time coefficient based on the ratio of the compensation angle to the scanning angle and the time difference.

[0115] In some embodiments, the initial position and the initial azimuth are in a first coordinate system, which corresponds to the point cloud frame; the update module 13 is further configured to obtain the correspondence between the first coordinate system and a second coordinate system; the second coordinate system corresponds to the image frame; according to the correspondence, the initial position and the initial azimuth are transformed to the second coordinate system to obtain the position to be compensated and the azimuth to be compensated; the linear velocity and angular velocity of the target object are determined; based on the compensation time coefficient and the linear velocity, the position to be compensated is updated to determine the updated position; based on the compensation time coefficient and the angular velocity, the azimuth to be compensated is updated to determine the updated azimuth; the updated position and the updated azimuth are transformed to the first coordinate system to determine the target position and the target azimuth.

[0116] In some embodiments, the updating module 13 is further configured to determine a position compensation amount based on the product of the compensation time coefficient and the linear velocity; when the target object is determined to be within the scanning angle based on the point cloud rotation direction, the position to be compensated is increased by the position compensation amount to determine the updated position; when the target object is determined to be outside the scanning angle based on the point cloud rotation direction, the position to be compensated is decreased by the position compensation amount to determine the updated position.

[0117] In some embodiments, the updating module 13 is further configured to determine the azimuth compensation amount based on the product of the compensation time coefficient and the angular velocity; when the target object is determined to be within the scanning angle based on the point cloud rotation direction, the azimuth to be compensated is increased by the azimuth compensation amount to determine the updated azimuth; when the target object is determined to be outside the scanning angle based on the point cloud rotation direction, the azimuth to be compensated is decreased by the azimuth compensation amount to determine the updated azimuth.

[0118] In some embodiments, the determining module 12 is further configured to determine an initial compensation time coefficient based on the product of the ratio and the time difference; and to determine the compensation time coefficient based on the initial compensation time coefficient and the target fluctuation value.

[0119] In some embodiments, the target object positioning device 1 further includes a distortion correction module; the distortion correction module is used to acquire the original point cloud data after the point cloud scanning is completed; and to perform distortion correction processing on the original point cloud data at the target synchronization time to determine the point cloud frame.

[0120] In some embodiments, the determining module 12 is further configured to determine at least one target synchronization moment during the point cloud scanning process based on the radar scanning frequency and the image acquisition frequency of the image sensor.

[0121] In some embodiments, the determining module 12 is further configured to determine objects within a target distance threshold as the target object.

[0122] It should be noted that the description of the above device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the device embodiments of the present invention, please refer to the description of the method embodiments of the present invention for understanding.

[0123] This application also provides an electronic device. Figure 11 This is a schematic diagram of an optional structure of the electronic device 3 provided in an embodiment of this application. For example... Figure 11 As shown, the electronic device 3 includes a memory 32 and a processor 33. The memory 32 and the processor 33 are connected via a communication bus 34. The memory 32 is used to store executable instructions. The processor 33 is used to execute the executable instructions stored in the memory 32 to implement the target object positioning method provided in this application embodiment.

[0124] This application also provides a vehicle, which includes an image sensor, a radar for point cloud scanning, and the electronic equipment described in this application.

[0125] This application provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by the processor, the processor will execute the target object positioning method provided in this application.

[0126] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0127] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0128] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts within a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0129] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0130] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0131] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0132] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0133] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method of locating a target object, characterized by, The method comprises: acquiring a point cloud frame and an image frame at a target synchronization time; the point cloud frame contains an initial position and an initial azimuth angle of a target object; determining a compensation time coefficient of the target object based on a scanning angle corresponding to the target synchronization time and the initial position; updating the initial position and the initial azimuth angle based on the compensation time coefficient to determine a target position and a target azimuth angle of the target object.

2. The method of claim 1, wherein, The method comprises: determining a time difference between the target synchronization time and a scanning start time corresponding to the point cloud frame; determining a compensation angle according to a line connecting the initial position and a scanning center point and an included angle between scanning lines at the target synchronization time; determining the compensation time coefficient based on a ratio of the compensation angle to the scanning angle and the time difference.

3. The method according to claim 1 or 2, characterized in that, The initial position and the initial azimuth angle are in a first coordinate system corresponding to the point cloud frame. The method comprises: obtaining a corresponding relationship between the first coordinate system and a second coordinate system corresponding to the image frame; converting the initial position and the initial azimuth angle to the second coordinate system to obtain a position to be compensated and an azimuth angle to be compensated according to the corresponding relationship; determining a linear velocity and an angular velocity of the target object; updating the position to be compensated based on the compensation time coefficient and the linear velocity to determine an updated position; updating the azimuth angle to be compensated based on the compensation time coefficient and the angular velocity to determine an updated azimuth angle; converting the updated position and the updated azimuth angle to the first coordinate system to determine the target position and the target azimuth angle.

4. The method of claim 3, wherein, The method comprises: determining a position compensation amount according to a product of the compensation time coefficient and the linear velocity; in a case where it is determined according to a point cloud rotation direction that the target object is within the scanning angle, increasing the position to be compensated by the position compensation amount to determine the updated position; in a case where it is determined according to a point cloud rotation direction that the target object is outside the scanning angle, decreasing the position to be compensated by the position compensation amount to determine the updated position.

5. The method of claim 3, wherein, The method comprises: determining an azimuth angle compensation amount according to a product of the compensation time coefficient and the angular velocity; in a case where it is determined according to a point cloud rotation direction that the target object is within the scanning angle, increasing the azimuth angle to be compensated by the azimuth angle compensation amount to determine the updated azimuth angle; In a case where it is determined that the target object is outside the scanning angle according to a rotation direction of the point cloud, the azimuth angle to be compensated is reduced by the azimuth angle compensation amount, and the updated azimuth angle is determined.

6. The method of claim 2, wherein, The determining the compensation time coefficient according to the ratio of the compensation angle to the scanning angle and the time difference includes: determining an initial compensation time coefficient according to a product of the ratio and the time difference; determining the compensation time coefficient based on the initial compensation time coefficient and a target fluctuation value.

7. The method of claim 1 or 2, wherein, The method further includes: acquiring original point cloud data of a point cloud scanning completion; performing de-distortion processing on the original point cloud data at the target synchronization moment to determine the point cloud frame.

8. The method of claim 1 or 2, wherein, The method further includes: determining at least one target synchronization moment in a point cloud scanning process according to a radar scanning frequency and an image acquisition frequency of the image sensor.

9. The method of claim 1 or 2, wherein, The method further includes: determining an object within a target distance threshold as the target object.

10. A target object positioning apparatus characterized by comprising: The apparatus includes: a control module configured to perform data acquisition at a target synchronization moment to obtain a point cloud frame and an image frame; the point cloud frame contains an initial position and an initial azimuth angle corresponding to a target object; a determination module configured to determine a compensation time coefficient corresponding to the target object based on a scanning angle corresponding to the target synchronization moment and the initial position; an update module configured to update the initial position and the initial azimuth angle based on the compensation time coefficient to determine a target position and a target azimuth angle corresponding to the target object.

11. An electronic device, comprising: The electronic device includes: a memory configured to store computer executable instructions or computer programs; a processor configured to execute the computer executable instructions or computer programs stored in the memory to implement the method in any one of claims 1 to 9.

12. A vehicle characterized by comprising: The vehicle includes an image sensor, a radar for point cloud scanning, and the electronic device in claim 11.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, wherein the computer-executable instructions or the computer program comprise the steps of: The computer executable instructions or computer programs are executed by the processor to implement the method in any one of claims 1 to 9. ​ 14. A computer program product comprising computer-executable instructions or a computer program, characterized in that, The computer executable instructions or computer programs are executed by the processor to implement the method in any one of claims 1 to 9. The computer executable instructions or computer programs are executed by the processor to implement the method in any one of claims 1 to 9.

Images