Wheel ground position determination method and device, electronic equipment and storage medium

Abstract

The embodiment of the disclosure discloses a wheel grounding position determination method and device, electronic equipment and storage medium, wherein the method comprises: determining the bounding box information and semantic segmentation information of a target vehicle in an image pixel coordinate system based on a to-be-processed image; determining the target observation type of the target vehicle based on the bounding box information; and determining the wheel grounding position of the target vehicle based on the bounding box information and the semantic segmentation information by using a determination method corresponding to the target observation type, wherein the wheel grounding position is a position in a vehicle coordinate system. The embodiment of the disclosure takes into account the coarse-grained observation at the bounding box level and the more accurate observation at the pixel level of the semantic segmentation information, thereby improving the accuracy and robustness of the determined wheel grounding position.

Description

Technical Field

[0001] This disclosure relates to computer vision technology, and in particular to a method, apparatus, electronic device, and storage medium for determining the ground contact position of a wheel. Background Technology

[0002] In the field of autonomous driving, perceiving information about surrounding target vehicles based on environmental images is crucial for downstream planning and control. For example, determining the wheel contact points (also known as wheel contact positions) of a target vehicle in the vehicle coordinate system based on environmental images is essential for determining the vehicle's position and yaw angle. This provides the necessary perception information for systems such as full-speed adaptive cruise control, active safety systems, and navigation assistance control systems. In related technologies, key points within the wheel frame are typically used as wheel contact points in the image pixel coordinate system, and the wheel contact positions in the vehicle coordinate system are then determined based on these points. However, this method is susceptible to errors in the wheel frame or missing key points within the wheel frame, leading to inaccurate determinations of the wheel contact positions. Summary of the Invention

[0003] The embodiments of this disclosure provide a method, apparatus, electronic device, and storage medium for determining the wheel ground contact position of a target vehicle in an image, thereby improving the accuracy of the wheel ground contact position.

[0004] According to one aspect of the present disclosure, a method for determining the wheel contact point position is provided, comprising: determining detection box information and semantic segmentation information of a target vehicle in an image pixel coordinate system based on an image to be processed; determining the target observation type of the target vehicle based on the detection box information; and determining the wheel contact point position of the target vehicle by adopting a determination method corresponding to the target observation type based on the detection box information and the semantic segmentation information, wherein the wheel contact point position is a position in a vehicle coordinate system.

[0005] According to another aspect of the present disclosure, a device for determining the wheel contact position is provided, comprising: a first processing module, configured to determine detection box information and semantic segmentation information of a target vehicle in an image pixel coordinate system based on an image to be processed; a second processing module, configured to determine the target observation type of the target vehicle based on the detection box information; and a third processing module, configured to determine the wheel contact point position of the target vehicle based on the detection box information and the semantic segmentation information, using a determination method corresponding to the target observation type, wherein the wheel contact point position is a position in a vehicle coordinate system.

[0006] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the wheel contact position determination method described in any of the above embodiments of the present disclosure.

[0007] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the wheel grounding position determination method described in any of the above embodiments of the present disclosure.

[0008] Based on the wheel contact point determination method, apparatus, electronic device, and storage medium provided in the above embodiments of this disclosure, the detection box information and semantic segmentation information of the target vehicle in the image pixel coordinate system can be determined based on the image to be processed. The observation type of the target vehicle can be determined based on the detection box information. Then, the detection box information and semantic segmentation information can be combined to adopt different determination methods for different observation types to determine the wheel contact point position of the target vehicle in the vehicle coordinate system. This takes into account both the coarse-grained observation at the detection box level and the more precise observation at the pixel level of semantic segmentation information, improving the accuracy and robustness of the determined wheel contact point. Moreover, it can realize the determination of the wheel contact point under various observation conditions, improving the versatility of the wheel contact point determination method.

[0009] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0010] Figure 1 This is an exemplary application scenario of the method for determining the wheel contact position provided in this disclosure;

[0011] Figure 2 This is a flowchart illustrating a method for determining the wheel contact position according to an exemplary embodiment of this disclosure;

[0012] Figure 3 This is a flowchart illustrating a method for determining the wheel contact position according to another exemplary embodiment of this disclosure;

[0013] Figure 4 This is a schematic diagram of a vehicle observation type provided in an exemplary embodiment of this disclosure;

[0014] Figure 5 This is a flowchart illustrating a method for determining the wheel contact position provided in yet another exemplary embodiment of this disclosure;

[0015] Figure 6 This is a flowchart illustrating step 2033 provided in an exemplary embodiment of this disclosure;

[0016] Figure 7 This is a schematic diagram of a first type of right truncation provided in an exemplary embodiment of this disclosure;

[0017] Figure 8 This is a schematic diagram illustrating the principle of vanishing point determination provided by an exemplary embodiment of this disclosure;

[0018] Figure 9 This is a schematic diagram illustrating an abnormal situation in finding the second grounding point, provided in an exemplary embodiment of this disclosure.

[0019] Figure 10 This is a schematic diagram of four sets of first wheel frame segmentation points provided in an exemplary embodiment of this disclosure;

[0020] Figure 11 This is a schematic diagram of a plurality of front wheels and a plurality of rear wheels provided in an exemplary embodiment of the present disclosure;

[0021] Figure 12 This is a schematic diagram illustrating the principle of determining the first position reference point provided in an exemplary embodiment of this disclosure;

[0022] Figure 13 This is a flowchart illustrating a method for determining the wheel contact position according to yet another exemplary embodiment of this disclosure;

[0023] Figure 14 This is a schematic diagram of the structure of a wheel contact position determination device provided in an exemplary embodiment of this disclosure;

[0024] Figure 15 This is a schematic diagram of the structure of a wheel contact point determination device provided in another exemplary embodiment of this disclosure;

[0025] Figure 16 This is a structural diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0026] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0027] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0028] This disclosure outlines

[0029] In developing this disclosure, the inventors discovered that in the field of autonomous driving, perceiving information about surrounding target vehicles based on environmental images is crucial for downstream planning and control. For example, determining the wheel contact points of a target vehicle in the vehicle coordinate system based on environmental images, and then determining the target vehicle's position and yaw angle (or heading angle) based on these wheel contact points, provides the necessary perception information for full-speed adaptive cruise control systems, active safety systems, and navigation assistance control systems. In related technologies, key points within the wheel frame are typically used as wheel contact points in the image pixel coordinate system, and then the wheel contact points in the vehicle coordinate system are determined based on these wheel contact points. However, this method of determining wheel contact point positions is susceptible to errors in the wheel frame or missing key points within the wheel frame, leading to inaccurate determinations of the wheel contact points.

[0030] Exemplary Overview

[0031] Figure 1 This is an exemplary application scenario of the method for determining the wheel contact position provided in this disclosure. For example... Figure 1 As shown, during vehicle operation, the vehicle can observe its surrounding environment using cameras installed on the vehicle, acquiring images of the surrounding environment as images to be processed. The cameras installed on the vehicle can include cameras with a front-view perspective, cameras with a left front-view perspective, cameras with a right front-view perspective, cameras with a left rear-view perspective, and cameras with a right rear-view perspective. In practical applications, the number of cameras installed on the vehicle and their viewing angles are not limited. Using the wheel contact point determination method disclosed herein, the detection box information and semantic segmentation information of the target vehicle in the image pixel coordinate system can be determined based on the image to be processed; based on the detection box information, the target observation type of the target vehicle can be determined; based on the detection box information and semantic segmentation information, the wheel contact point of the target vehicle can be determined using a determination method corresponding to the target observation type, where the wheel contact point is the position in the vehicle coordinate system. Furthermore, the position and heading angle of the target vehicle can be determined based on the wheel contact point, providing the necessary perception information for the vehicle's full-speed adaptive cruise control system, active safety system, and navigation assistance control system. The method disclosed herein combines coarse-grained observation at the detection box level with more precise observation at the pixel level of semantic segmentation information, which helps to improve the accuracy and robustness of determining the wheel ground contact position. Furthermore, it can determine the wheel ground contact position under various observation conditions, thereby improving the versatility of the method for determining the wheel ground contact position.

[0032] Exemplary methods

[0033] Figure 2 This is a flowchart illustrating a method for determining the wheel contact position according to an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, specifically, for example, on an in-vehicle computing platform. Figure 2 As shown, it includes the following steps:

[0034] Step 201: Based on the image to be processed, determine the detection box information and semantic segmentation information of the target vehicle in the image pixel coordinate system.

[0035] The images to be processed can be acquired using cameras mounted on the vehicle. For example... Figure 1 The images shown include cameras viewing the vehicle from the front, left front, right front, left rear, and right rear. The target vehicles are those observed around the vehicle in the image to be processed. The image pixel coordinate system refers to the pixel coordinate system corresponding to the image to be processed. Detection box information can be obtained by performing object detection on the image to be processed. Semantic segmentation information can be obtained by performing semantic segmentation on the image to be processed.

[0036] In some optional embodiments, vehicle detection can be performed on the image to be processed based on any implementable object detection algorithm or model to obtain the detection box information of the target vehicle in the image pixel coordinate system. The object detection model can be, for example, a series of models based on convolutional neural networks. Examples include object detection models based on YOLO, and object detection models based on RCNN (Region Convolutional Neural Network) and its series.

[0037] In some optional embodiments, semantic segmentation of the image to be processed can be performed based on any implementable semantic segmentation model to obtain semantic segmentation information of the target vehicle. Examples of semantic segmentation models include U-Net-based models, Fully Convolutional Networks-based models, and DeepLab and its related models.

[0038] In some optional embodiments, the detection box information may include at least one of the following: full vehicle frame information, front and rear frame information, side line information, wheel frame information, etc. Full vehicle frame information determines the full vehicle frame of the target vehicle in the image to be processed. Side line information determines the side line of the target vehicle in the image to be processed. Front and rear frame information may include either head frame information or tail frame information, or it may include information that does not include front and rear frames (or no front and rear frames were detected). Head frame information determines the detection box of the head of the target vehicle in the image to be processed. Tail frame information determines the detection box of the tail of the target vehicle in the image to be processed. Correspondingly, semantic segmentation information may include semantic classification information indicating whether each pixel in the image to be processed belongs to the target vehicle. For example, for any pixel, the semantic classification information includes probability information about whether the pixel belongs to the target vehicle, or a classification value indicating whether the pixel belongs to the target vehicle, such as 1 indicating that the pixel belongs to the target vehicle and 0 indicating that the pixel does not belong to the target vehicle. No specific limitation is made. Based on the semantic segmentation information, the semantic segmentation boundary point set of the target vehicle can be determined.

[0039] Step 202: Based on the detection frame information, determine the target observation type of the target vehicle.

[0040] The observation type can include at least two types. For example, it can include a first type, a second type, and a third type. The observation type can be classified according to the specific situation of the observed vehicle. For example, the target observation type can be determined based on the front and rear frame information and the full vehicle frame information included in the detection frame information. The target observation type can also be determined based on the side line information included in the detection frame information. Alternatively, the target observation type can be determined by combining the full vehicle frame information, the front and rear frame information, and the side line information. Specific limitations are not specified.

[0041] For example, if the observed detection box information includes the full vehicle frame information and the front and rear frame information of the target vehicle (and may also include other information, such as side line information, wheel frame information, etc.), the target observation type can be determined based on the full vehicle frame information and the front and rear frame information. For instance, if the front and rear frame information determines that the front and rear frame is located on the first side of the full vehicle frame (e.g., the left side), the target observation type of the target vehicle can be determined to be type one. If the front and rear frame is located on the second side of the full vehicle frame (e.g., the right side), the target observation type of the target vehicle can be determined to be type two. If the detection box information includes the full vehicle frame information and side line information (and may also include other information, such as wheel frame information), but does not include the front and rear frame information, the target observation type of the target vehicle can be determined based on the full vehicle frame information and the side line information. For instance, in the case where the front and rear frame information is not included, but the full vehicle frame information and the side line information are included, the target observation type of the target vehicle can be determined to be type three.

[0042] Step 203: Based on the detection box information and semantic segmentation information, determine the wheel ground contact position of the target vehicle using a determination method corresponding to the target observation type.

[0043] The wheel contact point location is the position in the vehicle coordinate system. The vehicle coordinate system refers to the vehicle's coordinate system where the camera is located when acquiring the image to be processed.

[0044] In some optional embodiments, a method for determining the wheel contact point corresponding to different observation types can be pre-set. Then, after obtaining the target vehicle's detection frame information and semantic segmentation information, the wheel contact point of the target vehicle can be determined using a method corresponding to the target observation type. The determination method can be based on the correlation between the observable full vehicle frame, front and rear frames, wheel frames, side lines, etc., under different observation types, and the semantic segmentation boundary points.

[0045] For example, if the entire target vehicle can be observed in the image to be processed, and if the detection box information includes the entire vehicle frame information, wheel frame information, and side line information, but does not include the front and rear frame information, the wheel contact points of the target vehicle can be determined by combining the relationship between the semantic segmentation boundary point set of the wheels and the side lines. Different observation types and different detection box information can correspond to different determination methods.

[0046] In some optional embodiments, the image to be processed can be a distorted image, i.e., an image with distortion. First, the grounding points of the target vehicle's wheels in the image pixel coordinate system are determined in the distorted image domain, and then transformed to the vehicle coordinate system of the vehicle corresponding to the image to be processed to obtain the grounding positions of the target vehicle's wheels.

[0047] The wheel contact point determination method provided in this embodiment can determine the detection box information and semantic segmentation information of the target vehicle in the image pixel coordinate system based on the image to be processed, and determine the observation type of the target vehicle based on the detection box information. Then, it can combine the detection box information and semantic segmentation information to adopt different determination methods for different observation types, thereby determining the wheel contact point of the target vehicle in the vehicle coordinate system. This method takes into account both the coarse-grained observation at the detection box level and the more precise observation at the pixel level of semantic segmentation information, improving the accuracy and robustness of the determined wheel contact point. Moreover, it can determine the wheel contact point under various observation conditions, thus improving the versatility of the wheel contact point determination method.

[0048] Figure 3 This is a flowchart illustrating a method for determining the wheel contact position according to another exemplary embodiment of this disclosure.

[0049] In some alternative embodiments, such as Figure 3As shown, step 202, based on the detection box information, determines the target observation type of the target vehicle, including:

[0050] Step 2021: In response to the detection frame information including full vehicle frame information, front and rear frame information and side line information, and the front and rear frame information being located on the first side of the full vehicle frame information, the target observation type is determined to be the first type.

[0051] The full vehicle bounding box information can include the position information of the detection box surrounding the target vehicle (whole or part of the target vehicle is located in the image to be processed). The front and rear bounding box information can include the position information of the detection box surrounding the front or rear of the target vehicle, or include information on any undetected front or rear bounding boxes. The side line information can include the pixel coordinates of the observed wheel contact points of the target vehicle in the image to be processed. In other words, the side line information is the wheel contact point information obtained based on the image to be processed. For example, two wheel contact points can define a straight line, which is the side line. Side line information obtained based on image detection may not be entirely accurate and is used to assist in determining more accurate wheel contact points.

[0052] In some optional embodiments, the detection frame information may also include other relevant detection information of the target vehicle, such as the wheel frame information of the target vehicle.

[0053] Step 2022: In response to the detection frame information including full vehicle frame information, front and rear frame information and side line information, and the front and rear frame information being located on the second side of the full vehicle frame information, the target observation type is determined to be the second type.

[0054] If the first side is the left side, then the second side is the right side. If the first side is the right side, then the second side is the left side.

[0055] Step 2023: In response to the detection frame information including full vehicle frame information, wheel frame information and side line information, but excluding front and rear frame information, determine the target observation type based on the side line information.

[0056] Here, wheel frame information refers to the positional information of the detection boxes surrounding the wheels observed in the image to be processed. Wheel frame information can include the positional information of the wheel frames of at least one vehicle. For example, for a small car, wheel frame information may include the positional information of the wheel frame of one wheel, or it may include the positional information of the wheel frames of two wheels. For large vehicles, wheel frame information can include the positional information of the wheel frames of more wheels. For cases where front and rear frame information is not included (e.g., the detection model misses the front and rear frames, or the target vehicle is facing the camera from the side, making the front and rear frames unobservable), the target observation type can be determined based on sideline information. Specifically, the slope of the sideline determined by the sideline information can be used to determine whether the target observation type belongs to type one, type two, or type three. If the absolute value of the sideline slope is within a certain range (e.g., [0, k1]), the target observation type can be determined to be type three. If the absolute value of the sideline slope exceeds this range, it may belong to type one, type two, or other types. For example, if the absolute value of the slope is within the range of [k1, k2], the target observation type can be determined to be type one or type two. Specifically, the sign of the sideline slope can be used to determine whether it is type one or type two. For large vehicles with truncated observations, the head and tail frame information may not be included, and the target observation type can be further determined based on the sideline information.

[0057] In some optional examples, Figure 4 This is a schematic diagram of a vehicle observation type provided in an exemplary embodiment of this disclosure. For example... Figure 4 As shown, the first type of detection frame information includes the entire vehicle frame, front and rear frames, and side lines, with the front and rear frames located on the left side of the entire vehicle frame. The second type of detection frame information includes the entire vehicle frame, front and rear frames, and side lines, with the front and rear frames located on the right side of the entire vehicle frame. The third type of detection frame information includes the entire vehicle frame, wheel frames, and side lines, but excludes the front and rear frames.

[0058] This embodiment determines the target observation type of the target vehicle based on the detection frame information, which helps to determine the determination method for determining the grounding position of the target vehicle's wheels, thereby obtaining the accurate grounding position of the target vehicle's wheels based on the determination method corresponding to the target observation type.

[0059] Figure 5 This is a flowchart illustrating a method for determining the wheel contact position according to another exemplary embodiment of the present disclosure.

[0060] In some optional embodiments, the detection frame information includes full vehicle frame information, front and rear frame information, and side line information; such as Figure 5 As shown, step 203, based on detection box information and semantic segmentation information, uses a determination method corresponding to the target observation type to determine the wheel contact position of the target vehicle, including:

[0061] Step 2031, in response to the target observation type being the first type or the second type, based on the side line information and the whole vehicle frame information, determine whether the target vehicle is observed to be truncated.

[0062] Among them, the observed truncation means that the complete target vehicle is not observed in the to-be-processed image, but only a part of the target vehicle is observed. For example, only the front half of the target vehicle is observed, and the rear half is not in the to-be-processed image; or only the rear half of the target vehicle is observed, and the front half is not in the to-be-processed image, etc. For example, for a large vehicle as the target vehicle, the shooting range of the camera of the host vehicle may only cover a part of the target vehicle, resulting in the target vehicle observed based on the to-be-processed image being observed to be truncated. Whether the target vehicle is observed to be truncated can be determined based on the positional relationship between the whole vehicle frame and the side lines in the to-be-processed image. For example, based on the distance between the boundary of the whole vehicle frame and the boundary of the to-be-processed image, the ratio of the width of the whole vehicle frame to the width of the to-be-processed image, the intersection situation between the side lines and the whole vehicle frame, etc., determine whether the target vehicle is observed to be truncated.

[0063] Exemplarily, if the distance d1 between the boundary line of the first side (such as the right side) of the whole vehicle frame of the target vehicle and the boundary line of the first side of the to-be-processed image is less than the distance threshold D (which can be called the first distance threshold), the ratio r of the width of the whole vehicle frame to the width of the to-be-processed image is greater than the ratio threshold, and the distance d2 between the boundary line of the second side of the whole vehicle frame and the boundary line of the second side of the to-be-processed image is greater than d1, it is determined that the target vehicle is observed to be truncated. Based on this, the observed truncation direction of the target vehicle can also be determined. For example, when the first side is the right side, it is determined that the target vehicle is truncated on the right side. The specific judgment method is not limited to this one. For example, it can also be determined whether the target vehicle is observed to be truncated based on d1, d2, the longitudinal pixel coordinate t of the upper boundary of the whole vehicle frame, the longitudinal pixel coordinate b of the lower boundary of the whole vehicle frame, and the height h of the to-be-processed image. It can also be determined whether the target vehicle is observed to be truncated and the truncation direction based on d1, d2, t, b, h, and r. For example, if d1 < D, d2 is greater than d1, t < a*h, b > c*h, and r > R, it is determined that the target is observed to be truncated, and the truncation direction is the first side truncation. Among them, a and c represent preset proportional coefficients, and R represents the ratio threshold.

[0064] Step 2032, in response to the target vehicle being observed to be truncated, based on the positional relationship between the whole vehicle frame information and the to-be-processed image, determine the target truncation direction of the target vehicle.

[0065] For the specific determination of the target truncation direction, reference can be made to the above example, which will not be elaborated here.

[0066] Step 2033, based on the whole vehicle frame information and the semantic segmentation information, perform a ground contact point search in the target search direction corresponding to the target truncation direction to determine the position of the ground contact point of the target vehicle's wheels.

[0067] Specifically, if the target truncation direction is right-side truncation, the target search direction is from left to right. If the target truncation direction is left-side truncation, the target search direction is from right to left. The search object is the set of semantic segmentation points included in the semantic segmentation information. Since the wheel contact point should be on the semantic segmentation boundary of the entire vehicle, the search object can specifically be the set of semantic segmentation boundary points. For example, the set of semantic segmentation boundary points of the entire vehicle can be determined from the semantic segmentation information based on the overall vehicle bounding box information. Based on the set of semantic segmentation boundary points of the entire vehicle, the set of semantic segmentation boundary points to be searched can be determined. By searching for the contact point in the set of semantic segmentation boundary points to be searched according to the contact point judgment rules, the position of the target vehicle's wheel contact point in the image to be processed can be obtained. Then, it can be transformed to the vehicle's coordinate system to obtain the wheel contact point position of the target vehicle in the vehicle coordinate system.

[0068] This embodiment determines whether the target vehicle is truncated. For truncated target vehicles, the set of semantic segmentation boundary points where the wheel contact points are located can be determined from the semantic segmentation information based on the full vehicle frame information. Then, the contact point search is performed according to the target search direction to determine the wheel contact point position of the target vehicle. Based on this information, the accurate determination of the wheel contact point position of the truncated target vehicle is achieved, thus improving the accuracy of the wheel contact point position.

[0069] Figure 6 This is a flowchart illustrating step 2033 provided in an exemplary embodiment of this disclosure.

[0070] In some optional embodiments, step 2033, based on the full vehicle frame information and semantic segmentation information, uses the target search direction corresponding to the target truncation direction to perform grounding point search and determine the wheel grounding point location of the target vehicle, includes:

[0071] Step 20331: Determine the first search range based on the full vehicle frame information and the target search direction.

[0072] The first search range may include a first search boundary and a second search boundary. For example, the first search boundary may be the left boundary and the second search boundary may be the right boundary, or the first search boundary may be the right boundary and the second search boundary may be the left boundary. The first search boundary and the second search boundary may be represented as horizontal pixel coordinates in the image to be processed, such as u1 and u2 respectively.

[0073] In some optional embodiments, the first search boundary is the boundary of the observed cutoff side. For example, if the target cutoff direction of the target vehicle is right cutoff, then the first search boundary is the right boundary. If the target cutoff direction is left cutoff, then the first search boundary is the left boundary.

[0074] For example, the first search boundary can be the full vehicle frame boundary on the cut-off side corresponding to the target cut-off direction. For instance, if the target cut-off direction is a right cut-off, then the first search boundary is the right boundary of the full vehicle frame.

[0075] In some optional embodiments, the second search boundary can be determined based on the front and rear frame information, the full vehicle frame information, and the side line information. For example, if the detection frame information includes front and rear frame information, the second search boundary can be determined based on the full vehicle frame information and the front and rear frame information. Alternatively, the second search boundary can be determined based on the full vehicle frame information and the side line information.

[0076] Step 20332: Based on the first search range, determine each first segmentation point in the semantic segmentation information that belongs to the first search range.

[0077] After determining the first search range, segmentation points belonging to the first search range can be obtained from the semantic segmentation information as the first segmentation points. Specifically, based on the first search boundary and the second search boundary of the first search range, semantic segmentation points near the bottom edge of the entire vehicle frame belonging to the first search range can be extracted from the semantic segmentation information. That is, semantic segmentation boundary points near the bottom edge of the entire vehicle frame belonging to the first search range can be extracted as the first segmentation points.

[0078] Step 20333: Search for each first segmentation point within the first search range according to the target search direction to determine the grounding point of each target wheel in the image pixel coordinate system of the target vehicle.

[0079] The target wheel ground contact point can include the ground contact point of at least one wheel. Searching for each first segmentation point allows for determination of whether each first segmentation point belongs to a wheel ground contact point based on corresponding wheel ground contact point judgment rules, thereby finding the target wheel ground contact point in the image pixel coordinate system of the target vehicle. Specific wheel ground contact point judgment rules can be set based on the relationship between the segmentation points of the wheel portion and the segmentation points of other portions in the full vehicle semantic segmentation boundary points.

[0080] Step 20334: Convert the contact points of each target wheel to the vehicle coordinate system to obtain the wheel contact point positions.

[0081] Specifically, based on the camera intrinsic and extrinsic parameters corresponding to the image to be processed, the contact points of each target wheel can be transformed to the vehicle coordinate system to obtain the wheel contact point positions. For example, the contact points of each target wheel can be transformed to the camera coordinate system first based on the camera intrinsic parameters, and then transformed to the vehicle coordinate system based on the camera extrinsic parameters.

[0082] In some alternative embodiments, when transforming the target wheel contact points to the camera coordinate system, depth information corresponding to each target wheel contact point is required. This depth information can be obtained in any feasible manner. For example, a depth image corresponding to the image to be processed can be obtained based on a depth camera, and the depth information corresponding to each target wheel contact point can be extracted from this depth image. Alternatively, a binocular image can be acquired based on a binocular camera (i.e., the image to be processed is a binocular image), and the depth information can be predicted based on the disparity between the binocular images. Furthermore, a depth prediction model can be used to predict the depth of the image to be processed, thereby obtaining the depth information corresponding to the image to be processed, and thus obtaining the depth information of each target wheel contact point.

[0083] This embodiment obtains a smaller effective search range by determining each first segmentation point within the first search range, and then performs a grounding point search on each first segmentation point to determine the wheel grounding point in the image pixel coordinate system, which helps to reduce the amount of search calculation and improve search efficiency.

[0084] In some optional embodiments, step 20331, which determines the first search range based on the full vehicle frame information and the target search direction, includes:

[0085] In response to the target vehicle being a large vehicle, a first search boundary is determined based on the target search direction and the full vehicle frame information; a second search boundary is determined based on the front and rear frame information, the full vehicle frame information, and the first boundary mapping rule; or, in response to the front and rear frame information being that no front or rear frame was detected, a second search boundary is determined based on the side line information, the full vehicle frame information, and the second boundary mapping rule; a first search range is determined based on the first search boundary and the second search boundary.

[0086] Wherein, if the target search direction is from left to right, the first search boundary is the right boundary. If the target search direction is from right to left, the first search boundary is the left boundary. The first boundary mapping rule can be determined based on the type of the target vehicle and the distance relationship between the wheel contact points and the front or rear of the vehicle. The second boundary mapping rule can be determined based on the relationship between the side line and the entire vehicle frame. The first search boundary and the second search boundary constitute the first search range.

[0087] For example, Figure 7 This is a schematic diagram illustrating a first type of right truncation provided in an exemplary embodiment of this disclosure. For example... Figure 7As shown, uov represents the image pixel coordinate system. Taking right truncation as an example, the first search boundary is the right search boundary rs (u coordinate), which is determined as the right boundary of the entire vehicle frame. The second search boundary is the left search boundary ls (u coordinate). If the left boundary of the head and tail frames determined by the head and tail frame information is represented as lside (u coordinate), and the right boundary of the entire vehicle frame is represented as rfull (u coordinate), then the first boundary mapping rule can be expressed as: ls = lside + e * (rfull - lside), where e is a preset value that can be set according to the distance between the wheel contact point and the front or rear of the vehicle. For example, for large vehicles, e can be set to 0.2, 0.25, 0.3, etc.

[0088] For example, such as Figure 7 As shown, taking the right truncated section as an example, the intersection point of the side line and the bottom edge of the entire vehicle frame can be determined based on the side line information. The lateral coordinate (i.e., u-coordinate) of this intersection point is represented as xcross, and the left boundary of the entire vehicle frame is represented as lfull (u-coordinate). Then, the second boundary mapping rule can be expressed as: ls = xcross - f * (xcross - lfull), where f is a preset value. For example, f can be set to 0.1 or other possible values.

[0089] In some alternative embodiments, the principles for determining the first and second search boundaries for left and right truncation are similar and will not be elaborated here.

[0090] This embodiment determines the first search boundary by using the full vehicle frame boundary on the cut-off side, and determines the second search boundary based on the first boundary mapping rule or the second boundary mapping rule, thereby obtaining the first search range. This helps to improve the accuracy and effectiveness of the search range, and further improves search efficiency.

[0091] In some optional embodiments, step 20333, which searches for each first segmentation point within the first search range according to the target search direction to determine each target wheel ground contact point in the image pixel coordinate system of the target vehicle, includes:

[0092] Based on pre-obtained camera parameters and side line information, the coordinates of the side line vanishing point are determined. Based on the side line vanishing point coordinates and each first segmentation point, the first slope corresponding to each first segmentation point is determined. Based on the relationship between the first slope corresponding to each first segmentation point and the second slope corresponding to the side line information, a first number of target segmentation points are determined. Based on the first number of target segmentation points, the corrected target side line is determined. Based on the target side line and the target search direction, the contact points of each target wheel are determined from each first segmentation point.

[0093] The camera parameters can include camera intrinsic parameters and camera extrinsic parameters.

[0094] In some optional embodiments, the vanishing point coordinates of the side lines can be determined based on the principle that the side lines of target vehicles facing the same direction in the distorted image will intersect at the same vanishing point. Therefore, the side line information can be distorted, and the vanishing point coordinates can be determined based on the distorted side line information. These vanishing point coordinates are the distorted coordinates. Then, each first segmentation point can be converted to the distorted image to obtain the distorted segmentation points corresponding to each first segmentation point. Based on the vanishing point coordinates of the side lines and each distorted segmentation point, the first slope corresponding to each first segmentation point is determined. The second slope corresponding to the side line information can be determined based on the distorted side line information; that is, the second slope is the slope of the distorted side line. Based on the relationship between the first and second slopes, a first number of target distorted segmentation points that satisfy the relationship condition can be determined. For example, the absolute values ​​of the differences between the first and second slopes can be sorted from smallest to largest, and the first number of first segmentation points with the highest sorting are selected as the target distorted segmentation points. The target distortion-reduced segmentation points are transformed to the distorted image coordinate system of the image to be processed to obtain the target segmentation points. The side lines are then corrected based on these segmentation points to obtain the corrected target side lines. Finally, based on the target side lines and the target search direction, a grounding point search is performed on each of the first segmentation points to obtain the grounding points of each target wheel.

[0095] This embodiment obtains the first slope corresponding to each first segmentation point by determining the coordinates of the vanishing point of the side line. Then, based on each first slope and the second slope corresponding to the side line information, a first number of target segmentation points are selected to correct the side line, which can improve the accuracy of the side line. Then, based on the corrected target side line, the target wheel contact point is searched from each first segmentation point, which helps to further improve the accuracy of the target wheel contact point.

[0096] In some optional embodiments, step 20333, which searches for each first segmentation point within the first search range according to the target search direction to determine each target wheel ground contact point in the image pixel coordinate system of the target vehicle, includes:

[0097] a1. Based on the pre-obtained camera parameters and side line information, determine the coordinates of the vanishing point of the side line.

[0098] In some alternative embodiments, Figure 8 This is a schematic diagram illustrating the principle of vanishing point determination provided by an exemplary embodiment of this disclosure, as shown below. Figure 8 As shown, the vanishing point coordinates of the side line are determined in the distortion-free image domain (i.e., the pixel coordinate system of the distortion-free image), denoted as ref_pt(refx, refy). Specifically, the camera intrinsic parameter matrix is ​​represented by K, and the rotation matrix is ​​represented by R. The calculation method from a point in 3D space (e.g., a point in the vehicle coordinate system or world coordinate system) to the distortion-free image domain is as follows:

[0099] s·pt image =P·pt world

[0100] P = K·R

[0101] Where s represents the observation factor, which is a pre-obtained value. pt image Points in the distorted image domain, pt world Let K represent a point in three-dimensional space, and P represent the transformation matrix. Since K and R are known, P is also known. P can be expressed as:

[0102]

[0103] Therefore, in the distortion-free image domain, the vanishing point (vp) where the yaw angle is 0. x ,vp y ) is represented as:

[0104] vp x =p 11 / p 31

[0105] vp y =p 21 / p 31

[0106] The vanishing line slope vp k It can be represented as:

[0107] vp k =(p 21 ×p 32 -p 22 ×p 31 ) / (p 11 ×p 32 -p 12 ×p 31 )

[0108] Vanishing line intercept vp b It can be represented as:

[0109] vp b =vp y -vp k ×vp x

[0110] Based on the system of equations formed by the vanishing line and the distortion-free side line, the coordinates of the vanishing point of the side line, ref_pt(refx,refy), can be obtained immediately.

[0111] a2. Based on the coordinates of the vanishing point of the side line and each of the first segmentation points within the first search range, determine the first slope corresponding to each of the first segmentation points.

[0112] Wherein, the first slope corresponding to any first segmentation point is the slope of the straight line fitted by the distortion-free segmentation point and the vanishing point of the side line corresponding to the first segmentation point.

[0113] a3. Based on the first slope corresponding to each first segmentation point and the second slope corresponding to the side line information, determine the first absolute value of the difference between each first slope and the second slope.

[0114] The second slope corresponding to the sideline information is the slope of the sideline after distortion correction. For each first slope, the difference between the first slope and the second slope is calculated, and the absolute value of this difference is taken as the first absolute value. Since each first segmentation point corresponds to a first slope, each first segmentation point corresponds to a first absolute value.

[0115] a4. Based on each first absolute value, determine the first number of target segmentation points with the smallest first absolute value from each first segmentation point.

[0116] The first quantity can be set according to actual needs. Based on the principle that the slope of the side line of the distorted image domain and the distorted image domain are approximately equal, the smaller the first absolute value, the closer the corresponding first segmentation point is to the side line. Therefore, the first number of target segmentation points with the smallest first absolute value is determined from all the first segmentation points to correct the side line.

[0117] a5. Based on the first number of target segmentation points, determine the corrected target side line.

[0118] Since the target segmentation points are semantic segmentation boundary points, they can more accurately reflect the true position of the side line. Therefore, by correcting the side line based on the first number of target segmentation points and determining the corrected target side line, the accuracy of the side line can be improved.

[0119] In some optional embodiments, the least squares method can be used to fit a first number of target segmentation points to obtain a more accurate corrected target sideline. This comprehensively considers both coarse-grained observed sidelines and precise semantic segmentation boundary points, improving the accuracy of the sidelines.

[0120] a6. Based on the target sideline, and according to the target search direction, determine the first distance between each first segmentation point in the first sub-range of the first search range and the target sideline.

[0121] In some optional embodiments, the first sub-range can be determined based on the boundaries of the front and rear frames and the boundary of the full vehicle frame. For example, in the case of a right truncation, the left boundary ls1 of the first sub-range can be the right boundary rsize of the front and rear frames, and the right boundary rs1 of the first sub-range can be a certain proportion of the width from the right boundary rsize of the front and rear frames to the right boundary rfull of the full vehicle frame, expressed as: rs1 = rsize + i * (rfull - rsize), where i represents a preset proportion value, for example, i can be 0.5, so that the first sub-range includes the contact point of the wheel near the front and rear frames. The first distance can be obtained based on the calculation method of the distance from a point to a line.

[0122] For example, suppose the distance from a point to a line has direction. For instance, a point on one side of the line (e.g., above) has a negative first distance, and a point on the other side of the line (e.g., below) has a positive first distance. The target side line is represented as y = kx + b, where x represents the horizontal coordinate (u coordinate) of the image coordinate system of the image to be processed, and y represents the vertical coordinate (v coordinate). The set of points formed by the first segmentation points within the first sub-range is represented as S1, and the coordinates of any first segmentation point within S1 are represented as (c x ,c y The first distance between the first dividing point and the target side line is denoted as d, and can be expressed as follows:

[0123]

[0124] In some optional embodiments, the first sub-range can also be determined based on the change in slope between each first segmentation point within the first search range and any target point on the target sideline. The target point can be any point on the target sideline; for example, the point whose lateral coordinates correspond to the starting point of the first search range on the target sideline can be taken as the target point. For each first segmentation point, the closer it is to the wheel contact point, the closer its slope is to the slope of the target sideline. Following the target search direction, the searched first segmentation points will continuously approach the first contact point, so their slopes will initially approach the slope of the target sideline, then move away from it. When the first segmentation point approaches the second wheel contact point, its slope gradually approaches the slope of the target sideline again. Based on this, the first sub-range is determined.

[0125] a7. Determine the first grounding point based on the first distance corresponding to each first segmentation point.

[0126] The first grounding point can be determined based on preset rules. These preset rules can be set according to actual needs.

[0127] In some alternative embodiments, the first dividing point with the largest first distance can be used as the first grounding point.

[0128] In some optional embodiments, the average of the plurality of first segmentation points with the largest first distance can also be used as the first grounding point. For example, the first distances can be sorted from largest to smallest, and the average of the top 3 points in the sorted first distance can be used as the first grounding point.

[0129] a8. Determine the third intercept based on the third slope of the first grounding point and the target side line.

[0130] The third slope of the target sideline refers to the slope of the target sideline. After determining the first grounding point, the first grounding point can be taken as a point on the target sideline. Then, combined with the slope of the target sideline, the intercept of the target sideline can be calculated, which is used as the third intercept.

[0131] a9. Determine the third sideline based on the third slope and third intercept of the target sideline.

[0132] With the slope and intercept known, a straight line can be determined, which serves as the third lateral line. This third lateral line is the lateral line when the first grounding point is taken as a point on the target lateral line.

[0133] a10. Based on the third side line, and following the target search direction, start the search from the next first segmentation point of the first grounding point within the first search range to determine the second grounding point.

[0134] In some alternative embodiments, a second sub-range can be determined, and a search can be performed at each of the first segmentation points within the second sub-range. A second grounding point is determined based on the distance between the searched first segmentation point and the third sideline. For example, the point within the second sub-range with the largest distance to the third sideline can be used as the second grounding point. The second sub-range can, for example, extend from the next first segmentation point after the first grounding point to the end of the first search range.

[0135] In some alternative embodiments, a second grounding point can also be determined based on a straight line fitted between the searched point and the first grounding point, and based on the relationship between the slope of the fitted straight line and the slope of the third side line.

[0136] In some optional embodiments, the number of target wheel grounding points can also be determined based on the number of wheel frames included in the detection frame information. If the number of wheel frames is greater than 2, after finding the second grounding point, the search can continue from the next grounding point after the second grounding point to determine the target wheel grounding points of the remaining wheels.

[0137] a11. Use the first and second grounding points as the grounding points of each target wheel.

[0138] In some optional embodiments, if the number of wheels of the target vehicle observed is greater than 2, in addition to the first grounding point and the second grounding point, the grounding points of other wheels can also be obtained, so that the first grounding point, the second grounding point and the grounding points of other wheels are used as the grounding points of each target wheel.

[0139] This embodiment improves the accuracy of the side line by correcting it. Then, based on the distance between the corrected target side line and each first segmentation point in the first sub-range, the first grounding point is found. Then, starting from the next first segmentation point after the first grounding point, the second grounding point is searched and determined, thereby achieving accurate and effective determination of the target wheel grounding point of the target vehicle.

[0140] In some optional embodiments, the method of this disclosure further includes:

[0141] b1. In response to the inability to determine the first grounding point based on each first distance, the first first division point of the first sub-range is taken as the first grounding point.

[0142] Specifically, for the first distance, an error tolerance of er can be set. The first distance d must satisfy d > -er. If the maximum first distance does not meet this condition, it is determined that the first grounding point cannot be obtained. Then, the first dividing point of the first sub-range can be used as the first grounding point.

[0143] Based on the third sideline of a10, following the target search direction, the search begins from the next first segmentation point after the first grounding point within the first search range. After determining the second grounding point, the following steps are also included:

[0144] b2. Determine the fourth intercept based on the third slope of the second grounding point and the target side line.

[0145] Specifically, the second grounding point is taken as a point on the target side line, and the fourth intercept corresponding to the target side line is determined.

[0146] b3. Determine the fourth side line based on the third slope and the fourth intercept.

[0147] The fourth side line is similar to the third side line mentioned above, so it will not be described in detail here.

[0148] b4. Determine the third grounding point based on the longitudinal coordinates of the fourth side line and the bottom edge of the entire vehicle frame.

[0149] In this case, given the fourth side edge line, the longitudinal coordinates of the bottom edge of the entire vehicle frame can be substituted into the linear equation of the fourth side edge line to obtain the lateral coordinates. Based on the longitudinal coordinates of the bottom edge of the entire vehicle frame and the obtained lateral coordinates, the third grounding point can be obtained.

[0150] b5. Use the third grounding point as the first grounding point.

[0151] In this embodiment, if the first grounding point cannot be found, the first first division point of the first sub-range is taken as the first grounding point. The accuracy of the first grounding point is not very high, while the accuracy of the second grounding point is higher. Therefore, the intercept of the target side line is updated again based on the second grounding point to update the first grounding point, which helps to improve the accuracy of the first grounding point.

[0152] In some optional embodiments, for cases where the large vehicle observation is truncated, the boundary point of the truncated side (the first segmentation point closest to the truncated side) can be used as the first grounding point. The target search direction is opposite to the target search direction described above. For example, if it is a right truncation, the target search direction is from right to left; if it is a left truncation, the search direction is from left to right. Then, the second grounding point can be determined according to steps a8 to a10 above. Alternatively, the second grounding point can be determined based on the first grounding point and the uncorrected side line. In this case, if the large vehicle is close to the vehicle and the side line is inaccurate, the determined second grounding point may be a point at the front or rear of the vehicle, leading to abnormal second grounding point location. To address this, the second grounding point can be corrected after obtaining it. Specifically, the correction can be performed in the following ways:

[0153] c1. Based on the second grounding point, determine the first target point from each of the first segmentation points whose longitudinal distance from the second grounding point meets the preset distance condition.

[0154] The preset distance condition can be the maximum longitudinal distance. Specifically, starting from the first segmentation point after the determined first grounding point, the longitudinal distance between each first segmentation point and the second grounding point can be calculated. For example, the longitudinal distance can be calculated using the longitudinal coordinates of each first segmentation point and the longitudinal coordinates of the second grounding point. For instance, for any first segmentation point, the absolute value of the difference between the longitudinal coordinates of the first segmentation point and the second grounding point is taken as the longitudinal distance between the first segmentation point and the second grounding point. The first segmentation point with the largest longitudinal distance is taken as the first target point.

[0155] c2. Determine the first radius of curvature based on the first target point, the first grounding point, and the second grounding point.

[0156] For example, Figure 9 This is a schematic diagram illustrating an abnormal situation in finding the second grounding point, provided in an exemplary embodiment of this disclosure. For example... Figure 9As shown, taking the right section of a large vehicle as an example, the first grounding point B is the segmentation boundary point corresponding to the right boundary of the entire vehicle frame (i.e., the first segmentation point closest to the right boundary of the entire vehicle frame). Due to the inaccuracy of the side line, the second grounding point determined based on the first grounding point and the side line is the second grounding point A in the figure. The second grounding point A is a point on the front or rear of the vehicle, not the actual second grounding point. For example, the actual second grounding point should be point M. Then, based on the second grounding point A, the point with the largest longitudinal distance from the second grounding point among all the first segmentation points is determined as the first target point, for example, the first target point is point M. The first radius of curvature is determined based on the first target point, the first grounding point B, and the second grounding point A.

[0157] For example, if the coordinates of the second grounding point A are (x1, y1), the coordinates of the first target point M are (x2, y2), and the coordinates of the first grounding point B are (x3, y3), then the area of ​​the triangle formed by these three points is:

[0158] S=‖x1×y2+x2×y3+x3×y1-y1×x2-y2×x3-y3×x1‖

[0159] If the length of AM is denoted as α, the length of BM as b, and the length of AB as c, then the first radius of curvature R can be expressed as follows:

[0160]

[0161] c3. In response to the first curvature radius being within the preset curvature radius range, the first target point is used as the second grounding point.

[0162] The preset radius of curvature range can be set according to actual needs, for example, it can be set to [R1, R2]. R1 can be 195, 200, 205, etc., and R2 can be 450, 455, 460, etc., without specific limitations. If the first radius of curvature is within the preset radius of curvature range, the first target point can be used as the second grounding point. Then, the first grounding point is predicted in reverse based on the second grounding point. For example, the first grounding point is determined according to steps b2 to b5 above. Thus, the accurate first and second grounding points are obtained.

[0163] This embodiment can correct the second grounding point in case of abnormal location, improve the accuracy of the second grounding point, and then predict the first grounding point based on the second grounding point, further improving the accuracy of the first grounding point.

[0164] In some optional embodiments, the detection box information further includes wheel frame information, which includes first frame information corresponding to at least one wheel. Step 203, which determines the wheel ground contact position of the target vehicle based on the detection box information and semantic segmentation information, using a determination method corresponding to the target observation type, further includes:

[0165] Step 2034: In response to the target vehicle not being observed and truncated, determine the first wheel frame segmentation point set corresponding to each wheel based on the wheel frame information and semantic segmentation information of the target vehicle.

[0166] In this context, "not observational truncation" means that the entire target vehicle is observed in the image to be processed, and no part of the target vehicle is truncated outside the image region. Based on the wheel frame information of the target vehicle, the set of segmentation points belonging to the wheels in the semantic segmentation information can be determined as the first wheel frame segmentation point set. For example, based on the lateral coordinates of the left and right boundaries of the wheel frame and the ordinates of the upper and lower boundaries, the set of semantic segmentation boundary points within the wheel frame range is extracted from the semantic segmentation information as the first wheel frame segmentation point set.

[0167] Step 2035: For any wheel corresponding to the first wheel frame segmentation point set, determine the first boundary point, the second boundary point, and the first intermediate point based on the first wheel frame segmentation point set.

[0168] Here, the first boundary point is the semantic segmentation boundary point closest to the first boundary of the wheel frame, such as the leftmost semantic segmentation boundary point within the wheel frame. Similarly, the second boundary point is the semantic segmentation boundary point closest to the second boundary of the wheel frame, such as the rightmost semantic segmentation boundary point within the wheel frame. The first intermediate point refers to the midpoint of the first wheel frame segmentation point set located between the first boundary point and the second boundary point.

[0169] Step 2036: Determine the target grounding point of the wheel based on the first boundary point, the second boundary point, and the first intermediate point.

[0170] For any wheel, the target grounding point of the wheel can be determined starting from the first boundary point, based on the change in distance between two adjacent segmentation points in the first wheel frame segmentation point set.

[0171] Step 2037: Determine the wheel contact point location of the target vehicle based on the target contact point corresponding to each wheel.

[0172] Since the number of wheels may vary depending on the type of target vehicle, for example, a small car may have two wheels on each side, while a large vehicle may have two or more sets of wheels on each side, with each set containing one or two wheels arranged side-by-side. Therefore, for different target vehicles, the wheel contact point positions can be determined based on corresponding rules. For example, for a small car, the target contact points corresponding to the two wheels can be used as the target wheel contact points in the image pixel coordinate system of the target vehicle, and then the target wheel contact points can be transformed to the vehicle's coordinate system to obtain the wheel contact point positions. For a large vehicle, if the target contact points include multiple wheel contact points in each of multiple sets of wheels, for each set of wheels, the target contact point closest to the bottom edge of the entire vehicle frame (i.e., the closest wheel contact point relative to the vehicle) can be used as the corresponding target contact point for that set of wheels to determine its position. Specific rules are not limited.

[0173] In this embodiment, for target vehicles that have not been observed and truncated, the wheel frame segmentation point set of each wheel can be extracted from the semantic segmentation information based on the wheel frame information of each wheel of the target vehicle. Then, the target ground point of the wheel can be determined based on the wheel frame segmentation point set, so as to achieve accurate and effective determination of the wheel ground point of the target vehicle that has not been observed and truncated.

[0174] In some optional embodiments, step 2036, which determines the target ground contact point of the wheel based on the first boundary point, the second boundary point, and the first intermediate point, includes:

[0175] d1. Starting from the first boundary point, determine the second distance between any two adjacent wheel frame segmentation points in the first wheel frame segmentation point set.

[0176] The second distance refers to the longitudinal distance. For example, the second distance is calculated based on the longitudinal coordinates of the dividing points of two adjacent wheel frames.

[0177] d2. Based on each second distance, determine the first starting segmentation point and the first ending segmentation point that satisfy the first condition.

[0178] The first condition can be that the second distance is continuously less than the second distance threshold. That is, the first starting point and the first ending point refer to the starting point and the ending point among multiple wheel frame segmentation points where the second distance is continuously less than the second distance threshold.

[0179] For example, Figure 10 This is a schematic diagram of four sets of first wheel frame segmentation points provided in an exemplary embodiment of this disclosure. For example... Figure 10As shown, the first wheel frame segmentation point set can include four cases: a, b, c, and d. Taking a search from left to right as an example, the first point 1 on the left in each case is the first boundary point, the rightmost point 2 is the second boundary point, and the middle point 3 is the first middle point. Figure 10 Taking case a as an example, if the last four points in a satisfy the first condition, then stard represents the first starting point and end represents the first ending point.

[0180] d3. Based on the first starting segmentation point and the first ending segmentation point, determine the target grounding point of the wheel.

[0181] The target grounding point of the wheel can be determined based on the location of the first starting segmentation point and the first ending segmentation point at the set of segmentation points on the first wheel frame.

[0182] For example, for Figure 10 Cases a, b, and d shown all have a first starting point and a first ending point, and the target grounding point differs in each case. For example, in case a, the target grounding point of the wheel is the first starting point (start). In case b, the target grounding point of the wheel is the first ending point (end). In case d, the target grounding point is the first intermediate point (3).

[0183] d4. In response to the absence of a first starting segmentation point and a first ending segmentation point that satisfy the first condition, the wheel frame segmentation point closest to the bottom edge of the wheel frame is taken as the target grounding point of the wheel.

[0184] If each second distance is greater than the second distance threshold, it indicates that the vertical variation of the segmentation points of adjacent wheel frames is consistently large. Figure 10 In case c, the wheel frame segmentation point closest to the bottom edge of the wheel frame (i.e., the bottom edge) (such as point 3 in c) is taken as the target ground point of the wheel.

[0185] This embodiment determines the target wheel point under various wheel frame segmentation point sets based on the longitudinal distance between adjacent wheel frame segmentation points, which helps to further improve the accuracy of the target wheel point.

[0186] In some optional embodiments, step 2036, which determines the target ground contact point of the wheel based on the first boundary point, the second boundary point, and the first intermediate point, further includes:

[0187] In response to the failure to determine the target ground point of the wheel or the ratio of the determined target ground point to the third distance of the bottom edge of the wheel frame to the height of the wheel frame being greater than a first threshold, the midpoint of the bottom edge of the wheel frame is taken as the target ground point of the wheel.

[0188] The first threshold can be set according to actual needs; for example, it can be 1 / 3, 2 / 5, etc., without specific limitations. If the target ground point of the wheel cannot be determined using the above method, the midpoint of the bottom edge of the wheel rim can be used as the target ground point. Alternatively, after determining the target ground point using the above method, the target ground point can be further judged by determining the third distance between the target ground point and the bottom edge of the wheel rim, and calculating the ratio of the third distance to the height of the wheel rim. If the ratio of the third distance to the height of the wheel rim is greater than the first threshold, it indicates that the target ground point is not accurate enough, and the midpoint of the bottom edge of the wheel rim can also be used as the target ground point.

[0189] This embodiment can further determine the third distance between the target grounding point and the bottom edge of the wheel frame. The target grounding point is verified based on the ratio of the third distance to the height of the wheel frame, so as to discover inaccurate target grounding points in time. Then, the midpoint of the relatively accurate bottom edge of the wheel frame is taken as the target grounding point, which helps to avoid finding the wrong wheel grounding point.

[0190] In some optional embodiments, step 2037, determining the wheel contact point location of the target vehicle based on the target contact point corresponding to each wheel, includes:

[0191] In response to the detection of wheel frame information including first frame information corresponding to multiple front wheels and first frame information corresponding to multiple rear wheels, the target front wheel is determined from multiple front wheels and the target rear wheel is determined from multiple rear wheels based on the second slope corresponding to the side line information; the target ground point of the target front wheel and the target ground point of the target rear wheel are transformed into the vehicle coordinate system to obtain the wheel ground point position.

[0192] In cases involving multiple front wheels and multiple rear wheels (i.e., many-to-many situations), the side lines can be used as a guide to determine the most suitable target front wheel and target rear wheel. Then, the target ground points of the target front wheel and the target ground points of the target rear wheel can be transformed into the vehicle coordinate system to obtain the target wheel ground point positions of the target vehicle.

[0193] For example, Figure 11 This is a schematic diagram of a plurality of front wheels and a plurality of rear wheels provided in an exemplary embodiment of this disclosure. Figure 11 As shown, it includes two sets of wheels (one set of front wheels and one set of rear wheels), with each set of wheels corresponding to two wheel frames. For example, if the head and tail frame is the head detection frame, then the two wheel frames on the left are the front wheel frames, and the two wheel frames on the right are the rear wheel frames. If the head and tail frame is the tail detection frame, then the left side is the rear wheel and the right side is the front wheel.

[0194] In some optional embodiments, based on the target ground points corresponding to multiple front wheels and multiple target ground points corresponding to multiple rear wheels, the slope of the straight line containing each front wheel target ground point and each rear wheel target ground point can be determined. The front wheel and the rear wheel with the second slope closest to the side line information can be determined as the target front wheel and the target rear wheel, respectively. The target ground points of the target front wheel and the target ground points of the target rear wheel can be transformed into the vehicle coordinate system to obtain the wheel ground point positions.

[0195] In some optional embodiments, step 2037, which determines the wheel contact point location of the target vehicle based on the target contact point corresponding to each wheel, further includes:

[0196] In response to the detection of wheel frame information including first frame information corresponding to multiple front wheels or multiple first frame information corresponding to multiple rear wheels, two front wheels or two rear wheels are determined based on two slopes; the first front wheel closest to the front and rear frames or the first rear wheel closest to the front and rear frames is taken as the target wheel; the target ground point of the target wheel is taken as the first ground point; a first straight line passing through the two ground points is determined based on the first ground point and the second slope; a second search range is determined based on the front and rear frame information and the whole vehicle frame information; each second segmentation point in the semantic segmentation information belonging to the second search range is determined based on the second search range; the current window is determined based on the target search direction, each second segmentation point in the second search range, and the preset sliding window rules; a first vector is determined based on the middle segmentation point and the second starting segmentation point in the current window; a second vector is determined based on the middle segmentation point and the second ending segmentation point in the current window; in response to the angle between the first vector and the second vector being within a preset angle range, the middle segmentation point is taken as the second wheel segmentation point; the second ground point is determined based on the second wheel segmentation point, the first ground point, and the first straight line; the wheel ground point position is determined based on the first ground point and the second ground point.

[0197] In cases where the detection frame information includes first frame information corresponding to multiple front wheels or multiple rear wheels (i.e., zero-to-many or many-to-zero cases), the two front wheels or two rear wheels can be determined based on the second slope corresponding to the side line information. For example, the leftmost and rightmost first frame information corresponding to multiple front wheels can be used as the first frame information corresponding to two front wheels to determine the two front wheels. Alternatively, the two front wheels whose slopes are closest to the second slope can be determined based on the relationship between the slope of the line connecting each pair of target ground points of multiple front wheels. The principle for determining the two rear wheels is similar to that of the front wheels.

[0198] In some optional embodiments, when the front and rear frames are the head frame, the front wheel closest to the head and rear frame (the first front wheel) is designated as the target wheel. When the front and rear frames are the rear frame, the rear wheel closest to the head and rear frame (the first rear wheel) is designated as the target wheel. The target contact point of the target wheel is designated as the first contact point. Then, based on the second slope corresponding to the first contact point and the side line information, a first straight line is determined. For example, the first contact point can be designated as a point on the first straight line, the second slope can be designated as the slope of the first straight line, and the intercept of the first straight line can be calculated. Thus, the first straight line can be obtained based on the second slope and the intercept.

[0199] In some optional embodiments, the second search range can be the side boundary of the entire vehicle frame from one side near the front and rear frames to the other side. For example, if one side of the front and rear frames is the left side, the second search range can be from the right boundary of the front and rear frames to the right boundary of the entire vehicle frame, and the corresponding target search direction is from left to right. The determination of the second segmentation point within the second search range is similar to the first segmentation point in the aforementioned embodiments, and will not be repeated here.

[0200] In some optional embodiments, preset sliding window rules can be set according to actual needs. For example, preset sliding window rules may include window size, sliding step size, etc. The sliding window slides along the target search direction within a second search range, and each window reached is taken as the current window. Based on the middle dividing point and the second starting dividing point in the current window, a first vector is determined. The second starting dividing point is the starting point in the current window. Based on the middle dividing point and the second ending dividing point in the current window, a second vector is determined. The second ending dividing point is the ending point in the current window. The preset angle range [A1, A2] can be set according to actual needs. For example, the preset angle range can be set to a range of [135 degrees, 160 degrees] or other feasible ranges, such as [140 degrees, 160 degrees]. The boundary values ​​of the specific preset angle range are not limited. If the angle between the first vector and the second vector is within the preset angle range, the middle dividing point of the current window can be taken as the second wheel dividing point. Then, based on the second wheel dividing point, the first grounding point, and the first straight line, the second grounding point is determined. Based on the first grounding point and the second grounding point, the position of the wheel grounding point is obtained.

[0201] In some optional embodiments, the second wheel segmentation point and the first ground contact point can be transformed into the vehicle coordinate system of the vehicle to obtain the first projection point corresponding to the second wheel segmentation point and the second projection point corresponding to the first ground contact point. The target distance between the first projection point and the second projection point is calculated. If the target distance is less than a preset multiple of the target vehicle's body length (this preset multiple can be determined based on the type of the target vehicle, specifically the ratio between the distance between the front and rear wheels of the target vehicle and the target vehicle's body length), or if the absolute value of the difference between the slope of the second straight line fitted by the second wheel segmentation point and the first ground contact point and the second slope corresponding to the side line information is greater than a preset threshold (e.g., 0.15, 0.2, 0.25, etc.), the window continues to slide, and the slid window is used as the current window. The search continues in the above manner until the target distance corresponding to the second wheel segmentation point is greater than a preset multiple of the target vehicle's body length, and the absolute value of the difference between the slope of the second straight line corresponding to the second wheel segmentation point and the second slope is less than the preset threshold. Then, the second wheel segmentation point is designated as the second ground contact point.

[0202] In this embodiment, for cases where the wheel frame information of the detection frame is many-to-zero or zero-to-many, an accurate and effective second grounding point can be determined based on a sliding window, which helps to improve the accuracy of the wheel grounding point position of the target vehicle.

[0203] In some optional embodiments, after determining the first and second grounding points, the first and second grounding points can be transformed from the distorted image domain of the image to be processed to the dedistorted image domain, and then transformed from the dedistorted image domain to the vehicle coordinate system to obtain the wheel grounding point positions of the two wheels in the vehicle coordinate system. It can then be determined whether the two wheel grounding point positions are the same. If they are not the same, the yaw angle of the target vehicle can be calculated based on the two wheel grounding point positions. Here, the yaw angle does not distinguish between the front and rear directions, and the final determined yaw angle of the target vehicle is within the range of [-90°, 90°]. The front and rear directions of the target vehicle can be distinguished based on the attributes of the head and tail bounding boxes. For example, when detecting the bounding box information of the target vehicle, the specific attributes of the head and tail bounding boxes can be directly predicted, such as 1 representing the head bounding box and 0 representing the tail bounding box, or other representation methods can be used, without limitation. Alternatively, head and tail bounding box region images can be extracted from the image to be processed based on the head and tail bounding box information, and secondary prediction can be performed based on a pre-trained head and tail bounding box attribute classification model to obtain the attributes of the head and tail bounding boxes. The specific method of obtaining the head and tail bounding box attributes is not limited. Based on the position of the wheel contact point in the vehicle coordinate system of the first or second contact point, the position of the target vehicle in the vehicle coordinate system can be determined.

[0204] In some optional embodiments, step 2037, which determines the wheel contact point location of the target vehicle based on the target contact point corresponding to each wheel, further includes:

[0205] In response to the failure to determine the second grounding point, the entire vehicle frame information is projected onto the vehicle coordinate system to obtain the first front bumper projection point and the first grounding point projection point in the vehicle coordinate system. Based on the first front bumper projection point, the first grounding point projection point, and the predicted vehicle body width, a first position reference point is determined. The first position reference point and the first grounding point projection point are used together to determine the heading angle and position of the target vehicle.

[0206] If the second grounding point cannot be determined using the sliding window method described above, the entire vehicle frame information can be projected onto the vehicle coordinate system to obtain the first front bumper projection point and the first grounding point projection point in the vehicle coordinate system. Then, based on the first front bumper projection point and the first grounding point projection point, combined with the target vehicle's width, a first position reference point is determined. Based on the first position reference point and the first grounding point projection point, the target vehicle's heading angle and position are determined. Subsequent planning and control can then be performed based on the target vehicle's heading angle and position. For example, it can be determined whether the target vehicle is a potential obstacle, i.e., whether there is a collision risk, and whether corresponding collision avoidance functions, such as automatic emergency braking, need to be activated. Specific subsequent applications of the target vehicle's heading angle and position are not limited.

[0207] For example, Figure 12 This is a schematic diagram illustrating the principle of determining a first position reference point provided in an exemplary embodiment of this disclosure. For example... Figure 12As shown, in the image pixel coordinate system (Uov coordinate system), the first front bumper point (i.e., the hidden front bumper point of the target vehicle away from its own vehicle) in the full vehicle frame information is p1, the first ground point is p3, and p2 is the pixel coordinate point of the first position reference point to be determined in the image pixel coordinate system. After the full vehicle frame information is projected onto the vehicle coordinate system (XOY coordinate system), the first front bumper projection point corresponding to p1 is P1, the first ground point projection point corresponding to p3 is P3, and P2 corresponding to p2 is the first position reference point. L represents the body length of the target vehicle, and W represents the body width of the target vehicle. The body length and body width of the target vehicle can be obtained in any feasible way. For example, the vehicle type of the target vehicle can be determined based on detection, and the body length and body width corresponding to that vehicle type can be obtained based on the vehicle type. The body length and body width can also be obtained based on other feasible methods. For example, the body length and body width of the target vehicle can be predicted based on a neural network prediction model. No specific limitations are imposed. Given P1, P3, and the vehicle width W, P2 can be determined as follows: Find a point P2 on a circle with P1P3 as its diameter. The distance between P2 and P1 is W. Specifically, P1 is represented as (x1, y1), and P3 as (x2, y2). The center of the circle with P1P3 as its diameter is (x3, y3), and its radius is R. The equation for solving point P2(x, y) can be expressed as follows:

[0208]

[0209] Based on the above equations, the problem can be determined as finding the intersection point of two circles, and the distance D between the centers is expressed as:

[0210]

[0211] The X-coordinate of the intersection point is:

[0212]

[0213] The Y-coordinate of the intersection point is:

[0214]

[0215] If the two circles are not tangent, there will be two solutions. The coordinates of P2 in one of these solutions can be determined based on the fact that P2 should be below the line connecting P1 and P3, thus obtaining the first position reference point. Although this first position reference point is not the wheel contact point, it can assist in calculating the target vehicle's heading angle and position.

[0216] In this embodiment, for cases where the second grounding point cannot be determined by the above sliding window method, the first position reference point can be determined based on the projection point of the first front bumper and the projection point of the first grounding point in the vehicle coordinate system, combined with the width of the target vehicle body. This helps to assist in calculating the heading angle and position of the target vehicle.

[0217] Figure 13 This is a flowchart illustrating a method for determining the wheel contact position according to yet another exemplary embodiment of this disclosure.

[0218] In some optional embodiments, the detection frame information includes wheel frame information and side line information. For example... Figure 13 As shown, step 203, based on detection box information and semantic segmentation information, uses a determination method corresponding to the target observation type to determine the wheel contact point location of the target vehicle, including:

[0219] Step 203a: In response to the target observation type being the third type, the second wheel frame segmentation point set corresponding to each wheel is determined based on the wheel frame information and semantic segmentation information.

[0220] The principle for determining the second wheel frame segmentation point set is similar to that of the first wheel frame segmentation point set in the aforementioned embodiment, and will not be repeated here.

[0221] Step 203b: Based on the second wheel frame segmentation point set and side line information corresponding to each wheel, determine the wheel contact point position of the target vehicle.

[0222] The specific operation of determining the wheel contact point position of the target vehicle based on the second wheel frame segmentation point set and side line information corresponding to each wheel can be referred to in steps 2035 to 2037 of the aforementioned embodiment, which describes the specific operation of determining the wheel contact point position of the target vehicle based on the first wheel frame segmentation point set. It will not be repeated here.

[0223] For target vehicles with observation type 3, this embodiment can determine the wheel contact point location of the target vehicle based on the wheel frame segmentation point set and side line information, thereby improving the accuracy of the wheel contact point location by combining the semantic segmentation boundary point set of the wheel.

[0224] The embodiments or optional examples disclosed herein can be implemented individually or in any combination without conflict. The specific implementation can be set according to actual needs, and this disclosure does not limit it.

[0225] Any of the wheel ground contact position determination methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the wheel ground contact position determination methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the wheel ground contact position determination methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0226] Exemplary device

[0227] Figure 14 This is a schematic diagram of a device for determining the wheel contact position according to an exemplary embodiment of this disclosure. The device in this embodiment can be used to implement corresponding embodiments of the method for determining the wheel contact position of this disclosure, such as... Figure 14 The device shown includes: a first processing module 51, a second processing module 52 and a third processing module 53.

[0228] The first processing module 51 is used to determine the detection box information and semantic segmentation information of the target vehicle in the image pixel coordinate system based on the image to be processed.

[0229] The second processing module 52 is used to determine the target observation type of the target vehicle based on the detection box information.

[0230] The third processing module 53 is used to determine the wheel contact point position of the target vehicle based on the detection box information and semantic segmentation information, using a determination method corresponding to the target observation type. The wheel contact point position is the position in the vehicle coordinate system.

[0231] Figure 15 This is a schematic diagram of the structure of a wheel contact point determination device provided in another exemplary embodiment of this disclosure.

[0232] In some optional embodiments, the second processing module 52 includes:

[0233] The first determining unit 521 is used to determine the target observation type as the first type in response to the detection frame information including the whole vehicle frame information, the front and rear frame information and the side line information, wherein the front and rear frame information is located on the first side of the whole vehicle frame information.

[0234] The second determining unit 522 is used to determine the target observation type as the second type in response to the detection frame information including the whole vehicle frame information, the front and rear frame information and the side line information, wherein the front and rear frame information is located on the second side of the whole vehicle frame information.

[0235] The third determining unit 523 is used to determine the target observation type based on the side line information in response to the detection frame information including the whole vehicle frame information, wheel frame information and side line information, but excluding the front and rear frame information.

[0236] In some optional embodiments, the detection frame information includes full vehicle frame information, front and rear frame information, and side line information. The third processing module 53 includes:

[0237] The first processing unit 531 is used to determine whether the target vehicle observation is truncated based on the side line information and the full vehicle frame information in response to whether the target observation type is the first type or the second type.

[0238] The second processing unit 532 is used to determine the target truncation direction of the target vehicle based on the positional relationship between the full vehicle frame information and the image to be processed in response to the target vehicle observation truncation.

[0239] The third processing unit 533 is used to perform grounding point search based on the full vehicle frame information and semantic segmentation information, using the target search direction corresponding to the target truncation direction, to determine the wheel grounding point position of the target vehicle.

[0240] In some optional embodiments, the third processing unit 533 is specifically used for:

[0241] Based on the full vehicle frame information and the target search direction, a first search range is determined. Based on the first search range, each first segmentation point belonging to the first search range in the semantic segmentation information is determined. Following the target search direction, each first segmentation point within the first search range is searched to determine the target vehicle's wheel contact points in the image pixel coordinate system. Each target wheel contact point is then transformed to the vehicle coordinate system to obtain the wheel contact point positions.

[0242] In some optional embodiments, the third processing unit 533 is specifically used for:

[0243] In response to the target vehicle being a large vehicle, a first search boundary is determined based on the target search direction and the full vehicle frame information; a second search boundary is determined based on the front and rear frame information, the full vehicle frame information, and the first boundary mapping rule; or, in response to the front and rear frame information being that no front or rear frame was detected, a second search boundary is determined based on the side line information, the full vehicle frame information, and the second boundary mapping rule; a first search range is determined based on the first search boundary and the second search boundary.

[0244] In some optional embodiments, the third processing unit 533 is specifically used for: determining the vanishing point coordinates of the sideline based on pre-obtained camera parameters and sideline information; determining the first slope corresponding to each first segmentation point based on the vanishing point coordinates and each first segmentation point; determining a first number of target segmentation points based on the relationship between the first slope corresponding to each first segmentation point and the second slope corresponding to the sideline information; determining the corrected target sideline based on the first number of target segmentation points; and determining the target wheel contact point from each first segmentation point based on the target sideline and the target search direction.

[0245] In some optional embodiments, the third processing unit 533 is specifically used for:

[0246] Based on pre-obtained camera parameters and sideline information, the vanishing point coordinates of the sideline are determined. Based on the vanishing point coordinates and each first segmentation point within the first search range, a first slope corresponding to each first segmentation point is determined. Based on the first slope corresponding to each first segmentation point and a second slope corresponding to the sideline information, a first absolute value of the difference between each first slope and the second slope is determined. Based on each first absolute value, a first number of target segmentation points with the smallest first absolute value are determined from among the first segmentation points. Based on the first number of target segmentation points, a corrected target sideline is determined. Based on the target sideline, and according to the target search direction, a first distance between each first segmentation point within the first sub-range of the first search range and the target sideline is determined. Based on the first distance corresponding to each first segmentation point, a first grounding point is determined. Based on the first grounding point and a third slope of the target sideline, a third intercept is determined. Based on the third slope and the third intercept of the target sideline, a third sideline is determined. Based on the third sideline, and according to the target search direction, starting from the next first segmentation point after the first grounding point within the first search range, a second grounding point is determined. The first and second grounding points are used as the grounding points for each target wheel.

[0247] In some optional embodiments, the third processing unit 533 is further configured to:

[0248] In response to the inability to determine the first grounding point based on the first distances, the first first segmentation point of the first sub-range is taken as the first grounding point. Based on the third sideline, and following the target search direction, the search begins from the next first segmentation point within the first search range after the first grounding point. After determining the second grounding point, the fourth intercept is determined based on the third slope of the second grounding point and the target sideline. The fourth sideline is determined based on the third slope and the fourth intercept. The third grounding point is determined based on the longitudinal coordinates of the fourth sideline and the bottom edge of the entire vehicle frame. This third grounding point is taken as the first grounding point.

[0249] In some optional embodiments, the third processing unit 533 is further configured to:

[0250] Based on the second grounding point, a first target point is determined from each of the first segmentation points whose longitudinal distance from the second grounding point meets a preset distance condition. Based on the first target point, the first grounding point, and the second grounding point, a first radius of curvature is determined. In response to the first radius of curvature being within a preset radius of curvature range, the first target point is used as the second grounding point.

[0251] In some optional embodiments, the detection frame information further includes wheel frame information, which includes first frame information corresponding to at least one wheel. The third processing module 53 further includes: a fourth processing unit 534, a fifth processing unit 535, a sixth processing unit 536, and a seventh processing unit 537.

[0252] The fourth processing unit 534 is used to determine the first wheel frame segmentation point set corresponding to each wheel based on the wheel frame information and semantic segmentation information of the target vehicle in response to the target vehicle not being observed to be truncated.

[0253] The fifth processing unit 535 is used to determine the first boundary point, the second boundary point, and the first intermediate point based on the first wheel frame segmentation point set corresponding to any wheel.

[0254] The sixth processing unit 536 is used to determine the target grounding point of the wheel based on the first boundary point, the second boundary point, and the first intermediate point.

[0255] The seventh processing unit 537 is used to determine the wheel grounding point position of the target vehicle based on the target grounding point corresponding to each wheel.

[0256] In some optional embodiments, the sixth processing unit 536 is specifically used for:

[0257] Starting from the first boundary point, determine the second distance between any two adjacent wheel frame segmentation points in the first wheel frame segmentation point set. Based on each second distance, determine the first starting segmentation point and the first ending segmentation point that satisfy the first condition. Based on the first starting segmentation point and the first ending segmentation point, determine the target grounding point of the wheel. In response to the absence of a first starting segmentation point and a first ending segmentation point that satisfy the first condition, take the wheel frame segmentation point in the first wheel frame segmentation point set that is closest to the bottom edge of the wheel frame as the target grounding point of the wheel.

[0258] In some optional embodiments, the sixth processing unit 536 is further configured to:

[0259] In response to the failure to determine the target ground point of the wheel or the ratio of the determined target ground point to the third distance of the bottom edge of the wheel frame to the height of the wheel frame being greater than a first threshold, the midpoint of the bottom edge of the wheel frame is taken as the target ground point of the wheel.

[0260] In some optional embodiments, the seventh processing unit 537 is specifically used for:

[0261] In response to the detection of wheel frame information including first frame information corresponding to multiple front wheels and first frame information corresponding to multiple rear wheels, the target front wheel is determined from multiple front wheels and the target rear wheel is determined from multiple rear wheels based on the second slope corresponding to the side line information; the target ground point of the target front wheel and the target ground point of the target rear wheel are transformed into the vehicle coordinate system to obtain the wheel ground point position.

[0262] In some optional embodiments, the seventh processing unit 537 is further configured to:

[0263] In response to the detection of wheel frame information including first frame information corresponding to multiple front wheels or multiple first frame information corresponding to multiple rear wheels, two front wheels or two rear wheels are determined based on two slopes; the first front wheel closest to the front and rear frames or the first rear wheel closest to the front and rear frames is taken as the target wheel; the target ground point of the target wheel is taken as the first ground point; a first straight line passing through the two ground points is determined based on the first ground point and the second slope; a second search range is determined based on the front and rear frame information and the whole vehicle frame information; each second segmentation point in the semantic segmentation information belonging to the second search range is determined based on the second search range; the current window is determined based on the target search direction, each second segmentation point in the second search range, and the preset sliding window rules; a first vector is determined based on the middle segmentation point and the second starting segmentation point in the current window; a second vector is determined based on the middle segmentation point and the second ending segmentation point in the current window; in response to the angle between the first vector and the second vector being within a preset angle range, the middle segmentation point is taken as the second wheel segmentation point; the second ground point is determined based on the second wheel segmentation point, the first ground point, and the first straight line; the wheel ground point position is determined based on the first ground point and the second ground point.

[0264] In some optional embodiments, the seventh processing unit 537 is further configured to:

[0265] In response to the failure to determine the second grounding point, the entire vehicle frame information is projected onto the vehicle coordinate system to obtain the first front bumper projection point and the first grounding point projection point in the vehicle coordinate system. Based on the first front bumper projection point, the first grounding point projection point, and the predicted vehicle body width, a first position reference point is determined. The first position reference point and the first grounding point projection point are used together to determine the heading angle and position of the target vehicle.

[0266] In some optional embodiments, the detection frame information includes wheel frame information and side line information. The third processing module 53 includes:

[0267] The eighth processing unit 53a is used to determine the second wheel frame segmentation point set corresponding to each wheel based on wheel frame information and semantic segmentation information in response to the target observation type being the third type.

[0268] The ninth processing unit 53b is used to determine the wheel contact point position of the target vehicle based on the second wheel frame segmentation point set and side line information corresponding to each wheel.

[0269] The beneficial technical effects corresponding to the exemplary embodiments of this device can be found in the corresponding beneficial technical effects of the exemplary method section above, and will not be repeated here.

[0270] Exemplary electronic devices

[0271] Figure 16 This is a structural diagram of an electronic device provided in an embodiment of the present disclosure, including at least one processor 11 and a memory 12.

[0272] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0273] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute one or more computer program instructions to implement the methods and / or other desired functions of the various embodiments of this disclosure described above.

[0274] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0275] The input device 13 may also include, for example, a keyboard, a mouse, etc.

[0276] The output device 14 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0277] Of course, for the sake of simplicity, Figure 16 Only some of the components of the electronic device 10 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0278] Exemplary computer program products and computer-readable storage media

[0279] In addition to the methods and apparatus described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods in the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0280] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0281] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the methods in the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0282] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0283] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0284] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A method for determining the contact point of a wheel, comprising: Based on the image to be processed, the detection box information and semantic segmentation information of the target vehicle in the image pixel coordinate system are determined. The detection box information includes at least one of the following: the full vehicle frame information, the front and rear frame information, the side line information, and the wheel frame information of the target vehicle. Based on the detection frame information, the target observation type of the target vehicle is determined; Based on the detection box information and the semantic segmentation information, the wheel contact point position of the target vehicle is determined by a determination method corresponding to the target observation type. The wheel contact point position is the position in the vehicle coordinate system. The step of determining the wheel contact point location of the target vehicle based on the detection box information and the semantic segmentation information, using a determination method corresponding to the target observation type, includes: In response to the target observation type being either type one or type two, based on the side line information and the full vehicle frame information, it is determined whether the target vehicle is observed truncated; the observation truncation indicates that the complete target vehicle is not observed in the image to be processed; in response to the target vehicle observation truncation, based on the positional relationship between the full vehicle frame information and the image to be processed, the target truncation direction of the target vehicle is determined, the target truncation direction representing the direction of the observation truncation of the target vehicle; based on the full vehicle frame information and the semantic segmentation information, a grounding point search is performed using a target search direction corresponding to the target truncation direction to determine the wheel grounding point position of the target vehicle, the target search direction representing the search from the direction opposite to the target truncation direction towards the target truncation direction; or, In response to the target observation type being the third type, based on the wheel frame information and the semantic segmentation information, a second wheel frame segmentation point set corresponding to each wheel is determined; based on the second wheel frame segmentation point set corresponding to each wheel and the side line information, the wheel contact point position of the target vehicle is determined.

2. The method according to claim 1, wherein, The step of determining the wheel contact point location of the target vehicle by searching for the ground point using a target search direction corresponding to the target truncation direction based on the full vehicle frame information and the semantic segmentation information includes: Based on the full vehicle frame information and the target search direction, a first search range is determined; Based on the first search range, determine each first segmentation point in the semantic segmentation information that belongs to the first search range; According to the target search direction, each of the first segmentation points within the first search range is searched to determine the grounding points of each target wheel in the image pixel coordinate system of the target vehicle. The contact points of each target wheel are transformed into the vehicle coordinate system to obtain the position of the wheel contact point.

3. The method according to claim 2, wherein, The step of determining the first search range based on the full vehicle frame information and the target search direction includes: In response to the fact that the target vehicle is a large vehicle, a first search boundary is determined based on the target search direction and the full vehicle frame information; Based on the head and tail frame information, the full vehicle frame information, and the first boundary mapping rule, the second search boundary is determined; or... In response to the head and tail frame information indicating that no head or tail frame was detected, the second search boundary is determined based on the side line information, the full vehicle frame information, and the second boundary mapping rule; The first search range is determined based on the first search boundary and the second search boundary.

4. The method according to claim 1, wherein, The wheel frame information includes first frame information corresponding to at least one wheel; The step of determining the wheel contact point location of the target vehicle based on the detection box information and the semantic segmentation information, using a determination method corresponding to the target observation type, further includes: In response to the target observation type being the first type or the second type, and the target vehicle not being an observation truncation, a first wheel frame segmentation point set corresponding to each wheel is determined based on the wheel frame information and the semantic segmentation information of the target vehicle; For any of the wheels, the first wheel frame segmentation point set is used to determine the first boundary point, the second boundary point, and the first intermediate point. Based on the first boundary point, the second boundary point, and the first intermediate point, determine the target ground contact point of the wheel; Based on the target grounding point corresponding to each wheel, the position of the wheel grounding point of the target vehicle is determined.

5. The method according to claim 4, wherein, Determining the target ground contact point of the wheel based on the first boundary point, the second boundary point, and the first intermediate point includes: Starting from the first boundary point, determine the second distance between any two adjacent wheel frame segmentation points in the first wheel frame segmentation point set; Based on each of the second distances, determine the first starting segmentation point and the first ending segmentation point that satisfy the first condition; Based on the first starting segmentation point and the first ending segmentation point, determine the target ground contact point of the wheel; or, In response to the absence of a first starting segmentation point and a first ending segmentation point that satisfy the first condition, the wheel frame segmentation point closest to the bottom edge of the wheel frame in the first wheel frame segmentation point set is taken as the target grounding point of the wheel.

6. The method according to claim 5, further comprising: In response to the failure to determine the target grounding point of the wheel or the ratio of the determined target grounding point to the third distance of the bottom edge of the wheel frame to the height of the wheel frame being greater than a first threshold, the midpoint of the bottom edge of the wheel frame is taken as the target grounding point of the wheel.

7. The method according to claim 4, wherein, Determining the wheel contact point location of the target vehicle based on the target contact point corresponding to each wheel includes: In response to detecting that the wheel frame information includes first frame information corresponding to multiple front wheels and first frame information corresponding to multiple rear wheels, a target front wheel is determined from the multiple front wheels and a target rear wheel is determined from the multiple rear wheels based on the second slope corresponding to the side line information; The target ground contact points of the front and rear wheels are transformed into the vehicle coordinate system to obtain the wheel ground contact point positions.

8. The method according to claim 1, wherein, Determining the target observation type of the target vehicle based on the detection frame information includes: In response to the detection frame information including full vehicle frame information, front and rear frame information and side line information, and the front and rear frame information being located on the first side of the full vehicle frame information, the target observation type is determined to be the first type; In response to the detection frame information including the full vehicle frame information, the front and rear frame information and the side line information, and the front and rear frame information being located on the second side of the full vehicle frame information, the target observation type is determined to be the second type; In response to the detection frame information including the full vehicle frame information, wheel frame information and side line information, but excluding the front and rear frame information, the target observation type is determined to be the third type.

9. A device for determining the contact position of a wheel, comprising: The first processing module is used to determine the detection box information and semantic segmentation information of the target vehicle in the image pixel coordinate system based on the image to be processed. The detection box information includes at least one of the following: the full vehicle frame information, the front and rear frame information, the side line information, and the wheel frame information of the target vehicle. The second processing module is used to determine the target observation type of the target vehicle based on the detection frame information; The third processing module is used to determine the wheel contact point position of the target vehicle based on the detection box information and the semantic segmentation information, using a determination method corresponding to the target observation type. The wheel contact point position is the position in the vehicle coordinate system. The third processing module includes: A first processing unit is configured to, in response to the target observation type being either type one or type two, determine whether the target vehicle is observed truncated based on the side line information and the full vehicle frame information; the observation truncation indicates that the complete target vehicle is not observed in the image to be processed; a second processing unit is configured to, in response to the target vehicle observation truncation, determine the target truncation direction of the target vehicle based on the positional relationship between the full vehicle frame information and the image to be processed, the target truncation direction representing the direction of the observation truncation of the target vehicle; a third processing unit is configured to, based on the full vehicle frame information and the semantic segmentation information, perform a grounding point search using a target search direction corresponding to the target truncation direction, to determine the wheel grounding point position of the target vehicle, the target search direction representing a search from a direction opposite to the target truncation direction towards the target truncation direction; or, The third processing module includes: The eighth processing unit is configured to, in response to the target observation type being the third type, determine the second wheel frame segmentation point set corresponding to each wheel based on the wheel frame information and the semantic segmentation information; the ninth processing unit is configured to, based on the second wheel frame segmentation point set corresponding to each wheel and the side line information, determine the wheel contact point position of the target vehicle.

10. A computer-readable storage medium storing a computer program for performing the method for determining the wheel contact position according to any one of claims 1-8.

11. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method for determining the wheel ground contact position as described in any one of claims 1-8.

Images