Vehicle orientation detection method and device, electronic equipment, medium, product and vehicle

Abstract

The invention relates to a vehicle orientation detection method and device, electronic equipment, a medium, a product and a vehicle. Key position information of a target vehicle is determined according to a vehicle image containing the target vehicle; wherein the key position information at least comprises position information of a license plate of the target vehicle; and determining orientation information of the vehicle according to the key position information. The vehicle orientation detection accuracy can be improved.

Description

Technical Field

[0001] This application relates to the field of vehicle orientation detection technology, and in particular to a vehicle orientation detection method, device, electronic equipment, medium, product and vehicle. Background Technology

[0002] Current vehicle orientation detection methods generally use wheel positions to identify vehicle orientation.

[0003] However, due to the complexity of driving scenarios, the vehicle cannot accurately determine the vehicle's orientation by relying solely on wheel detection. For example, when the front or rear of the vehicle in front is directly facing the vehicle, the vehicle cannot detect the wheels of the vehicle in front, which affects the recognition of the vehicle's orientation. Summary of the Invention

[0004] This application provides a vehicle orientation detection method, apparatus, electronic device, storage medium, product, and vehicle to improve the accuracy of vehicle orientation detection and at least partially solve the above-mentioned technical problems.

[0005] To achieve the above objectives, according to a first aspect of this application, a vehicle orientation detection method is provided, comprising:

[0006] The key location information of the target vehicle is determined based on a vehicle image containing the target vehicle; wherein the key location information includes at least the location information of the target vehicle's license plate.

[0007] Based on the key location information, the vehicle's orientation information is determined.

[0008] Optionally, determining the key location information of the target vehicle based on a vehicle image containing the target vehicle includes:

[0009] Obtain an encoded image of the vehicle image, wherein the encoded image is obtained by segmenting a vehicle image containing the target vehicle, and the pixel values ​​of the encoded image are used to indicate the category identifier corresponding to each pixel in the vehicle image, wherein the category identifier includes at least one of the following: rear of vehicle, front of vehicle, wheels, and license plate.

[0010] Edge detection is performed on the encoded image to obtain the key location information.

[0011] Optionally, the step of performing edge detection on the encoded image to obtain the key location information includes:

[0012] Edge detection is performed on the encoded image to obtain the edge contour of the license plate;

[0013] Polygon approximation processing is performed based on the pixel coordinates of the edge contour of the license plate, and the position information of the license plate is obtained based on the processing result.

[0014] Optionally, the step of performing polygon approximation processing based on the pixel coordinates of the license plate's edge contour, and obtaining the license plate's position information based on the processing result, includes:

[0015] The edge contour of the license plate is subjected to polygon approximation processing to obtain a geometric license plate image of the edge contour;

[0016] A perspective transformation is performed on the geometric license plate image to obtain the position information of the license plate corner points.

[0017] Optionally, determining the vehicle's orientation information based on the key location information includes:

[0018] The plane where the license plate is located is determined based on the key location information;

[0019] The vehicle's orientation information is determined based on the direction information corresponding to the normal vector of the plane.

[0020] Optionally, determining the vehicle's orientation information based on the direction information corresponding to the normal vector of the plane includes:

[0021] Based on the direction information corresponding to the normal vector of the plane and the category identifier in the encoded image used to indicate the vehicle's orientation, the vehicle's orientation information is determined, wherein the category identifier includes either the front or the rear of the vehicle.

[0022] Optionally, the key location information may also include: the contact point location information of the two wheels on the same side of the vehicle;

[0023] The step of performing edge detection on the encoded image to obtain the key location information further includes:

[0024] Edge detection is performed on the encoded image to obtain the edge contours of the two wheels on the same side;

[0025] The geometric center of the edge contour is obtained based on the pixel coordinates on the edge contour.

[0026] Based on the geometric center of the profile, obtain the contact point location information of the two wheels on the same side of the vehicle.

[0027] Optionally, determining the vehicle's orientation information based on the key location information includes:

[0028] The direction of the target vehicle is determined based on the contact point location information of the two wheels on the same side;

[0029] The vehicle's orientation information is obtained based on the direction and a category identifier in the vehicle image used to indicate the vehicle's orientation, wherein the category identifier includes either the front or rear of the vehicle.

[0030] Optionally, acquiring the encoded image of the vehicle image includes:

[0031] Based on the vehicle image, the coded image of the target vehicle is obtained using a preset vehicle segmentation model.

[0032] Optionally, obtaining the coded image of the target vehicle based on the vehicle image using a preset vehicle segmentation model includes:

[0033] The vehicle image is used as input, and feature extraction is performed through the feature extraction network in the vehicle segmentation model to obtain a feature image.

[0034] The feature image is classified using the convolutional layer in the vehicle segmentation model to obtain a category prediction image;

[0035] The pixels in the category prediction image are classified using the activation function in the vehicle segmentation model to obtain the encoded image of the target vehicle.

[0036] Optionally, before classifying the pixels in the category prediction image using the activation function in the vehicle segmentation model to obtain the encoded image of the target vehicle, the method further includes:

[0037] The category prediction image is upsampled using the deconvolution layer in the vehicle segmentation model.

[0038] Optionally, before acquiring the encoded image of the vehicle image, the method further includes:

[0039] The vehicle image is subjected to distortion correction processing to obtain a corrected vehicle image;

[0040] Based on the standard resolution of the image corresponding to the vehicle segmentation model, the resolution of the vehicle image after distortion correction is adjusted to obtain the processed vehicle image.

[0041] Optionally, the distortion correction processing of the vehicle image includes:

[0042] Using a distortion correction model, the radial distortion of each pixel in the vehicle image is corrected based on the calibration parameters of the target camera that captured the vehicle image, resulting in a corrected vehicle image.

[0043] Optionally, the training steps of the vehicle segmentation model include:

[0044] Obtain vehicle image samples, wherein the vehicle image samples are labeled with vehicle orientation categories, and the vehicle orientation categories include at least one of the following: rear of vehicle, front of vehicle, wheels, and license plate;

[0045] The vehicle segmentation model is trained based on the vehicle image samples to obtain the trained vehicle segmentation model.

[0046] Optionally, the annotation step of the vehicle image sample includes:

[0047] Obtain the angle between the marked vehicle and the corresponding target camera in the vehicle image sample;

[0048] Based on the included angle, the labeling information corresponding to the labeled vehicle is obtained, wherein the labeling information includes at least one of the following: rear labeling information, front labeling information, wheel labeling information, or license plate labeling information.

[0049] Optionally, determining the vehicle's orientation information based on the key location information includes:

[0050] The key location information is transformed from the pixel coordinate system to the vehicle coordinate system.

[0051] The vehicle's orientation information is determined based on key position information in the vehicle coordinate system.

[0052] Optionally, the step of performing coordinate system transformation on the key location information to transform the key location information from the pixel coordinate system to the vehicle coordinate system includes:

[0053] Based on the depth scale corresponding to the target camera that captured the vehicle image and the key position information in the pixel coordinate system, the first position information of the pixel in the vehicle image in the calibration coordinate system is obtained.

[0054] The first position information is back-projected based on the extrinsic parameter matrix of the target camera to obtain the second position information in the camera coordinate system.

[0055] The second position information is projected to obtain the key position information in the vehicle coordinate system.

[0056] Optionally, the method further includes:

[0057] The interactive interface displays at least one target vehicle and its corresponding orientation information.

[0058] Optionally, the interactive interface may also display at least one of the following: the location information of the target vehicle, the location information of the target vehicle's license plate, and the location information of the wheel contact points.

[0059] Optionally, the vehicle images include images captured by at least one fisheye camera on the vehicle.

[0060] According to a second aspect of this application, a vehicle orientation detection device is provided, comprising:

[0061] The first determining module is used to determine key location information of the target vehicle based on a vehicle image containing the target vehicle; wherein the key location information includes at least the location information of the license plate of the target vehicle;

[0062] The second determining module is used to determine the vehicle's orientation information based on the key location information.

[0063] Thirdly, this embodiment also provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the above-described method.

[0064] Fourthly, this embodiment also provides a computer-readable storage medium including a computer program, which, when run on an electronic device, causes the electronic device to perform the steps of the above-described method.

[0065] Fifthly, this embodiment also provides a computer program product, including a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of the above method.

[0066] Sixthly, this embodiment also provides a vehicle, which is equipped with at least one of the above-described vehicle orientation detection device, electronic device, computer-readable storage medium, and computer program product.

[0067] Seventhly, this embodiment also provides a vehicle, including:

[0068] Body;

[0069] A camera module is installed on the vehicle body, and the camera module is used to acquire vehicle images containing the target vehicle;

[0070] A control module is used to determine the vehicle's orientation information according to the aforementioned method.

[0071] Optionally, there may be multiple camera modules, with at least two camera modules disposed on different sides of the vehicle body.

[0072] Optionally, the camera module is a fisheye camera.

[0073] In summary, through the above technical solution, the vehicle can determine the location information of the target vehicle's license plate using a vehicle image containing the target vehicle, and then determine the vehicle's orientation information based on the license plate location information. Thus, using this vehicle orientation detection method, the present application can accurately detect any vehicle orientation, is universally applicable to various driving scenarios, and effectively improves vehicle detection performance.

[0074] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0075] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0076] To gain a more complete understanding of this application and its beneficial effects, the following description will be provided in conjunction with the accompanying drawings, wherein the same reference numerals in the following description denote the same parts.

[0077] Figure 1 This is the first schematic diagram of the relevant technology;

[0078] Figure 2 This is a second schematic diagram of the related technology;

[0079] Figure 3 This is a third schematic diagram related to the technology;

[0080] Figure 4 This is a schematic diagram of the first process for vehicle orientation detection provided in an exemplary embodiment of this application;

[0081] Figure 5 This is a schematic diagram of vehicle image sample calibration provided in an exemplary embodiment of this application;

[0082] Figure 6 This is a schematic diagram of the vehicle orientation visualization process provided in an exemplary embodiment of this application;

[0083] Figure 7 This is a schematic diagram of the coordinate system provided in an exemplary embodiment of this application;

[0084] Figure 8 This is a visual schematic diagram of the vehicle orientation of the interactive interface provided in an exemplary embodiment of this application;

[0085] Figure 9 This is a schematic diagram of the second process for vehicle orientation detection provided in an exemplary embodiment of this application;

[0086] Figure 10 This is a schematic diagram of a vehicle orientation detection device provided in an exemplary embodiment of this application;

[0087] Figure 11 This is a schematic diagram of the architecture of an electronic device provided in an exemplary embodiment of this application. Detailed Implementation

[0088] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0089] In related technologies, such as Figure 1 As shown, the method inputs a real-time RGB image of the preceding vehicle into a multi-level convolutional neural network, outputting the vehicle's position, vehicle type, wheel positions, and wheel contact point positions. The vehicle's orientation angle is then determined based on the wheel contact point positions. However, this method only predicts the vehicle's orientation from the onboard front-facing camera using the wheel contact points. When the front or rear of the vehicle is directly facing the camera, or when only one wheel can be detected, this method cannot determine the vehicle's orientation. Furthermore, relying solely on the contact points of two wheels cannot accurately predict whether the vehicle is facing forward or backward. Finally, this method relies on a deep neural network model for vehicle orientation detection, and the model's inference accuracy is limited by factors such as the distribution of data in various scenarios, the data scale, and the quality of the annotations.

[0090] In addition, such as Figure 2 As shown, in other related technologies, a target orientation detection method based on deep learning is used to input test data into the target orientation model to obtain orientation information. Although this method does not require obtaining the wheel contact point, it requires labeling the tilt parameters between the target and the custom reference direction during the labeling process. This results in problems such as high labeling difficulty and complexity, and the inability to guarantee labeling accuracy.

[0091] For example Figure 3 As shown, related technologies can also collect pose image data of a vehicle's 3D model at specific angles as a pose database. The target image is then input into a matching network to match the data in the pose database, and the result with the highest matching score is used as the predicted vehicle orientation angle. The problem with this method is that it can only output ten pre-set vehicle orientation angles. If the actual vehicle angle does not match any of the preset angles, the prediction result will have a large error.

[0092] In summary, existing vehicle orientation prediction methods suffer from several problems, including high labeling difficulty, complex networks leading to poor training results, and inability to accurately obtain the vehicle's frontal orientation. These problems collectively limit the effectiveness and reliability of these methods in practical applications, severely affecting the accuracy of vehicle orientation prediction and reducing vehicle performance.

[0093] To address the aforementioned problems, this application proposes a vehicle orientation detection method, apparatus, electronic device, computer-readable storage medium, computer program product, and vehicle.

[0094] In one embodiment, the vehicle orientation detection method of this application can be applied to a vehicle, which may be a gasoline-powered vehicle, a plug-in hybrid electric vehicle, or a new energy vehicle, etc. This application does not specifically limit the type of vehicle. Figure 4 As shown, the method may specifically include:

[0095] S10, determine the key location information of the target vehicle based on the vehicle image containing the target vehicle; wherein, the key location information includes at least the location information of the license plate of the target vehicle;

[0096] In this embodiment, the vehicle performing the vehicle orientation detection method can be the vehicle itself, and the target vehicle can be an adjacent vehicle detected by the vehicle itself. For example, the target vehicle can be the vehicle in front of the vehicle detected by the vehicle itself (i.e., the vehicle in front).

[0097] In this embodiment, the vehicle can use an onboard fisheye camera to capture vehicle images containing the target vehicle.

[0098] Furthermore, after acquiring a vehicle image containing the target vehicle, the vehicle can obtain key location information of the target vehicle based on the vehicle image.

[0099] The aforementioned key location information may include at least the location information of the license plate.

[0100] In addition, the license plate location information may also include the vehicle's grounding point location information.

[0101] S20, determine the vehicle's orientation information based on the key location information.

[0102] In this embodiment, the vehicle can determine its orientation information based on the detected license plate location information.

[0103] For example, in this embodiment, the vehicle can use the normal vector of the plane where the license plate is located as the direction of the vehicle, and then combine it with the category identifier corresponding to the vehicle image to determine the vehicle's orientation information. The category identifier can include any one of the following: the rear of the vehicle, the front of the vehicle, the wheels, and the license plate.

[0104] Therefore, in this embodiment, the vehicle can determine the location information of the target vehicle's license plate using a vehicle image containing the target vehicle, and then determine the vehicle's orientation information based on the license plate location information. Thus, using this vehicle orientation detection method, this application can accurately detect any vehicle orientation, is universally applicable to various driving scenarios, and effectively improves vehicle detection performance.

[0105] In one embodiment, S10 above, "determining the key location information of the target vehicle based on the vehicle image containing the target vehicle," may include:

[0106] S101, Obtain the encoded image of the vehicle image, wherein the encoded image is obtained by segmenting the vehicle image containing the target vehicle, and the pixel values ​​of the encoded image are used to indicate the category identifier corresponding to each pixel in the vehicle image, wherein the category identifier includes at least one of the following: rear of vehicle, front of vehicle, wheels, and license plate.

[0107] S102, perform edge detection on the encoded image to obtain the key location information.

[0108] In this embodiment, the vehicle can use a fisheye camera to capture a vehicle image containing the target vehicle, and then segment the vehicle image containing the target vehicle to obtain the coded image of the target vehicle.

[0109] In this embodiment, the encoded image can be a multi-category label encoded image. The pixel values ​​of the pixels in the encoded image can be used to indicate the category identifier of the vehicle orientation category corresponding to each pixel in the vehicle image. The category identifier can include at least one of the following: rear of the vehicle, front of the vehicle, wheels, and license plate.

[0110] Understandably, the target vehicle's orientation can be the direction of the target vehicle relative to the camera on the vehicle that captures its image. For example, if the angle between the rear of the target vehicle and the central axis of the camera is an acute angle, then the target vehicle's orientation category can be determined as the rear of the vehicle; otherwise, the target vehicle's orientation category is the front of the vehicle.

[0111] In one specific embodiment, the encoded image can be a single-channel image mask of size H x W, where the value of each pixel is a category identifier, for example, 0 represents the background, 1 represents the rear of the vehicle, 2 represents the front of the vehicle, etc. In this way, the vehicle can obtain key location information of the target vehicle based on the encoded image.

[0112] Furthermore, the vehicle can perform edge detection on the coded image to obtain key location information of the target vehicle.

[0113] In this embodiment, the key location information may include the location information of the wheel contact point and / or the location information of the license plate. Specifically, the location information of the license plate may be the location information of the corner point of the license plate.

[0114] For example, the vehicle can perform operations such as dilation and edge processing on the encoded image to extract the location information of the wheel grounding point and / or the location information of the license plate in the processed encoded image.

[0115] It is understandable that, considering the different orientations of the target vehicle compared to the vehicle itself, the vehicle itself may not be able to detect the wheels on the same side of the target vehicle at the same time, but may only be able to detect the license plate. Therefore, this embodiment can obtain the location information of the wheel contact point and / or the location information of the license plate. In addition, the key location information may also include the roof outline corner point, the bottom outline corner point, etc.

[0116] Furthermore, the vehicle can determine its orientation information based on the key location information and coded images detected above.

[0117] Compared to existing technologies that determine vehicle orientation solely based on wheels, this application can also determine vehicle orientation by utilizing the license plate position and combining it with coded images. Using this vehicle orientation detection method, this application can accurately detect any vehicle orientation, making it universally applicable to various driving scenarios and effectively improving vehicle performance.

[0118] In one embodiment, S102 above, "performing edge detection on the encoded image to obtain the key location information" may include:

[0119] S1021, perform edge detection on the encoded image to obtain the edge contour of the license plate;

[0120] S1022, perform polygon approximation processing based on the pixel coordinates of the edge contour of the license plate, and obtain the position information of the license plate based on the processing result.

[0121] It should be noted that, in this embodiment, to calculate the vehicle orientation more accurately, the encoded image needs to be post-processed to extract the positional information of the wheel contact points and the four corner points of the license plate. This information is then combined with the target segmentation category information of the vehicle orientation target segmentation in the encoded image to obtain the orientation information of the target vehicle. Therefore, the edge contour of the target vehicle can be either the edge contour of the wheels or the edge contour of the license plate.

[0122] The vehicle can use an edge detection algorithm based on image intensity gradient to extract the edge contours of the wheels and license plate by finding connected regions in the encoded image.

[0123] Furthermore, the vehicle can obtain key location information of the target vehicle based on the pixel coordinates on the edge contour.

[0124] In another embodiment, before performing edge detection on the encoded image, the vehicle can also perform image enhancement processing on the encoded image, specifically a dilation operation.

[0125] Thus, in this embodiment, the vehicle can read the coded image and use the dilation operation in morphology to enhance the connectivity and robustness of the wheel contour, so as to perform edge detection on the coded image after the dilation operation.

[0126] In one specific embodiment, the above-mentioned "performing polygon approximation processing based on the pixel coordinates of the edge contour of the license plate, and obtaining the position information of the license plate based on the processing result" may include:

[0127] The edge contour of the license plate is subjected to polygon approximation processing to obtain a geometric license plate image of the edge contour;

[0128] A perspective transformation is performed on the geometric license plate image to obtain the position information of the license plate corner points.

[0129] In this embodiment, when the edge contour is the edge contour of the license plate, the vehicle can perform polygon approximation processing based on the pixel coordinates of the edge contour of the license plate to approximate the edge contour of the license plate, obtain an approximate geometric shape representation, and obtain the position information of the license plate corner points in the key position information based on the processing result.

[0130] Specifically, for example, in this embodiment, the vehicle can use a polygon approximation algorithm to approximate the edge contour of the license plate to obtain an approximate geometric shape representation. Then, the license plate can be further corrected through perspective transformation to correct the tilt and distortion of the license plate. The perspective transformation formula used is as follows:

[0131]

[0132] Where (x,y) are the coordinates of a point in the original image, (x′,y′) are the coordinates of a point in the image after perspective transformation, and perspective_matrix is ​​the perspective transformation matrix.

[0133] Thus, this embodiment extracts the position information of the wheel ground contact point and the four corner points of the license plate by post-processing the encoded image, which is used for subsequent more accurate vehicle orientation calculation.

[0134] In one embodiment, when the key location information further includes the contact point location information of the two wheels on the same side of the vehicle, the step S102 above, "performing edge detection on the encoded image to obtain the key location information," may further include:

[0135] S1023, perform edge detection on the encoded image to obtain the edge contours of the two wheels on the same side;

[0136] S1024, Obtain the geometric center of the edge contour based on the pixel coordinates on the edge contour;

[0137] S1025, Based on the geometric center of the contour, obtain the contact point position information of the two wheels on the same side of the vehicle.

[0138] In this embodiment, when the edge contour is the edge contour of a wheel, the vehicle can determine the geometric center of the wheel contour based on the pixel coordinates on the contour using the centroid calculation formula, as follows:

[0139]

[0140] Where (xc, yc) are the centroid coordinates of the wheel profile (i.e., the geometric center of the wheel profile in this embodiment), (xi, yi) are the pixel coordinates on the wheel profile, and n is the number of pixels on the wheel profile.

[0141] Furthermore, based on the geometric center of the contour, the position information of the wheel contact point of the target vehicle can be calculated according to the tangent point between the wheel and the ground, which will not be elaborated here.

[0142] In one embodiment, in S20 above, "determining the vehicle's orientation information based on the key location information" may include:

[0143] S201, determine the plane where the license plate is located based on the key location information;

[0144] S202, Based on the direction information corresponding to the normal vector of the plane, determine the orientation information of the vehicle.

[0145] It should be noted that in this embodiment, after obtaining the key target information in the vehicle coordinate system, the vehicle orientation can be further determined. In this embodiment, it can be assumed that the vehicle is not obstructed by other obstacles, and the following situations exist: 1. The front of the vehicle is facing the camera, and the ground contact points of the front and rear wheels on the same side cannot be detected; 2. The rear of the vehicle is facing the camera, and the ground contact points of the front and rear wheels on the same side cannot be detected simultaneously; 3. The vehicle body is facing the camera, and the ground contact points of the front and rear wheels on the same side can be detected.

[0146] Based on this, in this embodiment, when the front or rear of the vehicle is facing the camera and the locations of the front and rear wheels on the same side cannot be detected, the vehicle can obtain the normal vector of the license plate plane based on the position information of the license plate corner point in the target key position, and then combine it with the target segmentation category information of the vehicle's orientation to obtain the orientation information of the current vehicle relative to the camera in the vehicle coordinate system.

[0147] In one specific embodiment, the above-mentioned "determining the vehicle's orientation information based on the direction information corresponding to the normal vector of the plane" may include:

[0148] Based on the direction information corresponding to the normal vector of the plane and the category identifier in the encoded image used to indicate the vehicle's orientation, the vehicle's orientation information is determined, wherein the category identifier includes either the front or the rear of the vehicle.

[0149] In this embodiment, if the wheels are obscured, the normal vector of the license plate plane can be obtained from the four corner points of the license plate as the direction of the vehicle. Combined with the category identifier of the front / rear of the vehicle, it can be determined whether the vehicle is facing forward or backward, thereby obtaining the accurate orientation of the vehicle.

[0150] For example, the normal vector of the license plate plane can be used as the direction of the vehicle. When the class is identified as the rear of the vehicle, it can be determined that the target vehicle is actually facing from the rear to the front (i.e., the rear of the target vehicle is closer to the vehicle itself). Similarly, when the class is identified as the front of the vehicle, it can be determined that the target vehicle is actually facing from the front to the rear (i.e., the front of the target vehicle is closer to the vehicle itself).

[0151] Therefore, even if the wheel position is not detected, the orientation information of the target vehicle can be identified by combining the license plate in this embodiment. Thus, this application can accurately detect any orientation of the vehicle and is generally applicable to various types of driving scenarios.

[0152] In one embodiment, S20 above, "determining the vehicle's orientation information based on the key location information," may further include:

[0153] S203, determine the direction of the target vehicle based on the contact point position information of the two wheels on the same side;

[0154] S204, Obtain the vehicle's orientation information based on the direction and the category identifier in the vehicle image used to indicate the vehicle's orientation.

[0155] In this embodiment, in conjunction with the above embodiments, when the vehicle body is facing the camera, the wheel contact points of the front and rear wheels on the same side can be detected. The vehicle can directly output the position information of the wheel contact points in the vehicle coordinate system. Combined with the target segmentation category information of the vehicle's orientation at this time, the orientation information of the current vehicle relative to the camera in the vehicle coordinate system can be obtained.

[0156] For example, the line connecting the grounding points of two vehicles on the same side can be used as the direction of the target vehicle. When the category is identified as the rear of the vehicle, it can be determined that the target vehicle is actually facing forward and backward (i.e., the rear of the target vehicle is closer to the vehicle itself). Similarly, when the category is identified as the front of the vehicle, it can be determined that the target vehicle is actually facing backward and backward (i.e., the front of the target vehicle is closer to the vehicle itself).

[0157] Thus, this application can determine the vehicle's orientation by using the vehicle's ground contact point location, and when the wheels cannot be detected, it can also determine the vehicle's orientation by using the license plate location. Using this vehicle orientation detection method, this application can accurately detect any orientation of the vehicle, and is generally applicable to various types of driving scenarios, thereby effectively improving vehicle performance.

[0158] In one embodiment, in S101 above, "acquiring the encoded image of the vehicle image" may include:

[0159] S1011, Based on the vehicle image, obtain the coded image of the target vehicle using a preset vehicle segmentation model.

[0160] In this embodiment, the mainstream network frameworks used for semantic segmentation include U-Net, SegNet, DeepLab series, FCN (Fully Convolutional Network), PSPNet (Pyramid Scene Parsing Network), and ENet. This embodiment uses FCN as an example to build a vehicle segmentation model and train it, enabling the trained vehicle segmentation model to output an encoded image based on the input vehicle image.

[0161] Specifically, for example:

[0162] (1) Before building the FCN model, it is necessary to first determine the basic network structure suitable for a specific task, such as VGG, ResNet, etc. Here, ResNet is selected as the feature extraction layer.

[0163] (2) Replace fully connected layers with fully convolutional layers, and convert the last few fully connected layers of the network into 1x1 convolutional layers to preserve spatial dimensions and allow the output to be an image-level prediction;

[0164] (3) Add deconvolution or upsampling layers. In the decoder part of the FCN model, add deconvolution or upsampling layers to upsample the low-resolution feature maps in the encoder to the same resolution as the input image.

[0165] (4) Add skip connections to make full use of feature information at different levels. Add skip connections to connect the feature maps in the encoder with the corresponding hierarchical feature maps in the decoder to improve the semantic understanding ability and segmentation performance of the model.

[0166] (5) Select a loss function. Commonly used loss functions include cross-entropy loss and pixel-level loss functions (such as Dice Loss, Jaccard Loss, etc.). In this embodiment, cross-entropy loss can be selected.

[0167] (6) Parameter initialization: Initialize the parameters using the pre-trained model to accelerate model convergence and improve training performance;

[0168] (7) Model compilation and optimizer selection. Commonly used optimizers include stochastic gradient descent (SGD), Adam, RMSprop, etc. In this embodiment, Adam can be selected as the optimizer.

[0169] (8) Training the model: In this embodiment, the labeled training data can be used to train the constructed FCN model.

[0170] In one embodiment, S1011 above, "obtaining the coded image of the target vehicle based on the vehicle image using a preset vehicle segmentation model," may include:

[0171] The vehicle image is used as input, and feature extraction is performed through the feature extraction network in the vehicle segmentation model to obtain a feature image.

[0172] The feature image is classified using the convolutional layer in the vehicle segmentation model to obtain a category prediction image;

[0173] The category prediction image is upsampled using the deconvolution layer in the vehicle segmentation model.

[0174] The pixels in the category prediction image are classified using the activation function in the vehicle segmentation model to obtain the encoded image of the target vehicle.

[0175] In this embodiment, the input measurement image is processed by the base network (ResNet, i.e. the feature extraction layer in this embodiment) of the vehicle segmentation model to extract features and obtain a feature image.

[0176] Furthermore, the feature images are passed through a 1x1 convolutional layer in the vehicle segmentation model to preserve their spatial structure and perform pixel-level classification.

[0177] Furthermore, in order to restore spatial resolution, deconvolutional layers in the vehicle segmentation model are used to upsample the feature maps to the same size as the input measurement image;

[0178] During the upsampling process, skip connections are used to combine low-level features to enhance the recovery of image details;

[0179] The final output category map uses the softmax activation function to classify pixels and generate a category label for each pixel.

[0180] Specifically, for example:

[0181] (1) Basic Network

[0182] To effectively extract image features, ResNet-50 was chosen as the base network. By introducing a residual structure, the vanishing gradient problem in deep network training was addressed. This network extracts features from the input image layer by layer through multiple convolutional layers, generating multi-level feature maps. The output feature maps of ResNet are then used as input to the FCN network for subsequent semantic segmentation tasks.

[0183] (2) Fully convolutional structure

[0184] Unlike traditional convolutional neural networks, this model replaces fully connected layers with 1x1 convolutional layers, preserving the spatial structure of the feature maps. This allows the network to handle input images of arbitrary sizes, and the final output is a pixel-level class prediction map with the same size as the input image. In this way, each pixel of the network is mapped to a class label, meeting the requirements of semantic segmentation tasks.

[0185] (3) Deconvolution layer

[0186] In the decoder part of the network, the model uses deconvolution layers to progressively restore the spatial resolution of the image. Low-resolution feature maps are upsampled to the resolution of the input image through learning. The deconvolution layer is implemented using transposed convolution, which improves the image's detail recovery while preserving spatial information. The goal of this stage is to generate an output image of the same size as the original input image, where each pixel corresponds to a class label.

[0187] (4) Jump connection

[0188] To fully utilize the low-level features of the encoder and improve segmentation accuracy, the model introduces skip connections. Skip connections help the model maintain high-resolution detail during upsampling by directly passing feature maps from the encoder to the corresponding levels in the decoder, better handling small objects and complex edges in the image, and improving the accuracy of the segmentation results.

[0189] (5) Output layer

[0190] At the very end of the network, a softmax activation function is used to classify each pixel, generating a category probability map for each pixel. The output of this layer is the same size as the input image, and the value of each pixel represents the probability that the location belongs to a specific category (such as "front of vehicle" or "rear of vehicle"). Finally, the model classifies the vehicles based on the highest probability, thus obtaining the complete vehicle segmentation result.

[0191] In one embodiment, before S101, "acquiring the encoded image of the vehicle image", the process may further include:

[0192] S103, Perform distortion correction processing on the vehicle image to obtain a corrected vehicle image;

[0193] S104, adjust the resolution of the vehicle image after distortion correction according to the standard resolution of the image corresponding to the vehicle segmentation model to obtain the processed vehicle image.

[0194] In this embodiment, before inputting the vehicle image into the vehicle segmentation model, distortion correction and image preprocessing can be performed on the vehicle image so that the processed vehicle image can be input into the trained vehicle segmentation model for real-time inference to obtain the target segmentation category information of the vehicle orientation and the position information of the key points of the vehicle orientation.

[0195] For example, the vehicle can perform distortion correction on the vehicle image to obtain a corrected vehicle image. Then, according to actual needs (such as the standard resolution corresponding to the vehicle segmentation model), the corrected vehicle image can be resized to the standard resolution and input into the trained vehicle segmentation model for model inference to obtain the encoded image of the vehicle orientation.

[0196] In a specific embodiment, S103 above, "performing distortion correction processing on the vehicle image to obtain a corrected vehicle image," may include:

[0197] S1031, using a distortion correction model, based on the calibration parameters of the target camera capturing the vehicle image, the radial distortion of each pixel in the vehicle image is corrected to obtain the corrected vehicle image.

[0198] In this embodiment, four fisheye cameras can be fixed on the test vehicle and calibrated to obtain the intrinsic parameters K, distortion coefficients D, extrinsic rotation matrix R, and extrinsic translation matrix T of each camera. Then, OpenCV is used to perform full-image distortion correction on the images acquired by the fisheye cameras. Since the radial distortion of the fisheye camera is much greater than the tangential distortion, the tangential distortion is ignored and only the radial distortion is considered during the distortion correction of the fisheye image. The distortion correction model Kannala-Brant is used to correct the fisheye image pixel by pixel. The distortion correction model Kannala-Brant can use the camera's intrinsic parameters K and distortion coefficients D to obtain the vehicle image after distortion correction.

[0199] Thus, this embodiment improves vehicle image quality by performing distortion correction and resolution adjustment on images acquired by four vehicle-mounted fisheye cameras, thereby enhancing the processing accuracy of the semantic segmentation-based vehicle segmentation model.

[0200] In one embodiment, the training step of the vehicle segmentation model includes:

[0201] S30, acquire vehicle image samples, wherein the vehicle image samples are labeled with vehicle orientation categories, and the vehicle orientation categories include at least one of the following: rear of vehicle, front of vehicle, wheels, and license plate;

[0202] S40, the vehicle segmentation model is trained based on the vehicle image samples to obtain the trained vehicle segmentation model.

[0203] In this embodiment, vehicle image samples within the field of view of the vehicle-mounted camera can be pre-collected, and the vehicle image samples can be labeled using a labeling tool to construct a vehicle semantic segmentation database, which is used to train the semantic segmentation network and obtain a vehicle segmentation model.

[0204] In one specific embodiment, the annotation step of the vehicle image sample includes:

[0205] S50, obtain the angle between the marked vehicle and the corresponding target camera in the vehicle image sample;

[0206] S60, based on the included angle, obtain the labeling information corresponding to the labeled vehicle, wherein the labeling information includes at least one of the following: rear labeling information, front labeling information, wheel labeling information, or license plate labeling information.

[0207] In this embodiment, the vehicle can acquire vehicle image samples within the field of view of the onboard camera and perform data annotation. The specific steps are as follows:

[0208] (1) Based on the task requirements, clarify the scenarios, vehicle types and data volume that need to be collected;

[0209] (2) Driving a vehicle and using an onboard camera to collect data under different driving conditions;

[0210] (3) Filter the collected data and delete duplicate scenes and extremely blurry images;

[0211] (4) Use annotation tools to annotate the selected images. Annotations include multiple categories such as the front and rear of the vehicle, license plates, and wheels. For example, taking the annotation of the front and rear of the vehicle as an example... Figure 5 As shown, using the camera's centerline as a reference, when the angle between the rear of the vehicle and the camera is acute, the entire vehicle is labeled as "CarTail"; otherwise, the entire vehicle is labeled as "CarHead". The wheel and license plate labels are defined by their smallest bounding rectangles. For an example of this labeling method, please refer to [link to documentation]. Figure 5 Examples of vehicle front facing the camera, vehicle body facing the camera, and vehicle rear facing the camera are provided.

[0212] Thus, this embodiment can use labeled vehicle image samples as input and train the vehicle segmentation model using a rectangular bounding box as a prompt. This allows the trained vehicle segmentation model to output accurate encoded images and target segmentation category information of vehicle orientation. The prompt can be a point, a rectangular bounding box, or a prompt word, and the rectangular bounding box contains all pixel information within the bounding box.

[0213] In one embodiment, the above-described S20, "determining the vehicle's orientation information based on the key location information," may further include:

[0214] S205, Perform coordinate system transformation on the key location information to transform the key location information from the pixel coordinate system to the vehicle coordinate system;

[0215] S206, Determine the vehicle's orientation information based on key position information in the vehicle coordinate system.

[0216] In this embodiment, as Figure 6 As shown, the orientation information of the detected target vehicle can be visualized in the vehicle coordinate system. For example, by using the camera's intrinsic and extrinsic parameters, the initial keypoint position information of the vehicle's orientation is transformed into the vehicle coordinate system. Combined with the vehicle's orientation category label, the vehicle's orientation information relative to the camera is obtained, and the final predicted vehicle orientation result is visualized.

[0217] In a specific embodiment, step S205 above, "performing a coordinate system transformation on the key location information to convert the key location information from a pixel coordinate system to a vehicle coordinate system," may include:

[0218] S2051, based on the depth scale corresponding to the target camera that captured the vehicle image and the key position information in the pixel coordinate system, obtain the first position information of the pixel in the vehicle image in the calibration coordinate system;

[0219] S2052, the first position information is back-projected according to the extrinsic parameter matrix of the target camera to obtain the second position information in the camera coordinate system;

[0220] S2053, Project the second position information to obtain the key position information in the vehicle coordinate system.

[0221] It should be noted that in this embodiment, four coordinate systems can be defined: pixel coordinate system, camera coordinate system, vehicle coordinate system, and calibration coordinate system. (Refer to...) Figure 7 Where oxy is the pixel coordinate system, OcXcYcZc is the camera coordinate system, OvXvYvZv is the vehicle coordinate system, and OsXsYsZs is the calibration coordinate system.

[0222] The vehicle coordinate system has its origin at the center of the rear axle. The x-axis points in the forward direction of the vehicle, with forward being positive. The y-axis points in the axle direction, with left being positive. The z-axis points in the vertical direction of the vehicle, with upward being positive, satisfying the right-hand coordinate system. The calibration coordinate system has the same coordinate axes as the vehicle coordinate system, but a different origin. The origin of the calibration coordinate system is located at the center of the four fisheye camera mounting positions. The camera coordinate system has its origin at the optical center of the camera. The z-axis is parallel to the optical axis, pointing forward of the camera, with positive direction. The x-axis points to the right of the camera, with positive direction, perpendicular to the horizontal direction of the camera and parallel to the horizontal direction of the image. The y-axis points downward of the camera, with positive direction, perpendicular to the vertical direction of the camera and parallel to the vertical direction of the image. The pixel coordinate system has its origin at the upper left corner of the image, with the x-axis and y-axis directions consistent with the camera coordinate system.

[0223] Based on this, the keypoint location information in the pixel coordinate system can be used as input. Utilizing the camera's intrinsic and extrinsic parameters, the depth scale Zc in the camera coordinate system is solved during the transformation from the pixel coordinate system to the calibration coordinate system. The formula used is:

[0224]

[0225] Where u and v are the pixel coordinates on the vehicle image after distortion correction in the pixel coordinate system, and Xw, Yw, and Zw are the corresponding three-dimensional coordinates in the calibration coordinate system. Several sets of pixel coordinates (Xw, Yw, Zw) of the ground corresponding to the key point positions are taken, with Zw being 0. Furthermore, T is the extrinsic translation matrix of the fisheye camera, Mc is the intrinsic matrix of the fisheye camera, and R is the extrinsic rotation matrix of the fisheye camera. These three parameters are fixed after camera calibration, so the depth scale Zc can be calculated.

[0226] Furthermore, the calculated depth scale in the camera coordinate system can be substituted into the above formula to obtain the three-dimensional coordinates Xw, Yw, and Zw corresponding to the key position information in the input pixel coordinate system in the calibration coordinate system (i.e., the first position information in this embodiment).

[0227] The three-dimensional coordinates in the calibration coordinate system are back-projected to the camera coordinate system using the camera extrinsic parameters to obtain the three-dimensional coordinates in the corresponding camera coordinate system (i.e., the second position information in this embodiment).

[0228] Then, the 3D coordinate points in the camera coordinate system are projected onto the vehicle coordinate system to obtain the key point location information in the vehicle coordinate system.

[0229] In one embodiment, the vehicle orientation detection method of this application may further include:

[0230] Step A: Display at least one target vehicle and its corresponding orientation information via an interactive interface.

[0231] In this embodiment, the vehicle can use the interactive interface of its in-vehicle display screen to display at least one target vehicle and the orientation information corresponding to the target vehicle.

[0232] Optionally, the interactive interface may also display at least one of the following: the location information of the target vehicle, the location information of the target vehicle's license plate, and the location information of the wheel contact points.

[0233] For example, such as Figure 8 The visualization shown is in a vehicle coordinate system, with the vertical axis being the y-axis (positive direction forward) and the horizontal axis being the x-axis (positive direction left). The unit is mm. The black dots in the coordinate system represent key points of the vehicle's orientation (wheel contact points / license plate corner points). The arrows indicate the final visualization effect of the vehicle's orientation. The vehicle at the origin of the coordinate system is the unicycle. In the corresponding fisheye image, there are 3 vehicles in the front view, 1 vehicle in the left view, 1 vehicle in the rear view, and 1 vehicle in the right view.

[0234] In one embodiment, the vehicle images include images captured by at least one fisheye camera on the vehicle.

[0235] The vehicle image in this embodiment can be an image taken by at least one fisheye camera on the vehicle (i.e., the vehicle itself), as described in the above embodiments, and will not be repeated here.

[0236] In general, such as Figure 9 As shown, this application constructs a vehicle semantic segmentation database by acquiring vehicle images within the field of view of an onboard camera and annotating the images using a labeling tool. A semantic segmentation network is used to train the model, resulting in a vehicle semantic segmentation model. The acquired fisheye images are corrected for distortion and preprocessed before being used as input to the segmentation model for inference, obtaining multi-class label information for target segmentation of vehicle orientation. The model's prediction results are post-processed to output the location information of key points for vehicle orientation. Finally, the camera's intrinsic and extrinsic parameters are used to transform the image into a vehicle coordinate system, and the vehicle's orientation relative to the camera is obtained by combining the target segmentation category information. The final predicted vehicle orientation result is then visualized.

[0237] Thus, (1) this application proposes an automatic vehicle orientation detection scheme based on semantic segmentation within the field of view of a surround-view fisheye camera. By constructing a vehicle segmentation model, the key point location information of vehicle orientation in the collected data is automatically obtained and transformed into the vehicle coordinate system. The prediction results of vehicle orientation within the field of view of four fisheye cameras are visualized in the panoramic image. (2) this application proposes a prediction scheme for any orientation of an unobstructed vehicle. By combining the location information of the key points of vehicle orientation with the target segmentation category of vehicle orientation, the true orientation of the vehicle relative to the camera within the current field of view can be accurately predicted. (3) this application proposes a post-processing scheme for obtaining key point data of vehicle orientation. The location information of the wheel contact point is obtained by using morphological dilation operation and centroid calculation formula. The location information of the license plate corner point is obtained by using edge detection algorithm and perspective transformation, providing reliable input information for vehicle orientation prediction.

[0238] Compared to existing technologies that predict vehicle orientation using classification networks, which often suffer from inaccurate predictions due to network complexity and poor training performance, this application accurately obtains the location information of key vehicle orientation points through semantic segmentation and uses camera intrinsic and extrinsic parameters to transform this information into the vehicle coordinate system for prediction, effectively solving the problem of unsatisfactory prediction results caused by network complexity. Compared to existing technologies that predict vehicle orientation solely based on wheels, this application combines the target segmentation category of vehicle orientation with the relevant location information of key vehicle orientation points, enabling accurate prediction of any orientation of unobstructed vehicles, effectively solving the problem of not being able to determine whether the vehicle is facing forward or backward. Compared to existing technologies that use complex annotations to build vehicle orientation prediction models, this application uses multi-class bounding boxes to annotate vehicles, employs a semantic segmentation model based on prompt information to obtain semantic segmentation annotations, and combines post-processing to determine vehicle orientation, outputting ideal vehicle orientation prediction results and solving the problem of high annotation costs. Compared to existing technologies that predict vehicle orientation through model matching, this application uses semantic segmentation and perspective transformation algorithms to predict the orientation angle of vehicles with arbitrary orientations, solving the problems of large vehicle-model matching errors and poor orientation angle prediction results.

[0239] Accordingly, embodiments of this application also provide a vehicle orientation detection device, such as... Figure 10 As shown, the device may include:

[0240] The first determining module 1001 is used to determine key location information of the target vehicle based on a vehicle image containing the target vehicle; wherein the key location information includes at least the location information of the license plate of the target vehicle.

[0241] The second determining module 1002 is used to determine the orientation information of the vehicle based on the key location information.

[0242] Optionally, the first determining module 1001 is further configured to:

[0243] Obtain an encoded image of the vehicle image, wherein the encoded image is obtained by segmenting a vehicle image containing the target vehicle, and the pixel values ​​of the encoded image are used to indicate the category identifier corresponding to each pixel in the vehicle image, wherein the category identifier includes at least one of the following: rear of vehicle, front of vehicle, wheels, and license plate.

[0244] Edge detection is performed on the encoded image to obtain the key location information.

[0245] Optionally, the first determining module 1001 is also used for:

[0246] Edge detection is performed on the encoded image to obtain the edge contour of the license plate;

[0247] Polygon approximation processing is performed based on the pixel coordinates of the edge contour of the license plate, and the position information of the license plate is obtained based on the processing result.

[0248] Optionally, the first determining module 1001 is also used for:

[0249] The edge contour of the license plate is subjected to polygon approximation processing to obtain a geometric license plate image of the edge contour;

[0250] A perspective transformation is performed on the geometric license plate image to obtain the position information of the license plate corner points.

[0251] Optionally, the second determining module 1002 is further configured to:

[0252] The plane where the license plate is located is determined based on the key location information;

[0253] The vehicle's orientation information is determined based on the direction information corresponding to the normal vector of the plane.

[0254] Optionally, the second determining module 1002 is further configured to:

[0255] Based on the direction information corresponding to the normal vector of the plane and the category identifier in the encoded image used to indicate the vehicle's orientation, the vehicle's orientation information is determined, wherein the category identifier includes either the front or the rear of the vehicle.

[0256] Optionally, the key location information may also include: the contact point location information of the two wheels on the same side of the vehicle;

[0257] The first determining module 1001 is also used for:

[0258] Edge detection is performed on the encoded image to obtain the edge contours of the two wheels on the same side;

[0259] The geometric center of the edge contour is obtained based on the pixel coordinates on the edge contour.

[0260] Based on the geometric center of the profile, obtain the contact point location information of the two wheels on the same side of the vehicle.

[0261] Optionally, the second determining module 1002 is further configured to:

[0262] The direction of the target vehicle is determined based on the contact point location information of the two wheels on the same side;

[0263] The vehicle's orientation information is obtained based on the direction and a category identifier in the vehicle image used to indicate the vehicle's orientation, wherein the category identifier includes either the front or rear of the vehicle.

[0264] Optionally, the first determining module 1001 is further configured to:

[0265] Based on the vehicle image, the coded image of the target vehicle is obtained using a preset vehicle segmentation model.

[0266] Optionally, the first determining module 1001 is further configured to:

[0267] The vehicle image is used as input, and feature extraction is performed through the feature extraction network in the vehicle segmentation model to obtain a feature image.

[0268] The feature image is classified using the convolutional layer in the vehicle segmentation model to obtain a category prediction image;

[0269] The pixels in the category prediction image are classified using the activation function in the vehicle segmentation model to obtain the encoded image of the target vehicle.

[0270] Optionally, the first determining module 1001 is further configured to:

[0271] The category prediction image is upsampled using the deconvolution layer in the vehicle segmentation model.

[0272] Optionally, the first determining module 1001 is further configured to:

[0273] The vehicle image is subjected to distortion correction processing to obtain a corrected vehicle image;

[0274] Based on the standard resolution of the image corresponding to the vehicle segmentation model, the resolution of the vehicle image after distortion correction is adjusted to obtain the processed vehicle image.

[0275] Optionally, the vehicle orientation detection device in this application further includes:

[0276] The distortion correction module is used to correct the radial distortion of each pixel in the vehicle image based on the calibration parameters of the target camera that captured the vehicle image, using a distortion correction model, to obtain a corrected vehicle image.

[0277] Optionally, the training of the vehicle segmentation model includes:

[0278] Obtain vehicle image samples, wherein the vehicle image samples are labeled with vehicle orientation categories, and the vehicle orientation categories include at least one of the following: rear of vehicle, front of vehicle, wheels, and license plate;

[0279] The vehicle segmentation model is trained based on the vehicle image samples to obtain the trained vehicle segmentation model.

[0280] Optionally, the annotation of the vehicle image samples includes:

[0281] Obtain the angle between the marked vehicle and the corresponding target camera in the vehicle image sample;

[0282] Based on the included angle, the labeling information corresponding to the labeled vehicle is obtained, wherein the labeling information includes at least one of the following: rear labeling information, front labeling information, wheel labeling information, or license plate labeling information.

[0283] Optionally, the second determining module 1002 is further configured to:

[0284] The key location information is transformed from the pixel coordinate system to the vehicle coordinate system.

[0285] The vehicle's orientation information is determined based on key position information in the vehicle coordinate system.

[0286] Optionally, the coordinate transformation module is also used for:

[0287] Based on the depth scale corresponding to the target camera that captured the vehicle image and the key position information in the pixel coordinate system, the first position information of the pixel in the vehicle image in the calibration coordinate system is obtained.

[0288] The first position information is back-projected based on the extrinsic parameter matrix of the target camera to obtain the second position information in the camera coordinate system.

[0289] The second position information is projected to obtain the key position information in the vehicle coordinate system.

[0290] Optionally, the vehicle orientation detection device in this application further includes:

[0291] The display module is used to display at least one target vehicle and its corresponding orientation information via an interactive interface.

[0292] Optionally, the interactive interface may also display at least one of the following: the location information of the target vehicle, the location information of the target vehicle's license plate, and the location information of the wheel contact points.

[0293] Optionally, the vehicle images include images captured by at least one fisheye camera on the vehicle.

[0294] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0295] Accordingly, embodiments of this application also provide an electronic device, such as... Figure 11 As shown, Figure 11This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 1100 includes a processor 1101 with one or more processing cores, a memory 1102 with one or more computer-readable storage media, and a computer program stored on the memory 1102 and executable on the processor. The processor 1101 and the memory 1102 are electrically connected. Those skilled in the art will understand that the vehicle structure shown in the figure does not constitute a limitation on the vehicle and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0296] The processor 1101 is the control center of the electronic device 1100. It connects various parts of the electronic device 1100 via various interfaces and lines. By running or loading software programs and / or units stored in the memory 1102, and by calling data stored in the memory 1102, it executes various functions and processes data of the electronic device 1100, thereby providing overall monitoring of the electronic device 1100. The processor 1101 can be a processor (Central Processing Unit, CPU), graphics processing unit (GPU), network processor (NP), etc., and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0297] In this embodiment, the processor 1101 in the electronic device 1100 loads the instructions corresponding to the processes of one or more applications into the memory 1102 according to the following steps, and the processor 1101 runs the applications stored in the memory 1102 to realize various functions, such as:

[0298] The key location information of the target vehicle is determined based on a vehicle image containing the target vehicle; wherein the key location information includes at least the location information of the target vehicle's license plate.

[0299] Based on the key location information, the vehicle's orientation information is determined.

[0300] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0301] Optional, such as Figure 11 As shown, the electronic device 1100 also includes: a touch display screen 1103, a radio frequency circuit 1104, an audio circuit 1105, an input unit 1106, and a power supply 1107. The processor 1101 is electrically connected to the touch display screen 1103, the radio frequency circuit 1104, the audio circuit 1105, the input unit 1106, and the power supply 1107. Those skilled in the art will understand that... Figure 11 The vehicle structure shown does not constitute a limitation on the vehicle and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0302] The touch display screen 1103 can be used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. The touch display screen 1103 may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the vehicle. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Optionally, the display panel can be configured using a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar technologies. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), generate corresponding operation commands, and execute the corresponding program according to the operation commands. Optionally, the touch panel may include two parts: a touch display system and a touch controller. The touch display system detects the user's touch location and the signal generated by the touch operation, transmitting the signal to the touch controller. The touch controller receives touch information from the touch display system, converts it into touch point coordinates, and sends it to the processor 1101. It can also receive and execute commands from the processor 1101. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 1101 to determine the type of touch event. Subsequently, the processor 1101 provides corresponding visual output on the display panel based on the type of touch event. In this embodiment, the touch panel and the display panel can be integrated into the touch display screen 1103 to achieve input and output functions. However, in some embodiments, the touch panel and the touch display screen 1103 can be implemented as two independent components to achieve input and output functions. That is, the touch display screen 1103 can also be used as part of the input unit 1106 to achieve input functions.

[0303] The radio frequency circuit 1104 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other vehicles, and to transmit and receive signals with network devices or other vehicles.

[0304] Audio circuit 1105 can be used to provide an audio interface between the user and the vehicle via a speaker and a microphone. Audio circuit 1105 can convert received audio data into electrical signals and transmit them to the speaker, where the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuit 1105, converted back into audio data, and processed by processor 1101 before being transmitted via radio frequency circuit 1104 to, for example, another vehicle, or output to memory 1102 for further processing. Audio circuit 1105 may also include an earphone jack to provide communication between external headphones and the vehicle.

[0305] The input unit 1106 can be used to receive input numbers, characters, or user characteristic information (such as fingerprints, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0306] Power supply 1107 is used to supply power to various components of electronic device 1100. Optionally, power supply 1107 can be logically connected to processor 1101 through a power management device, thereby enabling functions such as charging, discharging, and power consumption management through the power management device. Power supply 1107 may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0307] although Figure 11 As not shown in the diagram, the electronic device 1100 may also include a camera, sensor, wireless fidelity module, Bluetooth module, etc., which will not be described in detail here.

[0308] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0309] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0310] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of computer programs. These computer programs can be loaded by a processor to execute any of the vehicle orientation detection methods provided in this application. The computer program can execute the following steps of the vehicle orientation detection method:

[0311] The key location information of the target vehicle is determined based on a vehicle image containing the target vehicle; wherein the key location information includes at least the location information of the target vehicle's license plate.

[0312] Based on the key location information, the vehicle's orientation information is determined.

[0313] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0314] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0315] Since the computer program stored in the computer-readable storage medium can execute any of the vehicle orientation detection methods provided in the embodiments of this application, the beneficial effects that any of the vehicle orientation detection methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0316] This application embodiment also provides a vehicle, which may include:

[0317] Body;

[0318] A camera module is installed on the vehicle body, and the camera module is used to acquire vehicle images containing the target vehicle;

[0319] A control module is used to determine the vehicle's orientation information according to the aforementioned method.

[0320] Optionally, there may be multiple camera modules, with at least two camera modules disposed on different sides of the vehicle body.

[0321] Optionally, the camera module is a fisheye camera.

[0322] The beneficial effects that any vehicle orientation detection method provided in this application can achieve are detailed in the preceding embodiments and will not be repeated here.

[0323] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0324] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0325] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0326] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0327] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0328] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0329] Computer-readable media include both permanent and non-permanent, removable and non-removable media, which can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient media, such as modulated communication signals and carrier waves.

[0330] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0331] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0332] The embodiments, implementation methods, and related technical features of this application can be combined and substituted for each other without conflict.

[0333] The above are merely preferred embodiments of this application and are not intended to limit this application in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of this application without departing from the scope of the technical solution of this application shall still fall within the scope of the technical solution of this application.

Claims

1. A method for detecting vehicle orientation, characterized in that, The method includes: The key location information of the target vehicle is determined based on a vehicle image containing the target vehicle; wherein the key location information includes at least the location information of the target vehicle's license plate. Based on the key location information, the vehicle's orientation information is determined.

2. The vehicle orientation detection method according to claim 1, characterized in that, The step of determining the key location information of the target vehicle based on a vehicle image containing the target vehicle includes: Obtain an encoded image of the vehicle image, wherein the encoded image is obtained by segmenting a vehicle image containing the target vehicle, and the pixel values ​​of the encoded image are used to indicate the category identifier corresponding to each pixel in the vehicle image, wherein the category identifier includes at least one of the following: rear of vehicle, front of vehicle, wheels, and license plate. Edge detection is performed on the encoded image to obtain the key location information.

3. The method according to claim 2, characterized in that, The step of performing edge detection on the encoded image to obtain the key location information includes: Edge detection is performed on the encoded image to obtain the edge contour of the license plate; Polygon approximation processing is performed based on the pixel coordinates of the edge contour of the license plate, and the position information of the license plate is obtained based on the processing result.

4. The vehicle orientation detection method according to claim 3, characterized in that, The process of performing polygon approximation processing based on the pixel coordinates of the license plate's edge contour, and obtaining the license plate's position information based on the processing result, includes: The edge contour of the license plate is subjected to polygon approximation processing to obtain a geometric license plate image of the edge contour; A perspective transformation is performed on the geometric license plate image to obtain the position information of the license plate corner points.

5. The vehicle orientation detection method according to claim 2, characterized in that, Determining the vehicle's orientation information based on the key location information includes: The plane where the license plate is located is determined based on the key location information; The vehicle's orientation information is determined based on the direction information corresponding to the normal vector of the plane.

6. The method according to claim 5, characterized in that, Determining the vehicle's orientation information based on the direction information corresponding to the normal vector of the plane includes: Based on the direction information corresponding to the normal vector of the plane and the category identifier in the encoded image used to indicate the vehicle's orientation, the vehicle's orientation information is determined, wherein the category identifier includes either the front or the rear of the vehicle.

7. The method according to claim 2, characterized in that, The key location information also includes: the contact point location information of the two wheels on the same side of the vehicle; The step of performing edge detection on the encoded image to obtain the key location information further includes: Edge detection is performed on the encoded image to obtain the edge contours of the two wheels on the same side; The geometric center of the edge contour is obtained based on the pixel coordinates on the edge contour. Based on the geometric center of the profile, obtain the contact point location information of the two wheels on the same side of the vehicle.

8. The vehicle orientation detection method according to claim 7, characterized in that, Determining the vehicle's orientation information based on the key location information includes: The direction of the target vehicle is determined based on the contact point location information of the two wheels on the same side; The vehicle's orientation information is obtained based on the direction and a category identifier in the vehicle image used to indicate the vehicle's orientation, wherein the category identifier includes either the front or rear of the vehicle.

9. The vehicle orientation detection method according to any one of claims 2 to 8, characterized in that, The process of acquiring the encoded image of the vehicle image includes: Based on the vehicle image, the coded image of the target vehicle is obtained using a preset vehicle segmentation model.

10. The vehicle orientation detection method according to claim 9, characterized in that, The step of obtaining the coded image of the target vehicle based on the vehicle image using a preset vehicle segmentation model includes: The vehicle image is used as input, and feature extraction is performed through the feature extraction network in the vehicle segmentation model to obtain a feature image. The feature image is classified using the convolutional layer in the vehicle segmentation model to obtain a category prediction image; The pixels in the category prediction image are classified using the activation function in the vehicle segmentation model to obtain the encoded image of the target vehicle.

11. The vehicle orientation detection method according to claim 10, characterized in that, Before classifying the pixels in the category prediction image using the activation function in the vehicle segmentation model to obtain the encoded image of the target vehicle, the method further includes: The category prediction image is upsampled using the deconvolution layer in the vehicle segmentation model.

12. The vehicle orientation detection method according to claim 9, characterized in that, Before acquiring the encoded image of the vehicle image, the process further includes: The vehicle image is subjected to distortion correction processing to obtain a corrected vehicle image; Based on the standard resolution of the image corresponding to the vehicle segmentation model, the resolution of the vehicle image after distortion correction is adjusted to obtain the processed vehicle image.

13. The vehicle orientation detection method according to claim 12, characterized in that, The distortion correction processing of the vehicle image includes: Using a distortion correction model, the radial distortion of each pixel in the vehicle image is corrected based on the calibration parameters of the target camera that captured the vehicle image, resulting in a corrected vehicle image.

14. The vehicle orientation detection method according to claim 9, characterized in that, The training steps for the vehicle segmentation model include: Obtain vehicle image samples, wherein the vehicle image samples are labeled with vehicle orientation categories, and the vehicle orientation categories include at least one of the following: rear of vehicle, front of vehicle, wheels, and license plate; The vehicle segmentation model is trained based on the vehicle image samples to obtain the trained vehicle segmentation model.

15. The vehicle orientation detection method according to claim 14, characterized in that, The annotation steps for the vehicle image samples include: Obtain the angle between the marked vehicle and the corresponding target camera in the vehicle image sample; Based on the included angle, the labeling information corresponding to the labeled vehicle is obtained, wherein the labeling information includes at least one of the following: rear labeling information, front labeling information, wheel labeling information, or license plate labeling information.

16. The vehicle orientation detection method according to claim 1, characterized in that, Determining the vehicle's orientation information based on the key location information includes: The key location information is transformed from the pixel coordinate system to the vehicle coordinate system. The vehicle's orientation information is determined based on key position information in the vehicle coordinate system.

17. The vehicle orientation detection method according to claim 16, characterized in that, The step of performing coordinate system transformation on the key location information to convert the key location information from the pixel coordinate system to the vehicle coordinate system includes: Based on the depth scale corresponding to the target camera that captured the vehicle image and the key position information in the pixel coordinate system, the first position information of the pixel in the vehicle image in the calibration coordinate system is obtained. The first position information is back-projected based on the extrinsic parameter matrix of the target camera to obtain the second position information in the camera coordinate system. The second position information is projected to obtain the key position information in the vehicle coordinate system.

18. The vehicle orientation detection method according to claim 1, characterized in that, The method further includes: The interactive interface displays at least one target vehicle and its corresponding orientation information.

19. The vehicle orientation detection method according to claim 18, characterized in that, The interactive interface also displays at least one of the following: the location information of the target vehicle, the location information of the target vehicle's license plate, and the location information of the wheel contact point.

20. The vehicle orientation detection method according to claim 1, characterized in that, The vehicle images include images taken by at least one fisheye camera on the vehicle.

21. A vehicle orientation detection device, characterized in that, The device includes: The first determining module is used to determine key location information of the target vehicle based on a vehicle image containing the target vehicle; wherein the key location information includes at least the location information of the license plate of the target vehicle; The second determining module is used to determine the vehicle's orientation information based on the key location information.

22. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the method of any one of claims 1 to 20.

23. A computer-readable storage medium, characterized in that, It includes a computer program that, when run on an electronic device, causes the electronic device to perform the method of any one of claims 1 to 20.

24. A computer program product, characterized in that, The method includes a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the method of any one of claims 1 to 20.

25. A vehicle, characterized in that, The vehicle is equipped with at least one of the following: the vehicle orientation detection device of claim 21, the electronic device of claim 22, the computer-readable storage medium of claim 23, and the computer program product of claim 24.

26. A vehicle, characterized in that, include: Body; A camera module is installed on the vehicle body, and the camera module is used to acquire vehicle images containing the target vehicle; A control module, the control module being used to determine the vehicle's orientation information according to any one of claims 1-20.

27. The vehicle as claimed in claim 26, characterized in that, There are multiple camera modules, with at least two camera modules located on different sides of the vehicle body.

28. The vehicle as claimed in claim 27, characterized in that, The camera module is a fisheye camera.

Images