Vehicle rear overhang detection method, device, medium, and program product

Abstract

The present invention proposes a vehicle rear overhang detection method, device, medium, and program product, which belongs to the field of image processing technology. The method comprises: inputting a number of measured vehicle rear images into a first deep learning model to generate a number of candidate bounding boxes for the vehicle rear; screening all candidate bounding boxes to screen out a target bounding box; selecting any point of the target edge line of the target bounding box as the first position point; for each target bounding box, recording the vertical distance from the first position point to the center line of the measured vehicle rear image to which the target bounding box belongs; selecting the measured vehicle rear image corresponding to the minimum vertical distance and inputting it into a second deep learning model, identifying the contact point between the rear wheel and the ground as the second position point; obtaining the coordinates of the first position point and the second position point of the measured vehicle rear image corresponding to the minimum vertical distance in the world coordinate system, and obtaining the length of the rear overhang. The method is based on a deep learning model and can achieve reliable and accurate rear overhang detection.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a vehicle rear overhang detection method, device, medium, and program product. Background Art

[0002] When different vehicles are stored in the stacking parking spaces of the smart parking garage, the rear wheels are stored in the same position. If a vehicle with a rear overhang exceeding the designed length of the stacking parking space is stored, the rear of the vehicle will exceed the stacking parking space area and invade the shuttle lane, which will cause a collision risk and damage to the vehicle and equipment. In practice, some vehicles may have modified towing hooks or installed spare tires, resulting in uncontrollable actual rear overhang length. Therefore, it is necessary to detect the actual rear overhang of the vehicle before storing it in the stacking parking space to determine whether to allow it to be stored. Summary of the invention

[0003] In view of the shortcomings of the prior art, the present invention proposes a vehicle rear overhang detection method, system, medium, and program product. The method is based on a deep learning model and can achieve reliable and accurate rear overhang detection.

[0004] On one hand, the present invention provides a vehicle rear overhang detection method, comprising: obtaining a plurality of measured vehicle rear images; inputting the plurality of measured vehicle rear images into a first deep learning model to generate a plurality of candidate boundary boxes of the vehicle rear, wherein the first deep learning model is used to predict the position of the edge line of the projection point of the last position point of the vehicle rear on the ground in the image; screening all candidate boundary boxes to screen out target boundary boxes that meet preset screening conditions, wherein the preset screening conditions are set according to the confidence level and / or the size of the boundary box; selecting any point of the target edge line of the target boundary box as the first position point; for each of the target boundary boxes, recording the vertical distance from the first position point to the center line of the measured vehicle rear image to which the target boundary box belongs, wherein: defining the vehicle width square The first coordinate axis is the direction of the vehicle and the second coordinate axis is the length direction of the vehicle, the first coordinate axis is perpendicular to the second coordinate axis, the target edge line extends along the first coordinate axis and is close to the rear of the vehicle, and the center line forms an angle with the second coordinate axis; the measured rear image corresponding to the minimum vertical distance is selected and input into the second deep learning model, and the contact point between the rear wheel and the ground is identified as the second position point, wherein the second deep learning model uses the contact point between the rear wheel and the ground as the key feature point during model training; according to the coordinates of the first position point and the coordinates of the second position point of the measured rear image corresponding to the minimum vertical distance in the camera coordinate system, the coordinates of the first position point and the coordinates of the second position point in the world coordinate system are obtained to obtain the length of the rear overhang.

[0005] In one embodiment of the present invention, the length of the rear overhang is calculated based on the distance between the first position point and the second position point, and the angle between the line connecting the first position point and the second position point at the first position point and the first coordinate axis or at the angle between the second position point and the second coordinate axis.

[0006] In one embodiment of the present invention, the preset screening condition is set according to the confidence setting, and is set to exceed the confidence threshold; wherein, screening all candidate bounding boxes includes: among all candidate bounding boxes, calculating the confidence of each candidate bounding box; selecting the candidate bounding box with the highest confidence, and checking whether the confidence of the candidate bounding box exceeds the confidence threshold; if it exceeds the confidence threshold, retaining the candidate bounding box.

[0007] In one embodiment of the present invention, the preset screening condition is set according to the size of the bounding box, and is set to exceed the bounding box size threshold; wherein screening all candidate bounding boxes includes: calculating the size of each candidate bounding box according to the average size of the vehicle and the viewing angle of the camera; if there is a candidate bounding box whose size is smaller than the bounding box size threshold, filtering out the candidate bounding box; if there is a candidate bounding box whose size is greater than or equal to the bounding box size threshold, retaining the candidate bounding box.

[0008] In one embodiment of the present invention, the vehicle rear overhang detection method further includes: training the first deep learning model, including: collecting a number of training vehicle rear images and constructing a data set; annotating the edge of the vehicle rear in each image in the data set with a bounding box, drawing a rectangular bounding box around the edge of the vehicle rear, and recording the coordinates of the upper left corner, lower right corner, and center point of the bounding box; using the annotated data set to train the first deep learning model, and during the training process, the first deep learning model learns to recognize the features of the vehicle rear in the image and generates a candidate bounding box.

[0009] In one embodiment of the present invention, the vehicle rear overhang detection method further includes: training the second deep learning model, including: collecting a number of training vehicle rear images and constructing a data set; marking the contact point between the rear wheel and the ground for each image in the data set; and using the marked data set to train the second deep learning model.

[0010] In one embodiment of the present invention, a camera is installed on the top of the rear of a parking space, and the optical axis of the camera is perpendicular to the ground, so as to collect the plurality of measured vehicle rear images.

[0011] In one embodiment of the present invention, the camera is calibrated to obtain the camera's intrinsic parameter matrix and distortion coefficient; a world coordinate system is established with any point in the parking space as the origin, wherein the x0y plane is parallel to the ground; based on any point in the world coordinate system and the corresponding point of the arbitrary point in the pixel coordinate system, and the camera's intrinsic parameter matrix, the rotation matrix and translation vector of the camera coordinate system relative to the world coordinate system are solved; based on the coordinates of the first position point in the camera coordinate system, the coordinates of the second position point, the camera's intrinsic parameter matrix, the rotation matrix and the translation vector, the coordinates of the first position point and the coordinates of the second position point in the world coordinate system are obtained.

[0012] In one embodiment of the present invention, before the measured vehicle rear image is input into the first deep learning model or the second deep learning model, the measured vehicle rear image is corrected and dedistorted in advance according to the camera's intrinsic parameter matrix and distortion coefficient.

[0013] In one embodiment of the present invention, the first deep learning model adopts the yolov5 network, and the second deep learning model adopts the yolov7-pose network.

[0014] On the other hand, the present invention also provides a vehicle rear overhang detection device, which adopts the above-mentioned vehicle rear overhang detection method, and the vehicle rear overhang detection device at least includes: an image acquisition module, used to acquire a plurality of measured vehicle rear images; a candidate bounding box generation module, used to input the plurality of measured vehicle rear images into a first deep learning model to generate a plurality of candidate bounding boxes for the vehicle rear; a target bounding box screening module, used to screen all candidate bounding boxes, and screen out target bounding boxes that meet preset screening conditions, wherein the preset screening conditions are set according to the confidence level and / or the size of the bounding box; a first position point identification module, used to select any point on the target edge line of the target bounding box as the first position point; for each of the target bounding boxes, record the vertical distance from the first position point to the center line of the measured vehicle rear image to which the target bounding box belongs, wherein: the vehicle width is defined The first coordinate axis is in the direction of the degree and the second coordinate axis is in the direction of the vehicle length, the first coordinate axis is perpendicular to the second coordinate axis, the target edge line extends along the first coordinate axis and is close to the rear of the vehicle, and the center line forms an angle with the second coordinate axis; a second position point identification module is used to select the measured rear image of the vehicle corresponding to the minimum vertical distance and input it into the second deep learning model, and identify the contact point between the rear wheel and the ground as the second position point, wherein the second deep learning model uses the contact point between the rear wheel and the ground as the key feature point during model training; a rear overhang length calculation module is used to obtain the coordinates of the first position point and the coordinates of the second position point in the world coordinate system according to the coordinates of the first position point and the coordinates of the second position point in the measured rear image of the vehicle corresponding to the minimum vertical distance in the camera coordinate system, so as to obtain the length of the rear overhang.

[0015] In yet another aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the vehicle rear overhang detection method described above are implemented.

[0016] In yet another aspect, the present invention provides a computer program product, comprising a computer program, which implements the steps of the above-mentioned vehicle rear overhang detection method when executed by a processor.

[0017] It can be seen from the above scheme that the advantages of the present invention are:

[0018] The vehicle rear overhang detection method provided by the present invention utilizes a first deep learning model to identify the edge line where the projection point of the rearmost position point of the vehicle is located, and utilizes a second deep learning model to identify the contact point between the rear wheel and the ground as the second position point, and the target edge line where the projection point of the rearmost position point of the vehicle is located, and the length of the rear overhang can be determined based on the geometric relationship between the first position point and the second position point of the target edge line, thereby achieving reliable and accurate rear overhang detection. This division of labor enables each model to conduct more in-depth learning and optimization within its specific field, thereby improving the accuracy and efficiency of recognition. The vehicle rear overhang detection method, method, device, medium, and program product provided by the present invention not only significantly improve the reliability and accuracy of rear overhang detection, but also have a higher degree of automation than traditional methods, reducing errors caused by human intervention. This method is not only suitable for vehicle rear overhang measurement tasks in stereo garage scenarios, but can also be extended to other application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of the overall process of a vehicle rear overhang detection method disclosed in an embodiment of the present invention;

[0020] Figure 2(a) to Figure 2(c) Schematic diagram showing the camera capturing the rear of the vehicle when the vehicle enters a parking space; FIG2(a), FIG2(b), and FIG2(c) respectively illustrate the situations where the rear of the vehicle is not parked in the optimal measurement position, is parked in the optimal measurement position, and exceeds the optimal measurement position;

[0021] Figure 3 A schematic diagram of a camera taking a bird's-eye view is shown;

[0022] Figure 4 A schematic diagram of the rear overhang calculation is shown;

[0023] Figure 5 A schematic diagram of the rear overhang calculation is shown;

[0024] Figure 6 The figure is a schematic structural diagram of a vehicle rear overhang detection device disclosed in one embodiment of the present invention.

[0025] Wherein, the reference numerals are:

[0026] a: the contact point between the rear wheel and the ground, the second position point;

[0027] B: The actual vertical projection point of the rear end of the vehicle at its last position on the ground;

[0028] b: The projection point of the last position point of the rear of the vehicle on the ground and the first position point in the image;

[0029] ABCD: target bounding box;

[0030] A1B1C1D1: measured vehicle rear image;

[0031] AB: target edge line;

[0032] E1F1, E2F2: center line;

[0033] O: center point;

[0034] dis1, dis2: vertical distance;

[0035] L: the distance between the first position point and the second position point;

[0036] d: rear overhang length;

[0037] : Angle;

[0038] Xw, Yw, Zw: coordinate axes;

[0039] 10: Camera;

[0040] 20: parking space;

[0041] 200: Vehicle rear overhang detection device;

[0042] 210: image acquisition module;

[0043] 220: candidate bounding box generation module;

[0044] 230: target bounding box screening module;

[0045] 240: first position point identification module;

[0046] 250: second position point identification module;

[0047] 260: Rear overhang length calculation module. DETAILED DESCRIPTION

[0048] It should be noted that, in this application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0049] Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0050] In one embodiment of the present invention, a vehicle rear overhang detection method is also disclosed. Figure 1 As shown, Figure 1 The overall process diagram of the vehicle rear overhang detection method is shown, and the vehicle rear overhang detection method specifically includes the following steps:

[0051] Step S1, obtaining a number of measured vehicle rear images.

[0052] In one embodiment, in combination Figure 2(a) to Figure 2(c) , Figure 3 As shown, FIG2(a), FIG2(b), and FIG2(c) respectively illustrate the situations where the rear end of the vehicle is not parked in the optimal measurement position, is parked exactly in the optimal measurement position, and exceeds the optimal measurement position. Figure 3 The schematic diagram of the camera taking a bird's-eye view is shown. The camera is installed at the top of the rear of the parking space, and the optical axis of the camera is perpendicular to the ground. When the vehicle drives towards the parking space, the camera captures the measured rear image of the vehicle at a certain frame rate (for example, 20 frames per second) to obtain a number of measured rear images of the vehicle. In this embodiment, by a reasonable camera installation angle, it is only necessary to arrange a bird's-eye view camera above the place where the vehicle passes, and the measured rear image of the vehicle can be obtained.

[0053] like Figure 2(a) to Figure 2(c) , Figure 3 As shown in , in one embodiment, a world coordinate system is established with any point of the parking space 20 as the origin, wherein the x0y plane is parallel to the ground. The point of contact between the right rear wheel and the ground is point a, the actual vertical projection point of the last position of the rear end of the vehicle on the ground is point B, and the projection point of the last position of the rear end of the vehicle on the ground captured by the camera 10 is point b. As can be seen from FIG2(b), only when the rear end of the vehicle is located directly below the camera, the actual vertical projection point B coincides with the projection point b captured by the camera. In this embodiment, through the subsequent steps S2-S4, from the multiple images of the rear end of the vehicle driving towards the parking space, the image captured when the vehicle is parked in the best measurement position is selected to calculate the rear overhang length.

[0054] Step S2: input the plurality of measured vehicle rear end images into a first deep learning model to generate a plurality of candidate bounding boxes for the vehicle rear end.

[0055] In one embodiment, the first deep learning model is used to predict the position of the edge line where the projection point of the last position point of the rear of the vehicle on the ground is located in the image. Before the several measured rear images of the vehicle are input into the first deep learning model and several candidate bounding boxes of the rear of the vehicle are generated, the first deep learning model needs to be trained in advance. Specifically, firstly, several training rear images of the vehicle are collected and a data set is constructed. These images can come from different scenes, angles and lighting conditions to ensure the generalization ability of the model. In one embodiment, the first deep learning model is used to predict the position of the edge line where the projection point of the last position point of the rear of the vehicle on the ground is located in the image. The edge of the rear of the vehicle of each image in the data set is annotated with a bounding box, a rectangular bounding box is drawn around the edge of the rear of the vehicle, and the coordinates of the upper left corner, lower right corner and center point of the bounding box are recorded. Then, the annotated data set is used to train the first deep learning model. During the training process, the first deep learning model learns to identify the features of the rear of the vehicle in the image and generates a candidate bounding box.

[0056] In one embodiment, the first deep learning model adopts the yolov5 network, which is a real-time target detection network known for its high speed and accuracy. It is suitable for various target detection tasks, including vehicle tail detection. The yolov5 network includes a backbone network (Backbone), a neck structure (Neck) and a head structure (Head). These three parts work together to achieve the task of target detection. Among them, the backbone network is used to extract key feature points in the image. The backbone network mainly includes a Focus structure, a CSP (Cross Stage Partial) structure, a Conv module, a C3 module, an SPPF (Spatial Pyramid Pooling Fast) module, etc. The Focus structure slices the input image, similar to neighboring downsampling, and maps the length and width information of the image to the channel of the image, thereby reducing the amount of calculation, the amount of parameters and the memory usage, while keeping the information from being lost. The CSP structure effectively improves the feature extraction capability and computational efficiency of the network through partial connections across stages. The Conv module consists of a convolutional layer (Conv2d), a batch normalization layer (BatchNorm2d) and an activation function (such as SiLU), and is responsible for extracting key feature points in the image. The C3 module consists of three Conv modules and a Bottleneck module, which further enhances the network's feature extraction capabilities through residual connections and feature fusion. The SPPF module is used for multi-scale feature fusion to improve the robustness of the network. The neck structure is used to further fuse and enhance features, and includes the FPN (Feature Pyramid Networks) module and the PAN (Path Aggregation Network) module. The FPN module passes high-level semantic features to the lower layer through a top-down path, enhancing the semantic information of the low-level feature map. The PAN module passes low-level graphic features to the upper layer through a bottom-up path, realizing bidirectional fusion of features. The head structure generates candidate bounding boxes and category predictions for target detection based on the extracted feature maps. It mainly includes the Detect module, which consists of multiple 1x1 convolutional layers, and is used to perform dimension increase or dimension reduction operations on feature maps and generate the final detection results. The Detect module predicts multiple candidate bounding boxes and corresponding category probabilities for each grid cell through the anchor box mechanism.

[0057] Finally, by using the trained first deep learning model, the several measured vehicle rear images are input into the first deep learning model to generate several candidate bounding boxes for the vehicle rear. The candidate bounding boxes for the vehicle rear may have errors such as the bounding box being too large, too small, or not including the target vehicle rear. Therefore, it is necessary to further screen the candidate bounding boxes to screen out the target bounding box, which tightly surrounds the target vehicle rear and has a high confidence level. Ideally, the target bounding box is a rectangular bounding box formed by the projection points of the outermost edge points of the vehicle body on the ground, which includes the projection point of the last position point of the vehicle rear on the ground.

[0058] In addition, in one embodiment, before inputting the measured vehicle rear image into the first deep learning model, the measured vehicle rear image is corrected and dedistorted in advance according to the camera's intrinsic parameter matrix and distortion coefficient. Inputting the corrected and dedistorted image into the deep learning model for training can reduce the model's dependence on distortion, thereby improving the model's generalization ability and training effect. At the same time, the corrected and dedistorted image is more standardized, which can simplify the calculation process of the deep learning model, thereby accelerating the model's reasoning speed and improving real-time performance.

[0059] Step S3: Screen all candidate bounding boxes to select target bounding boxes that meet preset screening conditions, wherein the preset screening conditions are set based on the confidence level and / or the size of the bounding box.

[0060] The first deep learning model of this embodiment will generate a large number of candidate bounding boxes, which may contain the target object or may just be background or noise. Therefore, all candidate bounding boxes need to be further screened. Through screening, those bounding boxes with low confidence, unreasonable size or obviously not meeting the target characteristics can be removed, thereby improving the accuracy of the final detection results, which helps to reduce false detections and missed detections and make the detection results more reliable. Specifically, in one embodiment, preset screening conditions are constructed based on the confidence and the size of the bounding box.

[0061] In one embodiment, the preset screening condition is set according to the confidence level, and is set to exceed the confidence level threshold. Specifically, the confidence level of each candidate bounding box is calculated for all candidate bounding boxes; then the candidate bounding box with the highest confidence level is selected, and it is checked whether the confidence level of the candidate bounding box exceeds the confidence level threshold; if the confidence level exceeds the confidence level threshold, the candidate bounding box is retained.

[0062] In one embodiment, the preset screening condition is set according to the size of the bounding box, and is set to exceed the bounding box size threshold. Specifically, the size of each candidate bounding box is calculated according to the average size of the vehicle and the viewing angle of the camera; if there is a candidate bounding box whose size is smaller than the bounding box size threshold, the candidate bounding box is filtered out; if there is a candidate bounding box whose size is greater than or equal to the bounding box size threshold, the candidate bounding box is retained.

[0063] In one embodiment, a preset screening condition is constructed based on the confidence combined with the size of the bounding box, that is, first, the candidate key boxes that meet the confidence threshold are screened out, and then these candidate key boxes can be further screened based on the bounding box size threshold to screen out the final target parking space. Specifically, first, the confidence of each candidate bounding box is calculated for all candidate bounding boxes; then the candidate bounding box with the highest confidence is selected, and it is checked whether the confidence of the candidate bounding box exceeds the confidence threshold; if it exceeds the confidence threshold, the candidate bounding box is retained as the first bounding box. Then, the size of each first bounding box is calculated for all the first bounding boxes according to the average size of the vehicle and the camera's viewing angle; if there is a first bounding box whose size is smaller than the bounding box size threshold, the first bounding box is filtered out; if there is a first bounding box whose size is greater than or equal to the bounding box size threshold, the first bounding box is retained, thereby obtaining the final target bounding box.

[0064] In addition, in one embodiment, when the preset screening condition is constructed based on the confidence level and the size of the bounding box, the candidate key boxes that meet the bounding box size threshold may be screened out first, and then these candidate key boxes may be further screened based on the confidence level threshold to screen out the final target parking space. This method is similar to the above screening method, and will not be described in detail in this embodiment.

[0065] After the above screening operation, a target bounding box is screened out from the candidate bounding boxes of the vehicle rear. The target bounding box tightly surrounds the target vehicle rear and has a high confidence level. The target bounding box is a rectangular bounding box formed by the projection points of the outermost edge points of the vehicle body on the ground, which includes the projection point of the last position point of the vehicle rear on the ground. In this embodiment, the target bounding box of each measured vehicle rear image is screened out, and it is necessary to further screen out the measured vehicle rear image captured when the vehicle rear is directly below the camera as shown in FIG. 2(b) from these measured vehicle rear images to calculate the rear overhang length.

[0066] Step S4, select any point of the target edge line in the direction of the first coordinate axis of the target bounding box as the first position point; for each target bounding box, record the vertical distance from the first position point to the center line of the measured vehicle rear image to which the target bounding box belongs, wherein the vehicle width direction is defined as the first coordinate axis, the vehicle length direction is defined as the second coordinate axis, the first coordinate axis is perpendicular to the second coordinate axis, the target edge line is close to the vehicle rear along the direction of the second coordinate axis, and the center line forms an angle with the second coordinate axis.

[0067] In this embodiment, after the final target bounding box is screened, a coordinate system is first constructed for the corresponding image, wherein the vehicle width direction is defined as the first coordinate axis, the vehicle length direction is defined as the second coordinate axis, and the first coordinate axis is perpendicular to the second coordinate axis. It should be noted that the collected measured vehicle rear image is not limited to an image in the XY standard direction, and it is only necessary to define the vehicle width direction as the first coordinate axis, the vehicle length direction as the second coordinate axis, and the first coordinate axis is perpendicular to the second coordinate axis.

[0068] like Figure 4 As shown in Figure 4 In the figure, the measured image of the rear of the vehicle is taken as an example as an image in the XY standard direction, with the first coordinate axis being the X axis and the second coordinate axis being the Y axis. The target bounding box is ABCD, and the target edge line in the direction of the first coordinate axis of the target bounding box is the AB connecting line. The target edge line is close to the rear of the vehicle along the second coordinate axis, and the target edge line includes the vertical projection point of the last position point of the rear of the vehicle on the ground. The measured image of the rear of the vehicle corresponding to the target bounding box is A1B1C1D1, and the center lines E1F1 and E2F2 are the center lines of the measured image of the rear of the vehicle A1B1C1D1. In this embodiment, the camera is installed at the top of the rear of the parking space, and the optical axis of the camera is perpendicular to the ground. The center position of the camera is fixed, and the image of the rear of the vehicle taken from above is an image within a fixed viewing angle, that is, the photographed area is fixed. Figure 4 , Figure 5 As shown in , the measured rear image A1B1C1D1 in the two figures is fixed, the projection point of the camera center position on the ground is the center point O of the image, and the center lines E1F1 and E2F2 are straight lines passing through the center point O, that is, the center lines of the measured rear image of the vehicle, as shown in Figure 4The center lines E1F1 and E2F2 in the image. As shown in Figure 2(b), only when the rear of the car is directly below the camera will the actual vertical projection point B coincide with the projection point b captured by the camera. At this time, the first position point b of the target edge line AB of the target bounding box is closest to the center lines E1F1 and E2F2. In order to be able to compare the distances from the first position points of the target edge lines of different target bounding boxes to the center line, it is necessary to specifically define that the center line forms an angle with the second coordinate axis, that is, it cannot be parallel to the second coordinate axis. If they are parallel, the distances from the first position points of the target edge lines of each target bounding box to the center line are the same.

[0069] In this embodiment, any point on the target edge line in the direction of the first coordinate axis of the target boundary box is selected as the first position point. The first position point b can be any point on the AB connecting line, such as the rightmost edge point, that is, the vertical projection point of the last position point of the rear of the vehicle on the ground. However, this embodiment is not limited to the rightmost edge point, and any point on the target edge line can be selected as the first position point b. The vertical distance dis from the first position point b to the center line of the measured rear image of the target boundary box is further recorded. For example Figure 4 As shown in , the vertical distance dis1 corresponding to the center line E1F1, and the vertical distance dis2 corresponding to the center line E2F2.

[0070] For each target bounding box, a centerline is selected that has the same angle as the corresponding second coordinate axis, e.g. Figure 4 As shown in , a target bounding box selects the center line E1F1 parallel to the first coordinate axis (90° to the second coordinate axis), then each compared target bounding box selects the center line parallel to the first coordinate axis corresponding to itself. Record the vertical distance from the first position point b to the center line to obtain the vertical distances dis_1, dis_2, ....dis_n. Compare the vertical distances dis_1, dis_2, ....dis_n obtained by each target bounding box. The smaller the vertical distance, the closer the rear edge of the rear of the vehicle is to the camera optical axis. In theory, the projection point of the last position of the rear of the vehicle in the rear image on the ground is closer to the actual vertical projection point of the last position of the rear of the vehicle on the ground. Under ideal conditions, such as Figure 5As shown in , taking the center line E1F1 as the center line in the direction of the first coordinate axis as an example, that is, it is at an angle of 90° with the second coordinate axis, the center line E1F1 coincides with the target edge line AB, that is, the vertical distance is 0. At this time, when the rear of the car is directly below the camera, the actual vertical projection point B coincides with the projection point b captured by the camera. Therefore, the measured rear image corresponding to the minimum vertical distance is selected and input into the second deep learning model to further identify the contact point between the rear wheel and the ground. It should be noted here that when comparing the vertical distance from the first position point of each target bounding box to the center line, if the center line and the second coordinate axis are at an angle of less than 90°, then the first position points of each target bounding box need to be fixed to the same point, for example, all are selected as the rightmost edge point. If the center line and the second coordinate axis are at an angle of 90°, that is, for the same target bounding box, the distance from each first position point to the center line is the same, and at this time, it is not necessary to limit the first position points of each target bounding box to be fixed to the same point.

[0071] Step S5: select the measured vehicle rear image corresponding to the minimum vertical distance and input it into the second deep learning model, and identify the contact point between the rear wheel and the ground as the second position point, wherein the second deep learning model uses the contact point between the rear wheel and the ground as the key feature point during model training.

[0072] In one embodiment, before the second deep learning model processes the measured vehicle rear image, it is also necessary to pre-train the second deep learning model. Unlike the first deep learning model, the second deep learning model needs to annotate key feature points. In this embodiment, the contact point between the rear wheel and the ground is specifically selected as the key feature point. In practice, the contact point between the rear wheel and the ground cannot be actually seen in the image taken by the camera from above, so it is necessary to place a marker at the contact point between the rear wheel and the ground in advance, identify the coordinates of the marker, and annotate the contact point between the rear wheel and the ground by comparing the image with the marker and the image without the marker in the same scene. Specifically, first, a number of training vehicle rear images are collected and a data set is constructed. These images can come from different scenes, angles and lighting conditions to ensure the generalization ability of the model. Then, the contact point between the rear wheel and the ground is selected as the key feature point, and the contact point between the rear wheel and the ground is annotated for each image in the data set. Finally, the annotated data set is used to train the second deep learning model.

[0073] In one embodiment, the second deep learning model adopts the yolov7-pose network. The yolov7-pose network is improved on the basis of YOLOv7, mainly adding the function of key point detection, which realizes the precise positioning of key points. It adopts an end-to-end training method to integrate the target detection and key point detection tasks into one network, simplifying the training process and improving the overall performance. Specifically, the yolov7-pose network includes an input layer, a backbone network, a neck network, a head network, etc. Taking the processing of the measured rear image of the vehicle in this embodiment as an example, the input layer pre-processes the input measured rear image of the vehicle and is sent to the backbone network. Yolov7-pose uses the same backbone network as YOLOv7, such as ELAN-Net, etc., to extract the key feature points of the measured rear image of the vehicle, that is, the contact points between the rear wheels and the ground. The neck network is responsible for feature fusion, fusing feature maps at different levels to improve the accuracy of key feature point detection. The head network is used to predict the coordinate information of the key feature points. In the yolov7-pose network, each key feature point has a corresponding heat map output, which indicates the position probability of the key point. The core of the yolov7-pose network lies in the generation and analysis of key point heat maps. Through the forward propagation of the network, a key feature point heat map with the same size as the input image is generated. Each pixel value in the heat map indicates the probability that the position is a key feature point. Then, by post-processing the key feature point heat map, such as threshold processing, non-maximum suppression, etc., the coordinate information of the key feature points is extracted.

[0074] Finally, using the trained second deep learning model, the measured vehicle rear image corresponding to the minimum vertical distance is selected and input into the second deep learning model to identify the contact point between the rear wheel and the ground as the second position point, such as Figure 4 As shown in , the second position point is point a. Thus, the length of the rear overhang can be determined by using the first position point b and the second position point a of the measured vehicle rear image corresponding to the minimum vertical distance.

[0075] In addition, in one embodiment, before inputting the measured vehicle rear image into the second deep learning model, the measured vehicle rear image to be input is corrected and dedistorted in advance according to the camera's intrinsic parameter matrix and distortion coefficient. Inputting the corrected and dedistorted image into the deep learning model for training can reduce the model's dependence on distortion, thereby improving the model's generalization ability and training effect. At the same time, the corrected and dedistorted image is more standardized, which can simplify the calculation process of the deep learning model, thereby accelerating the model's reasoning speed and improving real-time performance.

[0076] Step S6, according to the coordinates of the first position point and the second position point of the measured vehicle rear image corresponding to the minimum vertical distance in the camera coordinate system, obtain the coordinates of the first position point and the second position point in the world coordinate system to obtain the length of the rear overhang.

[0077] In one embodiment, the installed camera is calibrated to obtain the camera's intrinsic parameter matrix and distortion coefficient. Figure 3 As shown, a world coordinate system is established with any point in the parking space as the origin, where the x0y plane is parallel to the ground. Based on any point in the world coordinate system and the corresponding point of the arbitrary point in the pixel coordinate system, as well as the intrinsic parameter matrix of the camera, the rotation matrix R and translation vector T of the camera coordinate system relative to the world coordinate system are solved. For example, given the coordinates of n points in the world coordinate system P1, P2, ..., Pn, the corresponding coordinates in the pixel coordinate system p1, p2, ..., pn and the intrinsic parameter matrix of the camera, the camera coordinate system (X c Y c Z c ) relative to the world coordinate system (X w Y w Z w ), that is, the rotation matrix R and translation vector T, is expressed as:

[0078]

[0079] Then, use the inverse K of the internal parameter matrix K -1 Convert pixel coordinates to normalized camera coordinates, normalized camera coordinate vector p norm It is expressed as:

[0080]

[0081] in, Representation vector The third component of the homogeneous coordinate, the w component, is used for normalization. Z represents the known depth value, which represents the Z-axis coordinate of the point in the world coordinate system. Since the world coordinate system is established on the ground plane when calibrating the external parameters, the Z-axis coordinates of all points on the ground plane, including the first position point a and the second position point b, are 0; p represents the homogeneous pixel coordinate [xy 1] T .

[0082] Then, use the transpose R of the rotation matrix R T The translation vector T transforms the normalized camera coordinates into world coordinates, and the world coordinate vector p world It is expressed as:

[0083]

[0084] Finally, the transformation formula from the camera coordinate system to the world coordinate system is obtained as follows:

[0085]

[0086] Therefore, the world coordinate vector p world Extract the X, Y, and Z components of the world coordinate system from world , Y world , Z world ,Right now , where (p world )1、(p world )2、(p world )3 represent the world coordinate vector p world The amount.

[0087] Finally, for the first position point and the second position point, the above-mentioned transformation formula from the camera coordinate system to the world coordinate system is used to obtain the coordinates of the first position point and the second position point in the world coordinate system according to the coordinates of the first position point and the second position point in the camera coordinate system, the intrinsic parameter matrix of the camera, the rotation matrix and the translation vector. Therefore, according to the distance between the first position point and the second position point, and the angle between the line connecting the first position point and the second position point at the first position point and the first coordinate axis or at the angle between the second position point and the second coordinate axis, the length of the rear overhang can be calculated using the sine and cosine theorems of right triangles.

[0088] Combination Figure 5 As shown, for example, in world coordinates, the coordinates of the first position point are expressed as (x1, y1, z1), the coordinates of the second position point are expressed as (x2, y2, z2), and the distance L between the first position point and the second position point is

[0089] , the angle between the line connecting the first position point and the second position point and the first coordinate axis (here, the X axis) at the first position point is , then the length of the rear overhang If the first position point is selected as the rightmost edge point and the second position point is selected as the contact point between the right rear wheel and the ground, or the first position point is selected as the leftmost edge point and the second position point is selected as the contact point between the left rear wheel and the ground, then =90°, then d=L.

[0090] In summary, the vehicle rear overhang detection method disclosed in the present invention can determine the length of the rear overhang based on the geometric relationship between the first position point and the second position point by using a reasonable camera installation angle and a deep learning model to identify the contact point between the rear wheel and the ground, as well as the projection point of the vehicle's last position point on the edge line, and realize reliable and accurate rear overhang detection. This method uses a 2D camera solution to solve the problem that 3D vision has poor imaging effect on the high-gloss surface of passenger cars and that photoelectric detection cannot locate the last edge point of the rear of the car. In addition, the deployment method of the present invention is simple and flexible, and the cost is low. It is not only suitable for vehicle rear overhang measurement tasks in stereo garage scenes, but can also be extended to other application scenarios.

[0091] Correspondingly, corresponding to the above-mentioned vehicle rear overhang detection method, the present invention further provides a vehicle rear overhang detection device 200, see Figure 6 As shown, Figure 6 A schematic structural diagram of the vehicle rear overhang detection device 200 is shown.

[0092] The vehicle rear overhang detection device 200 at least comprises:

[0093] The image acquisition module 210 is used to acquire a plurality of measured vehicle rear images.

[0094] The candidate bounding box generation module 220 is used to input the several measured vehicle rear images into the first deep learning model to generate several candidate bounding boxes for the vehicle rear, wherein the first deep learning model is used to predict the position of the edge line of the projection point of the last position point of the vehicle rear on the ground in the image.

[0095] The target bounding box screening module 230 is used to screen all candidate bounding boxes to select target bounding boxes that meet preset screening conditions, wherein the preset screening conditions are set according to the confidence level and / or the size of the bounding box.

[0096] The first position point identification module 240 is used to select any point of the target edge line in the direction of the first coordinate axis of the target boundary box as the first position point; for each of the target boundary boxes, the vertical distance from the first position point to the center line of the measured vehicle rear image to which the target boundary box belongs is recorded, wherein: the vehicle width direction is defined as the first coordinate axis, the vehicle length direction is defined as the second coordinate axis, the first coordinate axis is perpendicular to the second coordinate axis, the target edge line extends along the first coordinate axis direction and is close to the vehicle rear, and the center line forms an angle with the second coordinate axis.

[0097] The second position point identification module 250 is used to select the measured vehicle rear image corresponding to the minimum vertical distance and input it into the second deep learning model, and identify the contact point between the rear wheel and the ground as the second position point, wherein the second deep learning model uses the contact point between the rear wheel and the ground as the key feature point during model training.

[0098] The rear overhang length calculation module 260 is used to obtain the coordinates of the first position point and the second position point in the world coordinate system based on the coordinates of the first position point and the coordinates of the second position point in the measured vehicle rear image corresponding to the minimum vertical distance in the camera coordinate system, so as to obtain the length of the rear overhang.

[0099] In addition, those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the vehicle rear overhang detection device 200 can refer to the corresponding process in the aforementioned vehicle rear overhang detection method embodiment, which will not be repeated here.

[0100] In addition, the present invention also proposes an electronic device in another embodiment, including: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor executes the above-mentioned vehicle rear overhang detection method.

[0101] The present invention also proposes a storage medium in another embodiment for storing a computer program for executing the above-mentioned vehicle rear overhang detection method. It should be understood that the storage medium in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0102] In addition, the present invention also provides a computer program product in one embodiment, the computer program product includes a computer program, the computer program can be stored in a readable storage medium, and when the computer program is executed by a processor, the computer can execute the above-mentioned vehicle rear overhang detection method.

[0103] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A vehicle rear overhang detection method, characterized in that: Include: Acquire several measured vehicle rear images; Inputting the plurality of measured vehicle rear images into a first deep learning model to generate a plurality of candidate bounding boxes for the vehicle rear; Screening all candidate bounding boxes to select target bounding boxes that meet preset screening conditions, wherein the preset screening conditions are set according to confidence and / or the size of the bounding box; Select a point on the target edge line of the target bounding box as the first position point; for each target bounding box, record the vertical distance from the first position point to the center line of the measured vehicle rear image to which the target bounding box belongs, wherein: the vehicle width direction is defined as the first coordinate axis, the vehicle length direction is defined as the second coordinate axis, the first coordinate axis is perpendicular to the second coordinate axis, the target edge line extends along the first coordinate axis direction and is close to the vehicle rear, and the center line of the measured vehicle rear image forms an angle with the second coordinate axis; Selecting a measured vehicle rear image corresponding to the minimum vertical distance and inputting it into a second deep learning model, identifying a contact point between the rear wheel and the ground as a second position point, wherein the second deep learning model uses the contact point between the rear wheel and the ground as a key feature point during model training; According to the coordinates of the first position point and the second position point of the measured vehicle rear image corresponding to the minimum vertical distance in the camera coordinate system, the coordinates of the first position point and the second position point in the world coordinate system are obtained to determine the distance between the first position point and the second position point; according to the distance between the first position point and the second position point, and the angle between the first position point and the second position point at the first position point and the first coordinate axis or at the second position point and the second coordinate axis, the length of the rear overhang is calculated, wherein: the length of the rear overhang , is the angle between the line connecting the first position point and the second position point at the first position point and the first coordinate axis, L is the distance between the first position point and the second position point; or the length of the rear overhang , is the angle between the line connecting the first position point and the second position point at the second position point and the second coordinate axis, and L is the distance between the first position point and the second position point.

2. The vehicle rear overhang detection method according to claim 1, characterized in that: The preset screening condition is set according to the confidence level and is set to exceed the confidence level threshold; Among them, screening of all candidate bounding boxes includes: Among all candidate bounding boxes, calculate the confidence of each candidate bounding box; Selecting a candidate bounding box with the highest confidence, and checking whether the confidence of the candidate bounding box exceeds the confidence threshold; If the confidence threshold is exceeded, the candidate bounding box is retained.

3. The vehicle rear overhang detection method according to claim 1 or 2, characterized in that: The preset screening condition is set according to the size of the bounding box, and is set to exceed the bounding box size threshold; wherein screening all candidate bounding boxes includes: Calculate the size of each candidate bounding box according to the average size of the vehicle and the viewing angle of the camera; If there is a candidate bounding box whose size is smaller than the bounding box size threshold, filter out the candidate bounding box; If there is a candidate bounding box whose size is greater than or equal to the bounding box size threshold, the candidate bounding box is retained.

4. The vehicle rear overhang detection method according to claim 1, characterized in that: Also includes: Training the first deep learning model includes: Collect several training car rear images and construct a dataset; Annotate the edge of the rear of the vehicle in each image in the data set with a bounding box, draw a rectangular bounding box around the edge of the rear of the vehicle, and record the coordinates of the upper left corner, lower right corner, and center point of the bounding box; The first deep learning model is trained using the labeled data set. During the training process, the first deep learning model learns to recognize the features of the rear of the vehicle in the image and generates a candidate bounding box.

5. The vehicle rear overhang detection method according to claim 1, characterized in that: Also includes: Training the second deep learning model includes: Collect several training car rear images and construct a dataset; Annotating the contact point between the rear wheel and the ground for each image in the data set; The second deep learning model is trained using the labeled data set.

6. The vehicle rear overhang detection method according to claim 1, characterized in that: The camera is installed at the top of the rear of the parking space with the optical axis of the camera perpendicular to the ground to collect the plurality of measured vehicle rear images.

7. The vehicle rear overhang detection method according to claim 6, characterized in that: Calibrate the camera to obtain the camera's intrinsic parameter matrix and distortion coefficient; Take any point in the parking space as the origin and establish a world coordinate system, where the x0y plane is parallel to the ground; According to any point in the world coordinate system and its corresponding point in the pixel coordinate system, as well as the camera's intrinsic parameter matrix, the rotation matrix and translation vector of the camera coordinate system relative to the world coordinate system are solved; The coordinates of the first position point and the second position point in the world coordinate system are obtained according to the coordinates of the first position point and the second position point in the camera coordinate system, the intrinsic parameter matrix of the camera, the rotation matrix and the translation vector.

8. The vehicle rear overhang detection method according to claim 1, characterized in that: Before inputting the measured vehicle rear image into the first deep learning model or the second deep learning model, the measured vehicle rear image is corrected and distortion removed in advance based on the camera's intrinsic parameter matrix and distortion coefficient.

9. The vehicle rear overhang detection method according to claim 1, characterized in that: in, The first deep learning model adopts the yolov5 network, and the second deep learning model adopts the yolov7-pose network.

10. A vehicle rear overhang detection device, characterized in that: The vehicle rear overhang detection method according to any one of claims 1 to 9 is adopted, and the vehicle rear overhang detection device at least comprises: An image acquisition module, used to acquire a number of measured vehicle rear images; A candidate bounding box generation module, used for inputting the plurality of measured vehicle rear images into a first deep learning model to generate a plurality of candidate bounding boxes for the vehicle rear; A target bounding box screening module, used to screen all candidate bounding boxes to select target bounding boxes that meet preset screening conditions, wherein the preset screening conditions are set according to the confidence level and / or the size of the bounding box; A first position point identification module is used to select any point of the target edge line of the target boundary box as the first position point; for each target boundary box, record the vertical distance from the first position point to the center line of the measured vehicle rear image to which the target boundary box belongs, wherein: the vehicle width direction is defined as the first coordinate axis, the vehicle length direction is defined as the second coordinate axis, the first coordinate axis is perpendicular to the second coordinate axis, the target edge line extends along the first coordinate axis direction and is close to the vehicle rear, and the center line forms an angle with the second coordinate axis; A second position point identification module is used to select a measured vehicle rear image corresponding to the minimum vertical distance and input it into a second deep learning model to identify the contact point between the rear wheel and the ground as the second position point, wherein the second deep learning model uses the contact point between the rear wheel and the ground as a key feature point during model training; A rear overhang length calculation module is used to obtain the coordinates of the first position point and the second position point in the world coordinate system according to the coordinates of the first position point and the second position point in the measured vehicle rear image corresponding to the minimum vertical distance in the camera coordinate system, and determine the distance between the first position point and the second position point; calculate the length of the rear overhang according to the distance between the first position point and the second position point, and the angle between the first position point and the second position point at the first position point and the first coordinate axis or at the angle between the second position point and the second coordinate axis, wherein: the length of the rear overhang , is the angle between the line connecting the first position point and the second position point at the first position point and the first coordinate axis, L is the distance between the first position point and the second position point; or the length of the rear overhang , is the angle between the line connecting the first position point and the second position point at the second position point and the second coordinate axis, and L is the distance between the first position point and the second position point.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the vehicle rear overhang detection method described in any one of claims 1 to 9 are implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the vehicle rear overhang detection method described in any one of claims 1 to 9 are implemented.

Images