Vehicle visual detection method and device based on deep learning and storage medium

Abstract

The application provides a kind of based on deep learning's visual detection method for vehicle, device and storage medium, the method includes obtaining the vehicle scene picture to be identified;The vehicle scene picture to be identified is input to the preset visual detection model for vehicle, and identification result is generated;Wherein, the identification result includes parking space identification result, tire identification result and lane identification result;According to the identification result, vehicle posture is corrected or path planning is carried out;The application is identified to parking space, tire and lane by the preset visual detection model for vehicle respectively, and the corresponding result is generated, then these identification results are combined to assist planning driving strategy;Not only can different vehicle scenes be quickly and accurately identified, but also driving strategy can be reasonably and effectively formulated according to different vehicle scene identification results, to improve road traffic safety and vehicle management efficiency.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a deep learning-based vehicle vision inspection method, device, and storage medium. Background Technology

[0002] With the rapid pace of urban development, the number of vehicles in cities has exploded. However, road traffic management has always faced numerous challenges: First, finding parking spaces is time-consuming. Even with the proliferation of parking management systems, most rely on sensor and image processing technologies, which are not only costly but also lack reliable accuracy. Second, current systems typically identify obstacles and vehicle speeds, with limited capabilities for recognizing parking status or road surface smoothness. This makes it difficult to understand tire position information and the impact of potholes on vehicle management, resulting in inefficient road safety management. Third, existing vehicle scene recognition methods are often localized, such as detecting only one piece of information like road conditions or vehicle speed. This limited information can lead to biases when developing driving strategies. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this invention provides a deep learning-based vehicle vision detection method, device, and storage medium to solve the problems of inaccurate vehicle scene recognition and easy deviation in the formulation of driving strategies in existing technologies.

[0004] In a first aspect, the present invention provides a deep learning-based vehicle vision inspection method, the method comprising:

[0005] Obtain the vehicle scene image to be identified;

[0006] The vehicle scene image to be identified is input into a preset vehicle vision detection model to generate recognition results; wherein, the recognition results include parking space recognition results, tire recognition results, and lane recognition results;

[0007] The vehicle posture is corrected or a path is planned based on the recognition results.

[0008] In current vehicle management processes, vehicle scene recognition is often based on localized identification, such as the individual identification of a vehicle's parking position, speed, or obstacles. However, this approach, focusing solely on these details, fails to consider the impact of tires and potholes on vehicle movement or parking, and cannot yield fast or accurate results. Furthermore, isolated scene recognition often fails to effectively support driving strategies. Therefore, this approach first acquires images of the vehicle scene to be identified. Then, a pre-set automotive vision detection model is used to identify parking spaces, tires, and lanes, generating corresponding results. Finally, these results are combined to assist in planning driving strategies. The trained automotive vision detection model can accurately identify different vehicle scenes, and based on the identification results of different vehicle scenes, driving strategies can be rationally and effectively formulated, improving road traffic safety and the efficiency of vehicle management.

[0009] In one possible implementation, the vehicle vision detection model includes a parking space recognition model, a tire recognition model, and a lane recognition model; wherein,

[0010] The parking space recognition model and the tire recognition model are trained based on the Yolov3 algorithm;

[0011] The lane recognition model was trained based on the Yolov5 algorithm.

[0012] In this possible implementation, different recognition models are trained using different algorithms. Parking spaces and tires are relatively easier to label and identify than road potholes, therefore the YOLOv3 algorithm is used. Using the YOLOv3 algorithm to train the parking space recognition model improves recognition efficiency and accuracy compared to existing fuzzy C-means cluster classification systems and recurrent neural network systems for parking space recognition. Using the YOLOv3 algorithm to train the tire recognition model can accurately identify tire positions, avoiding problems such as traffic congestion caused by improper parking due to vehicle misalignment, occupying multiple public parking spaces, and vehicle scratches. This significantly improves the efficiency of traffic management and increases the utilization rate of public resources. Using the YOLOv5 algorithm to train the lane recognition model further reduces computational load and can accurately identify road potholes, assisting drivers in planning their routes and improving road safety.

[0013] In one possible implementation, before inputting the vehicle scene image to be identified into a preset vehicle vision detection model, the method further includes:

[0014] Training the parking space recognition model includes:

[0015] Obtain a first preset number of parking space scene images;

[0016] The vacant parking spaces, occupied parking spaces, and parked vehicles in the parking scene image are labeled to generate a first training set; wherein the parked vehicles include at least two parking directions;

[0017] Based on the Darknet framework, the Yolov3 algorithm is trained using the first training set to generate the parking space recognition model.

[0018] In this possible implementation, different parking space scene images were used. To accurately identify the usage status of parking spaces, both parking spaces and vehicles were labeled during annotation, and parking spaces were divided into vacant and occupied spaces. To enhance the model's recognition capability, parked vehicles were also included in at least two parking directions. Therefore, this training set can more comprehensively identify different parking space scene images. Furthermore, training the Yolov3 algorithm based on the Darknet framework can more quickly train a parking space recognition model.

[0019] In one possible implementation, before inputting the vehicle scene image to be identified into a preset vehicle vision detection model, the method further includes:

[0020] Training the tire recognition model includes:

[0021] Get a second preset number of parked vehicle images;

[0022] The tires in the images of the parked vehicles are labeled to generate a second training set; wherein the tires and the vehicle body form at least two different angles.

[0023] Based on the Darknet framework, the Yolov3 algorithm is trained using the second training set to generate the tire recognition model.

[0024] In this possible implementation, in order to accurately identify the position of the tire, the angle formed by the tire and the vehicle body is mainly marked and the YOLOv3 algorithm is trained. This enables the tire recognition model to accurately identify the position of the tire, thereby determining whether the vehicle body is off-center when parked. Similarly, in this implementation, the YOLOv3 algorithm is also trained based on the lightweight Darknet framework, thereby improving the training speed.

[0025] In one possible implementation, before inputting the vehicle scene image to be identified into a preset vehicle vision detection model, the method further includes:

[0026] Training the lane recognition model includes:

[0027] Obtain a third preset number of lane scene images;

[0028] The potholes in the lane scene images are labeled to generate a third training set;

[0029] The YOLOv5 algorithm is trained using the third training set to generate the lane recognition model; the YOLOv5 algorithm includes the CSP-Darknet feature network.

[0030] Road potholes are more difficult to identify than tires or parking spaces because parking spaces and tires have relatively obvious morphological features, and their coordinates can be determined simply by using bounding boxes for annotation. However, potholes are easily affected by weather, background, and lighting, which can alter their texture and shape, making identification more challenging. In this possible implementation method, the YOLOv5 algorithm is primarily used to accurately identify road potholes. The YOLOv5 algorithm can further improve the model's detection performance by changing the loss function, the cross-union ratio method, and the detection module structure. By employing CSP-Darknet, computational complexity can be reduced while maintaining accuracy.

[0031] In one possible implementation, training the Yolov5 algorithm using the third training set includes:

[0032] Based on the third training set, the Yolov5 algorithm is trained using mosaic training. The mosaic training involves scaling and rotating several randomly acquired labeled lane scene images and piecing them together to form a new lane scene image, which is then used to train the Yolov5 algorithm.

[0033] In this possible implementation method, mosaic training reads several images at a time and stitches four or more images together into one image by scaling, flipping, etc. This kind of training can make the network more generalizable and accurately identify the target in different situations.

[0034] In one possible implementation, training the Yolov5 algorithm using the third training set further includes:

[0035] Based on the third training set, the Yolov5 algorithm is trained using self-adversarial training; the self-adversarial training includes a first stage and a second stage.

[0036] The first stage is used to change the size of the annotated lane scene image;

[0037] The second stage is used for training based on the lane scene image with modified dimensions.

[0038] In this possible implementation method, training through this self-attack adversarial approach can increase the robustness of the network, enabling it to still identify targets even when they are blurred or incomplete in the image.

[0039] In one possible implementation, correcting the vehicle posture or performing path planning based on the identification result includes:

[0040] Based on the tire recognition results, the angle between the tires and the vehicle body of the parked vehicle is determined. When the angle is determined to be greater than a first threshold, a prompt is triggered to correct the current vehicle posture; or,

[0041] Based on the parking space recognition results and the lane recognition results, determine the number and location of road potholes and vacant parking spaces; and replan the vehicle's driving path based on the number and location of road potholes and vacant parking spaces.

[0042] In this possible implementation method, driving strategies are primarily adjusted based on various recognition results. Specifically, tire recognition results can determine the rationality of the vehicle's current parking posture. If the angle between the tires and the vehicle body is too large, the vehicle posture can be corrected to ensure that both the vehicle body and tires are within the parking space, avoiding scratches or other safety issues. Furthermore, based on the parking space's occupancy status and the smoothness of the road surface, the upcoming driving route can be determined, including how to avoid potholes and how to reach the most convenient parking space. Therefore, by integrating various recognition results, driving strategies can be formulated more accurately than based on a single recognition result, improving the efficiency of road management and road traffic safety.

[0043] Secondly, the present invention also provides a deep learning-based vehicle vision inspection device, the device comprising:

[0044] Image acquisition unit, used to acquire images of the vehicle scene to be identified;

[0045] The recognition unit is used to input the vehicle scene image to be recognized into a preset vehicle vision detection model and generate recognition results; wherein, the recognition results include parking space recognition results, tire recognition results and lane recognition results;

[0046] The planning unit is used to correct the vehicle posture or perform path planning based on the recognition results.

[0047] In this regard, the specific implementation of each module can also refer to the corresponding description of the method embodiment shown in the above embodiments, which will not be repeated here for the sake of simplicity.

[0048] Thirdly, the present invention also provides a computer storage medium storing at least one computer-executable program, wherein when the at least one program is executed by the computer, the computer performs the steps in the deep learning-based vehicle vision inspection method described above.

[0049] In this embodiment, the advantages and beneficial effects of the deep learning-based vehicle vision inspection method have been described above and will not be repeated here. Furthermore, since the deep learning-based vehicle vision inspection method is executed through a computer storage medium, the computer storage medium also has the same advantages and beneficial effects. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating a deep learning-based vehicle vision inspection method according to one embodiment of the present invention.

[0052] Figure 2 This is a schematic diagram illustrating the principle of the YOLO algorithm provided in one embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of the structure of the convolutional neural network involved in the YOLO algorithm provided in one embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of the training process of a parking space recognition model provided in one embodiment of the present invention;

[0055] Figure 5 This is a schematic diagram illustrating the principle of parking space image annotation according to one embodiment of the present invention;

[0056] Figure 6 The images provided are from one embodiment of the present invention and serve as a training set for a tire recognition model.

[0057] Figure 7 This is a schematic diagram of the Yolov5 network structure provided in one embodiment of the present invention;

[0058] Figure 8 This is a schematic diagram of the CSP-DenseNet network structure provided in one embodiment of the present invention;

[0059] Figure 9This is a schematic diagram of the structure of a deep learning-based vehicle vision inspection device provided in one embodiment of the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0061] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative positional relationship and movement of the components in a specific posture. If the specific posture changes, the directional indications will also change accordingly.

[0062] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the use of "and / or" or "and / or" throughout the text includes three parallel solutions. For example, "A and / or B" includes solution A, solution B, or a solution where both A and B are satisfied simultaneously. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0063] Currently, driving strategies are typically formulated based solely on certain vehicle states, such as obstacles or speed, and the accuracy of these recognitions cannot be guaranteed. However, in actual driving strategy formulation and road management, tire position, parking space status, and road conditions all affect management and strategy outcomes. To improve the accuracy and efficiency of vehicle scene recognition, assist in driving strategy formulation, and ultimately enhance road management efficiency, this invention provides a deep learning-based vehicle vision detection method. This method, through a trained vehicle vision detection model, can quickly and accurately identify and generate parking space recognition results, tire recognition results, and lane recognition results for vehicle attitude determination or path planning, thereby improving road safety.

[0064] Please see Figure 1 One embodiment of the present invention provides a deep learning-based vehicle vision inspection method, comprising:

[0065] S10. Obtain the vehicle scene image to be identified;

[0066] S20. Input the vehicle scene image to be identified into the preset vehicle vision detection model to generate recognition results; the recognition results include parking space recognition results, tire recognition results and lane recognition results;

[0067] S30. Correct the vehicle posture or perform path planning based on the recognition results.

[0068] In the current vehicle management process, the identification of vehicle scenarios is usually based on local identification, such as the individual identification of the vehicle's parking position, driving speed, or obstacles. However, the identification of these individual pieces of information does not take into account the impact of tires and potholes on vehicle driving or parking, and cannot obtain identification results quickly and accurately. Furthermore, isolated scenario identification often fails to effectively assist driving strategies.

[0069] In step S10, the vehicle scene image to be identified is first acquired. Typically, a vehicle scene video is acquired first, and then processed to convert the video into frame-by-frame vehicle scene images. Specifically, the vehicle scene images should at least include the vehicle itself and the surrounding road image, while the sky, trees, or pedestrians are considered background distractions.

[0070] In step S20, the vehicle scene image to be identified is input into a preset vehicle vision detection model to generate a recognition result. The preset vehicle vision detection model is primarily trained using a deep learning algorithm.

[0071] Furthermore, the recognition results include parking space recognition results, tire recognition results, and lane recognition results. Parking space recognition results mainly include the number and location of the identified parking spaces; tire recognition results mainly include the angle formed by the tire and the vehicle body; and lane recognition results mainly identify the number and location of potholes on the lane surface.

[0072] Finally, in step S30, the parking space posture can be corrected or a path can be planned based on the parking space recognition results, tire recognition results, and lane recognition results.

[0073] In this embodiment, the vehicle scene image to be identified is first acquired, and then the parking space, tires and lane are identified by a preset vehicle vision detection model to generate corresponding results. Finally, these identification results are combined to assist in planning driving strategies. Compared with existing recognition systems, the trained vehicle vision detection model can quickly and accurately identify different vehicle scenes. Furthermore, by comprehensively considering the identification results of different vehicle scenes, driving strategies can be formulated reasonably and effectively, improving road traffic safety and the efficiency of vehicle management.

[0074] In one embodiment, the vehicle vision detection model includes a parking space recognition model, a tire recognition model, and a lane recognition model; wherein...

[0075] The parking space recognition model and the tire recognition model were trained based on the Yolov3 algorithm;

[0076] The lane recognition model was trained based on the Yolov5 algorithm.

[0077] For detecting targets at close range and those at varying distances, numerous detection algorithms have been proposed, with YOLOv3 being one of them. The YOLOv3 network exhibits good stability for both close and small targets, addressing this issue to some extent. Based on the YOLOv2 model, YOLOv3 modifies certain aspects of the heterogeneous model, employing 3×3 and 1×1 convolutional layers. It references the remaining network, enabling fast connections to be formed directly, reaching up to 53 layers. Furthermore, it utilizes features at multiple scales to detect different scales and adds various scales for object detection. Finally, it replaces the Softmax classifier in the YOLOv2 model with a logistic classifier supporting multiple labeled objects.

[0078] See Figure 2 , Figure 2The principles of the YOLO algorithm are explained. YOLO redefines the object detection problem as a class regression problem. It applies a single convolutional image neural network partitioning to the entire convolutional image, dividing each image grid into multiple grids, and numerically predicting the probability of the image type box and the probability of the image bounding box within each grid partition. For example, a 100x100 rectangular image is uniformly divided into multiple grids, such as 7x7. Then, for each rectangular grid, the network uses four grid descriptors to precisely describe each grid bounding box: 1) the center of the bounding box; 2) the height; 3) the width; and 4) the value mapped to the class to which the object belongs. Furthermore, this prediction algorithm can even be used to predict the probability of movement of an object within a bounding box. If the data center location of an object falls within a grid control unit, this grid control unit system is responsible for automatically detecting the object. Each point in the grid contains the boundaries of multiple points. During this training, it is generally desired that there is only one unique object bounding box between each user object. Therefore, assignment is performed according to two overlap functions, and gradient prediction is performed using the given object variables. Finally, YOLO filters bounding boxes smaller than a threshold using a non-maximum suppression method for each object class, providing the experiments with well-predicted images. The predicted bounding boxes have corresponding probabilities for each object type.

[0079] See Figure 3 , Figure 3 A schematic diagram of the convolutional neural network structure involved in the YOLO algorithm is provided.

[0080] Convolutional Layers: Convolutional layers use convolution to reduce the dimensionality of input images and extract features. The computation function of convolution is a linear convolution calculation, and the function that needs to activate the validation kernel in a neural network is a non-linear connection function. Therefore, it is quite similar to the function structure of a fully linearly connected network. Researchers only need to add an activation function, and commonly used functions include Sigmoid, Tanh, and ReLU.

[0081] Pooling layers: Pooling layers do not change the depth of a 3D matrix, but they can reduce the matrix size. Pooling layers mimic the human visual nervous system, converting high-resolution images into low-resolution images to avoid excessive dimensionality, which could lead to excessive processing time and missed fits. Pooling layers reduce the number of nodes in the final fully connected layer. This achieves dimensionality reduction, improving computational speed and efficiency; reduces the probability of overfitting, minimizing its occurrence; and reduces sensitivity to image translation and rotation.

[0082] Fully Connected Layers: After processing through multiple layers of the network, the data arrives at the fully connected layers. The final classification result is provided by the last one or two fully connected layers of the convolutional neural network. After several rounds of processing, the information in the image can be considered to have acquired high-level features. We can consider the main work in the previous networks as an automatic extraction of image features. The final classification still requires fully connected layers. However, fully connected layers themselves do not have self-learning capabilities; they only rely on the high-level information transmitted from the upper network structures to learn.

[0083] Output Layer: The output layer is primarily used for classification problems. It is typically located at the bottom of a neural convolutional network. It uses functions to perform classification labeling. Structurally, the output layer can serve as the core of object classification, determining size and orientation, and accurately reproducing the grouping results of each pixel in photo segmentation classification.

[0084] Generally, the YOLO algorithm has the following advantages compared to other deep learning algorithms:

[0085] The recognition speed is fast because it uses a regression method and the framework is simple to construct.

[0086] Predicting based on information from the entire image allows for greater compatibility in identifying image features. In contrast, other translation-based detection frameworks can only make predictions based on local image information.

[0087] In this embodiment, different recognition models were trained using different algorithms. Parking spaces and tires are relatively easier to label and identify than potholes in the road, therefore the Yolov3 algorithm was used.

[0088] The Yolov3 algorithm is used to train and generate a parking space recognition model, which improves recognition efficiency and accuracy compared with existing fuzzy C-means cluster classification systems and recurrent neural network systems for parking space recognition.

[0089] A tire recognition model trained using the YOLOv3 algorithm can accurately identify tire positions, preventing traffic congestion caused by improper parking due to vehicle misalignment, the occupation of multiple public parking spaces, and vehicle scrapes. This significantly improves the efficiency of traffic management and increases the utilization rate of public resources.

[0090] By using the YOLOv5 algorithm to train and generate a lane recognition model, the computational load can be further reduced and the road potholes can be accurately identified, assisting drivers in planning their routes and improving road traffic safety.

[0091] In one embodiment, before inputting the vehicle scene image to be identified into a preset vehicle vision detection model, the method further includes:

[0092] Training the parking space recognition model includes:

[0093] Obtain a first preset number of parking space scene images;

[0094] The vacant parking spaces, occupied parking spaces, and parked vehicles in the parking scene images are labeled to generate the first training set; among them, parked vehicles include at least two parking directions;

[0095] Based on the Darknet framework, the Yolov3 algorithm is trained using the first training set to generate a parking space recognition model.

[0096] See Figure 4 , Figure 4 The process for training a parking space recognition model is provided.

[0097] First, select the basic operating system. In this embodiment, Ubuntu is preferred. Ubuntu is based on the Linux operating system and is cleaner than Windows. The PC configuration is as follows: Intel i7-7700HQ processor, 16GB of RAM, and a GTX-1070 GPU with 8GB of video memory. The development environment is Ubuntu 18.04, and the required setup includes CUDA-10.02 (for calling the GPU in deep learning), OpenCV 3.4.5 (for displaying computation results in real time), and a Yolov3 network detection model built on the Darknet framework.

[0098] Then select the labeled objects, create a dataset based on the labeled images, train the Yolov3 algorithm using the dataset, judge the model's recognition accuracy based on the training results, and if the recognition accuracy does not meet the requirements, readjust the data and retrain until the parking space recognition model can meet the recognition accuracy.

[0099] In this embodiment, the advantages of choosing to build the Darknet framework include:

[0100] 1) Easy program installation: The required components (CUDA, cuDNN, OpenCV, etc.) can be selected directly from the Makefile folder and compiled quickly.

[0101] 2) No add-ins required: The entire framework is written in C and does not require any other libraries. The authors of OpenCV have also written functions that can replace it.

[0102] 3) The framework has a clear structure, making it easy to modify and view the source code. The functions used to define the detection and to classify can be found in the example folder, and the basic files of the framework are all in the src folder.

[0103] 4) User-friendly Python interface: Although the framework is designed based on the C language, it also provides a Python interface, which can be used to directly detect the relevant models.

[0104] 5) Easy to move to other terminals: The Darknet framework is extremely easy to move to the local computer, and the framework can automatically select the processor or graphics card to use according to the computer's situation. It is especially suitable for local terminal deployment of detection and recognition tasks. Darknet is a lightweight pure C framework designed for YOLO and is the best platform to cooperate with YOLO, allowing users to experience YOLO functions.

[0105] Please see Figure 5 , Figure 5 A schematic diagram illustrating the principle of image annotation in this embodiment is provided. Figure 5 Three scenarios are provided, each containing a different number of parking spaces and parked vehicles. The shapes of the parking spaces also vary, as do the number of vacant and occupied spaces. Because the dataset includes both parking space shapes and parking directions, it can improve the model's generalization ability during training.

[0106] It should be noted that, in order to improve recognition accuracy, this embodiment refines the granularity of object labeling, labeling according to the status of parking spaces. For occupied parking spaces, both the space and the car are labeled together; cars not in parking spaces are not labeled. Because parking spaces are divided into occupied and vacant spaces, if only the space and vehicle are labeled without distinguishing between their usage status, it will be impossible to determine whether a parking space is vacant during subsequent recognition, thus failing to help drivers plan their routes correctly to find a suitable parking space. Therefore, this embodiment employs refined labeling, enabling the model to recognize the vehicle and parking space usage status during training.

[0107] Preferably, the software Labelimg can be used to save image processing annotations. After the processing is complete, an XML file corresponding to the image annotations will be automatically regenerated. The annotated file will have the same filename as the annotation file in the original image to prevent annotation format errors. The obtained dataset can be processed by Labelimg to obtain an XML file, which stores various information about the bounding boxes (image size and position parameters of each bounding box).

[0108] In one specific implementation, the dataset with annotations for the three scenes totaled 1753 images, which were then used to train the model. Since the experiment required the identification of two objects, the number of training iterations was set to 4000, taking 1.649822 seconds, with a total of 26784 training images.

[0109] It is understood that the parking space recognition model in this embodiment can also be applied to parking lot monitoring rooms and parking lot entrance guidance locations. With slight modifications, the parking lot image can be updated in real time according to the parking space situation. Using it in the parking lot monitoring room can improve the manager's monitoring of parking spaces; using it at the parking lot entrance can allow people coming to park to have a more intuitive understanding of the parking space situation and to choose their parking space more conveniently.

[0110] Therefore, this embodiment uses different parking space scene images. Detailed annotation information accurately identifies the usage status of parking spaces, assisting car owners in determining the accurate information about the parking space. Training the Yolov3 algorithm based on the Darknet framework allows for faster training of the parking space recognition model. A more comprehensive dataset enhances the model's generalization ability.

[0111] In one embodiment, before inputting the vehicle scene image to be identified into a preset vehicle vision detection model, the method further includes:

[0112] Training the tire recognition model includes:

[0113] Get a second preset number of parked vehicle images;

[0114] Label the tires in the images of parked vehicles to generate a second training set; wherein the tires and the vehicle body form at least two different angles.

[0115] Based on the Darknet framework, the Yolov3 algorithm is trained using a second training set to generate a tire recognition model.

[0116] When training a tire recognition model, the quality of the dataset directly affects the training effect of the network model. In this embodiment, it is necessary to screen and correct the sample images collected from various scenarios. Since the object to be detected is vehicle wheels, the target is relatively simple and the types are not very different. However, due to the influence of lighting and environmental factors, there may be obstacles such as occlusion. Therefore, it is necessary to collect wheel images of various types and from different perspectives.

[0117] During training, the image annotation information and the images themselves are input into the network. The annotation information includes the target category ID and the coordinates of the four corner points of the target bounding box. Labelimg was used to annotate the selected images. Labelimg is a visual image labeling tool that is simple to use and easy to install. It can obtain the data labels required by different object detection networks such as Faster, R-CNN, YOLO, and SSD. A total of 1200 images were collected. After initial screening, 1085 suitable images were selected for labeling. During annotation, vehicle wheels were used as targets, and other objects were considered background. A rectangular bounding box was used to select the target, and the image was labeled "wheel". All this information needs to be manually labeled and cannot be automated. Therefore, image annotation is a tedious and time-consuming task, requiring a significant investment of manpower. Furthermore, attention must be paid to the format of the generated labeled images; otherwise, XML files may not be generated.

[0118] Each XML file contains complete information about the corresponding image, including the image name, image path, image size, target label category, and the coordinates of the four corner points of the bounding box. Next, the `voc_label.py` file is run to convert the XML files into TXT files. The TXT files record the categories and the positions of the manually labeled bounding boxes. At this point, the dataset preparation is essentially complete. Most of the images are selected for the second training set (e.g., 800 images), and the remaining 200+ images are used as the test set. Figure 6 As shown, the images used for training under ideal conditions are as follows. Then, the Yolov3 algorithm is trained using the second training set based on the Darknet framework, and the trained model is verified using the test set. When the recognition accuracy meets the preset requirements, the final tire recognition model is generated.

[0119] In this embodiment, by marking the angle formed by the tire and the vehicle body, the tire recognition model can accurately identify the position of the tire, thereby determining whether the vehicle body is off-center when parked. Similarly, in this embodiment, the Yolov3 algorithm is trained based on the lightweight Darknet framework, thereby improving the training speed.

[0120] In one embodiment, the tire recognition model can also identify tires in videos. Specifically, real-world dynamic vehicle videos are used as test subjects. A traffic light intersection with high traffic volume, diverse vehicle types, and rapid traffic flow is selected as the video capture location. During the test video, the model can still capture, identify, and label the wheel positions of dynamic vehicles, demonstrating its effectiveness in detecting dynamic data. Therefore, it can be concluded that the model can accurately identify tires regardless of whether they are static images (which are relatively easy to detect) or dynamic videos (which are more challenging). To further improve the recognition accuracy of the tire recognition model, video samples can be used as the training set during training to enhance the model's generalization ability.

[0121] In one embodiment, before inputting the vehicle scene image to be identified into a preset vehicle vision detection model, the method further includes:

[0122] Training the lane recognition model includes:

[0123] Obtain a third preset number of lane scene images;

[0124] The potholes in the lane scene images are labeled to generate the third training set;

[0125] The YOLOv5 algorithm was trained using the third training set to generate a lane recognition model; the YOLOv5 algorithm includes the CSP-Darknet feature network.

[0126] Generally, potholes are more difficult to identify than tires or parking spaces because parking spaces and tires have relatively obvious morphological features, and their coordinates can be determined simply by using a bounding box when marking them. However, potholes are easily affected by weather, background, and lighting, which can change their texture and shape, making them more difficult to identify. To accurately identify potholes, this embodiment mainly uses the Yolov5 algorithm.

[0127] Figure 7 The YOLOv5 network structure is provided. The feature extraction network (backbone) of YOLOv5 is CSP-Darknet, which reduces computation and maintains accuracy compared to YOLOv3. The process of CSP-Darknet is basically borrowed from CSP-DenseNet, the difference being the residual blocks.

[0128] Figure 8The CSP-DenseNet architecture is provided. In the input module, image data processing is performed, such as Mosaic and matrix training. In the backbone module, features from high, medium, and low layers are extracted, and the CSP-DenseNet computation method is used to reduce computational load and improve speed. In the neck module, features from various layers are fused to extract large, medium, and small feature maps, similar to the function of fully connected layers. The final detection part is performed in the head module, applying anchor boxes to the feature maps and generating a final output vector with class probabilities, object scores, and bounding boxes. The loss function calculates the predicted results and compares them with the ground truth loss to achieve backpropagation and update the model parameters.

[0129] The CSP-DenseNet network divides the input into two parts. When the input is a purple feature map, one half is a multi-layered neural network, while the other half selectively skips k layers and directly connects to the later parts. Therefore, skipping the half is equivalent to reducing the amount of computation.

[0130] Common object detection datasets include PASCAL VOC, ImageNet, and COCO.

[0131] The PASCAL VOC dataset has 20 categories, such as people, cats, dogs, boats, cars, televisions, and chairs. The dataset contains 11,530 images for training and validation.

[0132] The ImageNet dataset is characterized by having evaluation data for classification, localization, and detection tasks. The localization task has 1000 categories, and the detection of 200 targets has 470,000 images.

[0133] The COCO dataset contains 200,000 images with over 500,000 object annotations across 80 categories. It is the most comprehensive object detection dataset and the most authoritative, highly regarded, and unique competition in the field of computer vision, bringing together Google, Microsoft, and numerous top universities and outstanding innovative companies from home and abroad.

[0134] The dataset (third training set) in this embodiment is 5,000 publicly shared images from the IEEE BigData 2020 Global Road Damage Detection Challenge, divided into three countries with different levels of infrastructure to increase data diversity. In addition, 300 annotated images from real-world scenarios are added to the dataset, and 50 images are used as a validation set to improve the model's generalization ability.

[0135] Typically, pothole detection presents the following challenges: numerous dark areas, cluttered backgrounds, difficulty in annotation, diverse and irregular shapes, and high detection speed requirements. Therefore, to improve the quality of the training set, it is crucial to capture the detailed features of potholes as clearly as possible when photographing them. Since potholes vary in shape and require recording numerous features, it is necessary to photograph the same target from multiple angles. Although the dataset is pre-set to include many scenes, the data for each scene is insufficient and too rudimentary.

[0136] In this embodiment, the Yolov5 algorithm can further improve the model's detection performance by changing the loss function, the cross-union ratio method, and the detection module structure. By employing CSP-Darknet, computational complexity can be reduced while maintaining accuracy.

[0137] In one possible implementation, the Yolov5 algorithm is trained using a third training set, including:

[0138] Based on the third training set, the Yolov5 algorithm was trained using mosaic training.

[0139] Mosaic training involves scaling and rotating several randomly acquired labeled lane scene images, then piecing them together to form a new lane scene image, which is then used to train the Yolov5 algorithm.

[0140] In this possible implementation method, mosaic training reads several images at a time and stitches four or more images together into one image by scaling, flipping, etc. This kind of training can make the network more generalizable and accurately identify the target in different situations.

[0141] In one possible implementation, training the Yolov5 algorithm using a third training set also includes:

[0142] Based on the third training set, the Yolov5 algorithm is trained using self-adversarial training; self-adversarial training includes a first stage and a second stage.

[0143] The first stage is used to change the size of the labeled lane scene image;

[0144] The second stage is used for training based on the lane scene images with modified dimensions.

[0145] Self-adversarial training is conducted in two distinct phases. In the first phase, the neural network enlarges or reduces the size of the original image instead of modifying the network weights. The advantage of this approach is that the neural network performs a form of self-attack, altering the original image to make it determine that there is no target in the image, and then identifying the target based on the results of the self-adversarial training.

[0146] In the next stage, the neural network is trained on these resized images. The trained neural network then performs normal target detection on the modified images. This self-attack adversarial training increases the network's robustness, meaning that it can still identify targets even when they are blurred or incomplete in the image.

[0147] In this possible implementation, training through such self-attack adversarial methods can increase the robustness of the network and improve the model's recognition ability.

[0148] In one embodiment, correcting vehicle posture or performing path planning based on the recognition results includes:

[0149] Based on the tire recognition results, the angle between the tires and the vehicle body of the parked vehicle is determined. When the angle is determined to be greater than a first threshold, a prompt is triggered to correct the current vehicle posture; or,

[0150] Based on the parking space recognition results and the lane recognition results, determine the number and location of road potholes and vacant parking spaces; and replan the vehicle's driving path based on the number and location of road potholes and vacant parking spaces.

[0151] In this embodiment, various recognition results are primarily used to assist in adjusting the driving strategy. Specifically, tire recognition results can determine whether the current vehicle parking posture is reasonable. When the angle between the tires and the vehicle body is too large, the vehicle posture can be corrected to ensure that both the vehicle body and tires are within the parking space, avoiding scratches or other safety issues. Furthermore, based on the parking space's occupancy status and the smoothness of the road surface, the upcoming driving route can be determined, including how to avoid potholes and how to reach the most convenient parking space. Therefore, by combining various recognition results, a more accurate driving strategy can be formulated compared to a single recognition result, improving the efficiency of road management and road traffic safety.

[0152] Based on the same inventive concept as the method described above, another embodiment of the present invention discloses a deep learning-based vehicle vision inspection device. Please refer to... Figure 9 The present invention provides a deep learning-based vehicle vision inspection device, comprising:

[0153] Image acquisition unit 10 is used to acquire images of the vehicle scene to be identified;

[0154] The recognition unit 20 is used to input the vehicle scene image to be recognized into a preset vehicle vision detection model and generate recognition results; wherein, the recognition results include parking space recognition results, tire recognition results and lane recognition results;

[0155] The planning unit 30 is used to correct the vehicle posture or perform path planning based on the recognition results.

[0156] In the apparatus disclosed in this embodiment, the specific implementation of each module can also refer to the corresponding description of the method embodiment shown in the above embodiment. For the sake of simplicity, it will not be repeated here.

[0157] One embodiment of the present invention also discloses a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps of any of the above-described methods. If the constituent modules of the above-described signal processing apparatus are implemented as software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium.

[0158] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive SSD), etc.

[0159] In summary, by implementing the embodiments of the present invention, raw audio data is acquired, and feature classification is performed on the raw audio data to obtain at least one audio category; features are extracted from the audio categories to obtain feature information; the feature information is converted into corresponding user interface elements; and when modification instructions for the user interface elements are obtained, the raw audio data is adjusted to obtain corrected audio data. The present invention can provide users with a visual audio experience through user interface elements, enabling users to more intuitively perceive and understand the content of the audio data, and to adjust the audio data by modifying the user interface elements.

[0160] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0161] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the electronic devices, apparatuses, etc. described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0162] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A deep learning-based vehicle vision inspection method, characterized in that, The method includes: Obtain the vehicle scene image to be identified; The vehicle scene image to be identified is input into a preset vehicle vision detection model to generate recognition results; wherein, the recognition results include parking space recognition results, tire recognition results, and lane recognition results; the vehicle vision detection model includes a parking space recognition model, a tire recognition model, and a lane recognition model; wherein, the parking space recognition model and the tire recognition model are trained based on the Yolov3 algorithm; the lane recognition model is trained based on the Yolov5 algorithm; Training the tire recognition model includes: Get a second preset number of parked vehicle images; The tires in the images of the parked vehicles are labeled to generate a second training set; wherein the tires and the vehicle body form at least two different angles. Based on the Darknet framework, the Yolov3 algorithm is trained using the second training set to generate the tire recognition model; Training the lane recognition model includes: Obtain a third preset number of lane scene images; The potholes in the lane scene images are labeled to generate a third training set; The YOLOv5 algorithm is trained using the third training set to generate the lane recognition model; the YOLOv5 algorithm includes the CSP-Darknet feature network. The vehicle posture is corrected or a path is planned based on the recognition results.

2. The deep learning-based vehicle vision inspection method as described in claim 1, characterized in that, Before inputting the vehicle scene image to be identified into the preset vehicle vision detection model, the method further includes: Training the parking space recognition model includes: Obtain a first preset number of parking space scene images; The vacant parking spaces, occupied parking spaces, and parked vehicles in the parking scene image are labeled to generate a first training set; wherein the parked vehicles include at least two parking directions; Based on the Darknet framework, the Yolov3 algorithm is trained using the first training set to generate the parking space recognition model.

3. The deep learning-based vehicle vision inspection method as described in claim 1, characterized in that, The step of training the Yolov5 algorithm using the third training set includes: Based on the third training set, the Yolov5 algorithm is trained using mosaic training. The mosaic training involves scaling and rotating several randomly acquired labeled lane scene images, and then piecing them together to form a new lane scene image, in order to train the Yolov5 algorithm.

4. The deep learning-based vehicle vision inspection method as described in claim 1, characterized in that, The step of training the Yolov5 algorithm using the third training set also includes: Based on the third training set, the Yolov5 algorithm is trained using self-adversarial training; the self-adversarial training includes a first stage and a second stage. The first stage is used to change the size of the annotated lane scene image; The second stage is used for training based on the lane scene image with modified dimensions.

5. The deep learning-based vehicle vision inspection method as described in claim 1, characterized in that, The step of correcting the vehicle posture or performing path planning based on the recognition result includes: Based on the tire recognition results, the angle between the tires and the vehicle body of the parked vehicle is determined. When the angle is determined to be greater than a first threshold, a prompt is triggered to correct the current vehicle posture; or, Based on the parking space recognition results and the lane recognition results, determine the number and location of road potholes and vacant parking spaces; and replan the vehicle's driving path based on the number and location of road potholes and vacant parking spaces.

6. A deep learning-based vehicle vision inspection device, characterized in that, The device includes: Image acquisition unit, used to acquire images of the vehicle scene to be identified; The recognition unit is used to input the vehicle scene image to be recognized into a preset vehicle vision detection model and generate recognition results; wherein, the recognition results include parking space recognition results, tire recognition results, and lane recognition results; the vehicle vision detection model includes a parking space recognition model, a tire recognition model, and a lane recognition model; wherein, the parking space recognition model and the tire recognition model are trained based on the Yolov3 algorithm; and the lane recognition model is trained based on the Yolov5 algorithm. Training the tire recognition model includes: Get a second preset number of parked vehicle images; The tires in the images of the parked vehicles are labeled to generate a second training set; wherein the tires and the vehicle body form at least two different angles. Based on the Darknet framework, the Yolov3 algorithm is trained using the second training set to generate the tire recognition model; Training the lane recognition model includes: Obtain a third preset number of lane scene images; The potholes in the lane scene images are labeled to generate a third training set; The YOLOv5 algorithm is trained using the third training set to generate the lane recognition model; the YOLOv5 algorithm includes the CSP-Darknet feature network. The planning unit is used to correct the vehicle posture or perform path planning based on the recognition results.

7. A computer storage medium storing at least one computer-executable program, characterized in that, When the at least one program is executed by the computer, the computer performs the steps of the deep learning-based vehicle vision inspection method according to any one of claims 1 to 5.

Images