A road condition analysis method and device

Abstract

The present invention provides a road condition analysis method and apparatus, including: splicing real-time images, lane driving direction indication images, and road signal lamp images according to lanes of a target intersection to obtain corresponding spliced images, and then splitting the spliced images according to the lanes of the target intersection to obtain split images corresponding to each lane; inputting the spliced image corresponding to the target intersection into a trained vehicle classification model to output a first classification result image, and inputting the split images corresponding to the lanes of the target intersection into the trained vehicle classification model to output second classification result images corresponding to each lane; obtaining a road condition analysis result of the target intersection based on the fused classification result image. Since the global features and local features of the spliced image are utilized in the model processing process, the accuracy of vehicle classification is improved, and further the accuracy of road condition analysis is improved. At the same time, since the vehicle classification results at the lane level are obtained, lane-level road condition analysis can be performed.

Description

Technical Field

[0001] The present invention relates to the technical field of map navigation, and particularly to a road condition analysis method and device. Background Art

[0002] In recent years, with the accelerating growth rate of the number of automobiles, the problems of road congestion and safety in various regions have also become more serious. Therefore, it is of great significance for traffic managers and travelers to obtain the traffic status of each road in a timely and accurate manner.

[0003] Currently, many navigation software will provide a road condition analysis function to provide reference for users' driving and avoid congestion. Most of these navigation software rely on users' reported information to analyze the traffic flow, and then predict the road conditions to determine whether there is a traffic jam.

[0004] However, the road condition analysis results obtained by the above methods are often inaccurate and cannot provide lane-level road condition analysis results. Therefore, how to improve the accuracy of road condition analysis and provide lane-level road condition analysis results is an urgent problem to be solved. Summary of the Invention

[0005] The present invention provides a road condition analysis method and device to solve the problems of low accuracy of road condition analysis and inability to provide lane-level road condition analysis results in the prior art.

[0006] On the one hand, the present invention provides a road condition analysis method, including:

[0007] Obtaining the position information of the current vehicle, and determining at least one target intersection within the preset range of the current vehicle based on the position information;

[0008] Obtaining the real-time image, lane driving direction indication image, and road signal lamp image of each target intersection, splicing the real-time image, lane driving direction indication image, and road signal lamp image according to the arrangement order and arrangement position of the lanes of the target intersection to obtain a corresponding spliced image, and then splitting the spliced image according to the lanes of the target intersection to obtain split images corresponding to each lane;

[0009] Inputting the spliced image corresponding to the target intersection into a trained vehicle classification model to output a first classification result image, and inputting the split images corresponding to the lanes of the target intersection into a trained vehicle classification model to output second classification result images corresponding to each lane, where specific types of vehicles are marked in both the first classification result image and the second classification result images, and the specific types of vehicles are pre-determined vehicles that will cause congestion at the intersection;

[0010] Fusing the first classification result image and the second classification result images to obtain a fused classification result image, and obtaining a road condition analysis result of the target intersection based on the fused classification result image.

[0011] According to a road condition analysis method provided by the present invention, a real-time image, a lane driving direction indication image, and a road signal light image are spliced according to the arrangement order and arrangement position of each lane of a target intersection to obtain a corresponding spliced image, including:

[0012] For each of the lane driving direction indication image and the road signal light image, the image is split by lane to obtain a corresponding first lane information sub-image;

[0013] The width of each first lane information sub-image is scaled to be the same as the width of the corresponding lane in the real-time image to obtain a second lane information sub-image;

[0014] The second lane information sub-images are spliced with the real-time image in the width direction according to the arrangement order and arrangement position of each lane of the target intersection to obtain a corresponding spliced image.

[0015] According to a road condition analysis method provided by the present invention, the first classification result image is a spliced image of specific type vehicles marked by a first annotation box, and the second classification result image is a split image of specific type vehicles marked by a second annotation box;

[0016] Fusing the first classification result image and the second classification result image to obtain a fused classification result image, including:

[0017] Overlapping the second classification result images corresponding to each lane with the lanes corresponding to the first classification result image respectively, and obtaining a third annotation box based on the first annotation box and the second annotation box;

[0018] Obtaining a fused classification based on the overlapped image and each third annotation box.

[0019] According to a road condition analysis method provided by the present invention, the first annotation box and the second annotation box also carry the classification probabilities of the corresponding specific type vehicles;

[0020] Obtaining a third annotation box based on the first annotation box and the second annotation box, including:

[0021] Taking the non-overlapping first annotation box and second annotation box as the corresponding third annotation box respectively, and taking the classification probabilities carried by the non-overlapping first annotation box and second annotation box as the classification probabilities carried by the corresponding third annotation box;

[0022] Taking the union of the overlapping first annotation box and second annotation box to obtain the corresponding third annotation box, and weighting the classification probabilities carried by the overlapping first annotation box and second annotation box according to a preset weight to obtain the classification probability carried by the corresponding third annotation box.

[0023] A road condition analysis method provided by the present invention, the specific type of vehicle includes at least one of the following:

[0024] Vehicles with incorrect driving lanes;

[0025] Vehicles with too slow driving speed;

[0026] Vehicles involved in traffic accidents.

[0027] A road condition analysis method provided by the present invention, the trained vehicle classification model is obtained in the following way:

[0028] Obtain a preset number of real spliced images, and label the vehicles that cut in line to other lanes or whose turn signals do not match the driving direction of the lane in each real spliced image as vehicles with incorrect driving lanes, label the vehicles that are more than a preset distance away from the vehicle in front in each real spliced image as vehicles with too slow driving speed, and label the vehicles with hazard warning lights on, doors not closed or people around the vehicle in each real spliced image as vehicles involved in traffic accidents, to obtain a preset number of spliced image samples;

[0029] Split the preset number of spliced image samples by lane to obtain the preset number of split image samples corresponding to each lane;

[0030] Use the preset number of spliced image samples and the preset number of split image samples corresponding to each lane to train the initial vehicle classification model to obtain the trained vehicle classification model.

[0031] A road condition analysis method provided by the present invention, each training process includes:

[0032] Input any spliced image sample into the initial vehicle classification model to obtain the corresponding first loss value, and input the split image samples corresponding to each lane of any spliced image sample into the initial vehicle classification model to obtain the corresponding second loss values;

[0033] Based on the second preset weight, weight the first loss value and each second loss value to obtain the total loss value, and use the total loss value to adjust the network parameters of the initial vehicle classification model.

[0034] In a second aspect, the present invention also provides a road condition analysis device, including:

[0035] A target intersection determination module, configured to obtain the position information of the current vehicle, and determine at least one target intersection within the preset range of the current vehicle based on the position information;

[0036] An image acquisition module, configured to acquire real-time images, lane driving direction indication images, and road signal lamp images of each target intersection, splice the real-time images, lane driving direction indication images, and road signal lamp images according to the arrangement order and arrangement positions of the lanes of the target intersection to obtain corresponding spliced images, and then split the spliced images according to the lanes of the target intersection to obtain split images corresponding to the lanes;

[0037] A vehicle classification module, configured to input the spliced image corresponding to the target intersection into a trained vehicle classification model to output a first classification result image, and input the split images corresponding to the lanes of the target intersection into the trained vehicle classification model to output second classification result images corresponding to the lanes, wherein specific types of vehicles are marked in both the first classification result image and the second classification result images, and the specific types of vehicles are pre-determined vehicles that can cause traffic congestion at intersections;

[0038] A road condition analysis result acquisition module, configured to fuse the first classification result image and the second classification result image to obtain a fused classification result image, and obtain a road condition analysis result of the target intersection based on the fused classification result image.

[0039] According to the present invention, there is also provided a road condition analysis device. Specifically, the image acquisition module is configured to:

[0040] For each image in the lane driving direction indication image and the road signal lamp image, split the image according to the lanes to obtain corresponding first lane information sub-images;

[0041] Scale the width of each first lane information sub-image to be the same as the width of the corresponding lane in the real-time image to obtain a second lane information sub-image;

[0042] Splice the second lane information sub-images in the width direction according to the arrangement order and arrangement positions of the lanes of the target intersection with the real-time image to obtain corresponding spliced images.

[0043] According to the present invention, there is also provided a road condition analysis device. The first classification result image is a spliced image with specific types of vehicles marked by first annotation frames, and the second classification result image is a split image with specific types of vehicles marked by second annotation frames;

[0044] Specifically, the road condition analysis result acquisition module is configured to:

[0045] Overlap the second classification result images corresponding to the lanes with the lanes corresponding to the first classification result image respectively, and obtain third annotation frames based on the first annotation frames and the second annotation frames;

[0046] Obtain a fused classification based on the overlapped images and each third annotation frame.

[0047] According to the present invention, there is also provided a road condition analysis device, and the first annotation box and the second annotation box also carry the classification probabilities of corresponding specific types of vehicles;

[0048] The road condition analysis result acquisition module is further configured to:

[0049] Take the non-overlapping first annotation box and second annotation box as the corresponding third annotation box respectively, and take the classification probabilities carried by the non-overlapping first annotation box and second annotation box as the classification probabilities carried by the corresponding third annotation box;

[0050] Take the union of the overlapping first annotation box and second annotation box to obtain the corresponding third annotation box, and weight the classification probabilities carried by the overlapping first annotation box and second annotation box according to a preset weight to obtain the classification probabilities carried by the corresponding third annotation box.

[0051] According to the present invention, there is also provided a road condition analysis device, and the specific types of vehicles include at least one of the following:

[0052] Vehicles with incorrect driving lanes;

[0053] Vehicles with too slow driving speed;

[0054] Vehicles involved in traffic accidents.

[0055] According to the present invention, there is also provided a road condition analysis device, and the device further includes a training module for:

[0056] Obtain a preset number of real spliced images, and label the vehicles that cut in line to other lanes or whose turn signals do not match the driving direction of the lane in each real spliced image as vehicles with incorrect driving lanes, label the vehicles that are more than a preset distance away from the vehicle in front in each real spliced image as vehicles with too slow driving speed, and label the vehicles with hazard warning lights, open doors or people around the vehicle in each real spliced image as vehicles involved in traffic accidents, to obtain a preset number of spliced image samples;

[0057] Split the preset number of spliced image samples by lane to obtain the preset number of split image samples corresponding to each lane;

[0058] Use the preset number of spliced image samples and the preset number of split image samples corresponding to each lane to train the initial vehicle classification model to obtain a trained vehicle classification model.

[0059] According to the present invention, there is also provided a road condition analysis device, and specifically, the training module is configured to:

[0060] Input any spliced image sample into the initial vehicle classification model to obtain the corresponding first loss value, and input the split image samples corresponding to each lane of any spliced image sample into the initial vehicle classification model to obtain the corresponding second loss values;

[0061] Based on the second preset weight, weight the first loss value and each second loss value to obtain the total loss value, and use the total loss value to adjust the network parameters of the initial vehicle classification model.

[0062] In a third aspect, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements any of the above road condition analysis methods.

[0063] In a fourth aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements any of the above road condition analysis methods.

[0064] In a fifth aspect, the present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements any of the above road condition analysis methods.

[0065] A road condition analysis method and device provided by the present invention splice the real-time image of a target intersection, the lane driving direction indication image, and the road signal lamp image by lane to obtain the corresponding spliced image, and then split the spliced image by lane to obtain the split image samples corresponding to each lane. Then, use the trained vehicle classification model to process the spliced image and the split image samples of each lane respectively, so as to obtain the first classification result image containing specific types of vehicles in the spliced image, and the second classification result image containing specific types of vehicles in the split image samples of each lane. Fuse the two classification result images to obtain the corresponding fused classification result image, and use this fused result image to obtain the road condition analysis result. Since the global features and local features of the spliced image are utilized in the model processing process, the accuracy of vehicle classification is improved, and thus the accuracy of road condition analysis is improved. At the same time, since the vehicle classification results at the lane level are obtained, lane-level road condition analysis can be performed. Description of the Drawings

[0066] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0067] Figure 1Flow chart of a road condition analysis method provided by the present invention;

[0068] Figure 2 Block diagram of a road condition analysis device provided by the present invention;

[0069] Figure 3 Schematic diagram of the structure of an electronic device provided by the present invention. Detailed implementation manners

[0070] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without making creative efforts shall fall within the protection scope of the present invention.

[0071] An embodiment of the present invention provides a system on which a road condition analysis method depends. The system may include a road condition analysis background server and on-vehicle terminals of one or more vehicles. Among them, software or application programs providing road condition analysis services may be installed on the on-vehicle terminals (such as mobile phones and in-vehicle computers) of the vehicles, and service users (such as drivers) may obtain road condition analysis services by using the software or application programs. Specifically, the service user triggers a road condition analysis request by operating the software or application program on the on-vehicle terminal, and the on-vehicle terminal sends the road condition analysis request to the road condition analysis background server through the network. After receiving the road condition analysis request, the road condition analysis background server first determines a target intersection for which road condition analysis is to be performed, then obtains relevant information (including real-time images, etc.) of the target intersection from a third-party platform (such as a traffic police platform), then performs road condition analysis based on the obtained information, and feeds back the obtained road condition analysis result to the on-vehicle terminal and displays it to the service user. The service user selects whether to drive through the intersection according to the obtained road condition analysis result, or further selects which lane to drive in when driving to the intersection. It can be understood that the above-mentioned road condition analysis background server may be arranged on the vehicle or in the cloud.

[0072] The road condition analysis method in the embodiment of the present invention will be described in detail below.

[0073] Figure 1 Flow chart of a road condition analysis method provided by an embodiment of the present invention. The execution subject of this method may be the aforementioned road condition analysis background server, as Figure 1 shown, this method may include:

[0074] S101, obtain the position information of the current vehicle, and determine at least one target intersection within the preset range of the current vehicle based on the position information.

[0075] Among them, the position information of the current vehicle can be included in the foregoing road condition analysis request and sent to the road condition analysis server by the in-vehicle terminal. The road condition analysis server can determine a preset range according to this position information. For example, a preset range with the position of the current vehicle as the center and a radius of five kilometers can be determined. Since the current vehicle may pass through any intersection within this preset range, all intersections within this preset range can be determined as target intersections, that is, the intersections where road condition analysis is to be performed. It should be noted that the target intersections need to have clear lane divisions, lane driving direction indications and other information.

[0076] Furthermore, after receiving the road condition analysis request, the road condition analysis background server can also obtain one or more of the historical driving trajectory, current driving destination, current driving planned route, etc. of the current vehicle to determine the target intersections. Specifically, the road condition analysis background server can obtain the historical driving trajectory of the current vehicle, count and extract the frequencies of the current vehicle passing through each intersection within the preset range, and determine the preset number of intersections with the highest frequencies as target intersections, and perform road condition analysis in descending order of frequency. The road condition analysis background server can also plan possible driving routes according to the current driving destination, or directly obtain possible driving routes through the current driving planned route, and determine the intersections within the preset range to be passed through in the possible driving routes as target intersections, and perform road condition analysis in ascending order of the distance from the current vehicle.

[0077] Through the setting of the preset range in the above process of confirming the target intersections, the real-time performance of road condition analysis can be improved, and the waste of computing power caused by analyzing too many intersections can also be avoided.

[0078] S102, obtain the real-time image, lane driving direction indication image and road signal lamp image of each target intersection, splice the real-time image, lane driving direction indication image and road signal lamp image according to the arrangement order and arrangement position of the lanes of the target intersection to obtain the corresponding spliced image, and then split the spliced image according to the lanes of the target intersection to obtain the split images corresponding to the respective lanes.

[0079] Among them, the real-time image of the target intersection includes each lane of the target intersection. Specifically, the real-time image is an image of each lane on the road on the possible driving direction side when the current vehicle travels to the target intersection. In order to obtain a more accurate road condition analysis effect, the real-time image may include vehicles within a certain length range of all lanes and the indicator lights on the vehicles. The real-time image can be directly obtained from a certain third-party platform or can be obtained by fusing multiple images from multiple third-party platforms. For example, it can be the fusion of the high-definition camera monitoring screen of the traffic police platform and the in-vehicle shooting pictures uploaded by other individuals or platform users. Specifically, the light pictures of each vehicle can be fused into the corresponding vehicles in the high-definition camera monitoring screen of the traffic police platform in the form of stickers, so that the real-time image has the real-time light information of each vehicle, which is beneficial to further improving the accuracy of subsequent road condition analysis. For example, when determining the vehicle type subsequently, it can be determined whether the vehicle is driving on the correct road through the turning signal situation of the vehicle.

[0080] Among them, the lane driving direction indication image is used to indicate the correct driving directions ("left turn", "right turn", "going straight", "U-turn", etc.) of each lane of the target intersection. The lane driving direction indication image can be generated after obtaining the direction information from the municipal map or can be a real-time captured image of the driving direction sign.

[0081] Among them, the road signal lamp image is used to indicate the traffic signal lamp status ("red light", "green light", "yellow light", etc.) of each lane of the target intersection. The road signal lamp image can be a real-time captured image of the road signal lamp.

[0082] Specifically, when subsequently classifying the vehicles at the target intersection, the image is processed to obtain the intersection traffic information, and then the specific type of vehicle classification (that is, determining the specific type of vehicle) is carried out. The image needs to contain at least the real-time vehicle distribution information of each lane, the driving direction information of each lane, and the real-time traffic signal lamp information of each lane. Therefore, in the embodiment of the present invention, the real-time image of the target intersection, the lane driving direction indication image, and the road signal lamp image are spliced one by one according to the lanes to obtain the corresponding spliced image. Then the obtained spliced image is split according to the lanes to obtain the split images corresponding to each lane.

[0083] S103, input the spliced image corresponding to the target intersection into the trained vehicle classification model, and output the first classification result map. Input the split images corresponding to each lane of the target intersection into the trained vehicle classification model, and output the second classification result maps corresponding to each lane. Among them, the specific type of vehicle is marked in both the first classification result map and the second classification result maps, and the specific type of vehicle is a vehicle that is pre-determined to cause traffic congestion at the intersection.

[0084] Among them, the vehicle classification model can be an existing Vision Transformer, such as Swin, ViT, etc.

[0085] A Vision Transformer is a deep learning model based on Transformer for processing image and video data. Different from traditional Convolutional Neural Networks (CNNs), Vision Transformers use the self-attention mechanism to extract key features in images and have the advantages of a global receptive field and parallel computing.

[0086] The core components of a Vision Transformer are the Encoder and the Decoder. The Encoder converts the image input into a set of vectors, and the Decoder then converts these vectors back into an image output. In the Encoder, each layer transforms and compresses the input image through the self-attention mechanism to extract higher-level features. In the Decoder, each layer uses the features output by the previous layer to reconstruct and restore the detailed information of the image.

[0087] Vision Transformers have made significant progress in tasks such as image classification, object detection, and image generation. For example, in the image classification task, the ViT model (Vision Transformer) uses the self-attention mechanism to extract features and classify images, achieving performance comparable to that of CNNs. In the object detection task, the DETR model (Detection Transformer) uses Transformer for object detection, achieving excellent performance and scalability.

[0088] Among them, ViT (Vision Transformer) is an image classification model based on Transformer for processing image and video data. Different from traditional Convolutional Neural Networks (CNNs), ViT uses the self-attention mechanism to extract key features in images and has the advantages of a global receptive field and parallel computing. Swin can be understood as an improvement of ViT. The core design of Swin is to use the method of Shifted Windows to convert the image input into a series of local window feature representations, and then perform feature extraction and classification through the Transformer Encoder. Specifically, Swin divides the input image into multiple overlapping local windows, and the size and stride of each window can be adjusted according to the actual situation. Then, the features of each local window are interacted and fused through the self-attention mechanism to extract higher-level feature representations.

[0089] In the embodiments of the present invention, the vehicle classification model is used to classify the objects in the image. After training, the embodiments of the present invention can determine whether the vehicle in the input image is a specific type of vehicle.

[0090] It should be noted that there are many types of specific vehicles in the embodiments of the present invention, and they all impede the traffic at intersections and cause traffic jams. For example, the specific type of vehicle can be a vehicle in the wrong driving lane; a vehicle driving too slowly; a vehicle involved in a traffic accident, etc.

[0091] Specifically, in the previous step, the stitched image corresponding to the target intersection and the split images corresponding to each lane of the target intersection are obtained. Although the trained vehicle classification model can directly process the stitched image to obtain the specific type of vehicle at the target intersection, this method can only utilize the global features of the stitched image. Therefore, in order to improve the accuracy of traffic condition analysis, it is necessary to also process the split images corresponding to each lane with the trained vehicle classification model to utilize the local features of the stitched image. Therefore, in the embodiments of the present invention, the stitched image is input into the trained vehicle classification model to obtain the first classification result image, and the split images corresponding to each lane are input into the trained vehicle classification model to obtain the corresponding second classification result images.

[0092] Furthermore, the solution of the embodiments of the present invention can further split the split image corresponding to each lane to obtain more sub-split images, and use the trained vehicle classification model to process these sub-split images to obtain the corresponding third classification result images. Subsequently, the third classification result images can be used to replace the second classification result images to obtain the traffic condition analysis result. This method can further improve the utilization rate of local features, thereby improving the accuracy of traffic condition analysis.

[0093] Furthermore, when obtaining the first classification result image, the input image of the trained vehicle classification model can be the stitched image, or the corresponding split image or sub-split image (corresponding to the training process). In this way, the vehicle classification model can be understood as a multi-task model. One is to obtain the first classification result image according to the split image or sub-split image, and the other is to obtain the second classification result image or the third classification result image according to the split image or sub-split image.

[0094] S104, fuse the first traffic condition analysis classification result image and the second traffic condition analysis classification result image to obtain a fused classification result image, and obtain the traffic condition analysis result of the target intersection for traffic condition analysis based on the fused traffic condition analysis classification result image.

[0095] Specifically, in the first classification result image obtained, each specific type of vehicle at the target intersection obtained by using global features is included. In the second classification result image obtained, each specific type of vehicle at the target intersection obtained by using local features is included. The two result images are fused to obtain a fused classification result, and then the intersection analysis result can be obtained based on the fused classification result. Since the acquisition of specific types of vehicles is carried out lane by lane during the processing, the intersection analysis result can include not only whether the target lane is congested, but also which lanes at the target intersection are congested.

[0096] The solution provided by the present invention splices the real-time image, the lane driving direction indication image, and the road signal lamp image of the target intersection lane by lane to obtain a corresponding spliced image, and then splits the spliced image lane by lane to obtain split images corresponding to each lane. Then, the trained vehicle classification model is used to process the spliced image and the split images of each lane respectively, and then a first classification result image including specific types of vehicles in the spliced image and a second classification result image including specific types of vehicles in the split images of each lane are obtained. The two classification result images are fused to obtain a corresponding fused classification result image, and the road condition analysis result can be obtained by using this fused result image. Since the global features and local features of the spliced image are utilized in the model processing process, the accuracy of vehicle classification is improved, and thus the accuracy of road condition analysis is improved. At the same time, since the lane-level vehicle classification result is obtained, lane-level road condition analysis can be performed.

[0097] In an alternative embodiment of the present invention, splicing the real-time image, the lane driving direction indication image, and the road signal lamp image according to the arrangement order and arrangement position of each lane of the target intersection to obtain a corresponding spliced image includes:

[0098] For each image in the lane driving direction indication image and the road signal lamp image, the image is split lane by lane to obtain corresponding first lane information sub-images;

[0099] The width of each first lane information sub-image is scaled to be the same as the width of the corresponding lane in the real-time image to obtain a second lane information sub-image;

[0100] The second lane information sub-images are spliced with the real-time image in the width direction according to the arrangement order and arrangement position of each lane of the target intersection to obtain a corresponding spliced image.

[0101] Specifically, when splicing the real-time image, the lane driving direction indication image, and the road signal lamp image, it is necessary to splice according to the intersection. Specifically, it is necessary to splice according to the arrangement order and arrangement position of each lane in the target intersection. When the sizes are inconsistent, the corresponding parts of each lane in the lane driving direction indication image and the road signal lamp image need to be scaled.

[0102] In an alternative embodiment of the present invention, the first classification result map is a spliced image of a specific type of vehicle marked by a first annotation box, and the second classification result map is a split image of a specific type of vehicle marked by a second annotation box;

[0103] Fusing the first classification result map and the second classification result map to obtain a fused classification result map, including:

[0104] Overlap the second classification result map corresponding to each lane with the lane corresponding to the first classification result map, and obtain a third annotation box based on the first annotation box and the second annotation box;

[0105] Obtain the fused classification result map based on the overlapped image and each third annotation box.

[0106] Furthermore, the first annotation box and the second annotation box also carry the classification probabilities of the corresponding specific type of vehicle;

[0107] Obtaining a third annotation box based on the first annotation box and the second annotation box includes:

[0108] Take the non-overlapping first annotation box and second annotation box as the corresponding third annotation box respectively, and take the classification probabilities carried by the non-overlapping first annotation box and second annotation box as the classification probabilities carried by the corresponding third annotation box;

[0109] Take the union of the overlapping first annotation box and second annotation box to obtain the corresponding third annotation box, and weight the classification probabilities carried by the overlapping first annotation box and second annotation box according to a first preset weight to obtain the classification probability carried by the corresponding third annotation box.

[0110] Specifically, multiple first annotation boxes are marked in the first type of classification result map. The first annotation box indicates the position of the specific type of vehicle in the spliced image and the corresponding classification probability. The so-called classification probability is the probability that the vehicle in the first annotation box pointed out by the model is a specific type of vehicle. Similarly, multiple second annotation boxes are marked in the second classification result map. The second annotation box indicates the position of the specific type of vehicle in the split image and the corresponding classification probability. In order to take into account the classification results using global features and local features of the model, it is necessary to fuse each second classification result map with the first classification result map. In the fusion process, it involves the fusion of the indication ranges of the classification boxes and the fusion of the classification probabilities.

[0111] Specifically, the first classification result map is actually multiple first annotation boxes added to the stitched image, and the second classification result map is actually multiple second annotation boxes added to the split image. Then, when fusing each first classification result map by lane with the second classification result map, if the first annotation box and the second annotation box do not intersect (i.e., there is no overlapping area between the two), it is directly used as the third annotation box of the fused classification result, and the classification probability it carries is directly used as the classification probability of the corresponding third annotation box. If the first annotation box and the second annotation box overlap, then the union of the coverage ranges of the first annotation box and the second annotation box is taken as the coverage range of the corresponding third annotation box (this range may include multiple specific types of vehicles), and the classification probabilities of the two annotation boxes are weighted to obtain the classification probability carried by the corresponding third annotation box. By this method, it is possible to avoid missing some specific types of vehicles and improve the accuracy of subsequent road condition analysis.

[0112] After obtaining the fused classification result map, the classification probabilities carried by each third annotation box can be obtained, and the vehicles indicated by the third annotation boxes with classification probabilities not less than a preset value are determined as vehicles that will cause congestion. Furthermore, it is possible to determine whether the target intersection is congested and which lanes are congested.

[0113] In an alternative embodiment of the present invention, the specific type of vehicle includes at least one of the following:

[0114] Vehicles with incorrect driving lanes;

[0115] Vehicles with too slow driving speed;

[0116] Vehicles involved in traffic accidents.

[0117] Specifically, vehicles with incorrect driving lanes can be judged by whether the vehicle's turn signal is consistent with the driving direction of the lane. Vehicles with too slow driving speed can be judged by whether the distance between the vehicle and the vehicle in front exceeds a preset distance (this preset distance can be set according to the actual situation, for example, set to 1.5 meters). Vehicles involved in traffic accidents can be judged by whether the vehicle's door is closed or whether there are people around the vehicle. It can be understood that there are many other types of specific vehicles, which can be added according to actual traffic experience to construct a database of specific types of vehicles.

[0118] In an alternative embodiment of the present invention, the trained vehicle classification model is obtained in the following manner:

[0119] Obtain a preset number of real spliced images, and label the vehicles that cut in line to other lanes or have turn signals inconsistent with the driving direction of the lane in each real spliced image as vehicles with incorrect driving lanes, label the vehicles that are more than a preset distance away from the vehicle in front in each real spliced image as vehicles with slow driving, and label the vehicles with hazard lights, unclosed doors or people around the vehicle in each real spliced image as vehicles involved in traffic accidents, so as to obtain a preset number of spliced image samples;

[0120] Split the preset number of spliced image samples by lane to obtain the preset number of split image samples corresponding to each lane;

[0121] Use the preset number of spliced image samples and the preset number of split image samples corresponding to each lane to train the initial vehicle classification model to obtain a trained vehicle classification model.

[0122] Furthermore, each training process includes:

[0123] Input any spliced image sample into the initial vehicle classification model to obtain a corresponding first loss value, and input the split image samples corresponding to each lane of any spliced image sample into the initial vehicle classification model to obtain corresponding second loss values;

[0124] Based on a second preset weight, weight the first loss value and each second loss value to obtain a total loss value, and use the total loss value to adjust the network parameters of the initial vehicle classification model.

[0125] Specifically, in the model training process of the embodiments of the present invention, since the model uses two types of input images, two types of loss values are required. One is the loss value when classifying vehicles using global features, and the other is the loss value when classifying vehicles using local features. And these two loss values are weighted and summed, and then the obtained total loss value is used to adjust the model parameters until a trained model is obtained.

[0126] Figure 2 The structural block diagram of a road condition analysis device provided by the present invention is as Figure 2 shown. The device may include: a target intersection determination module 201, an image acquisition module 202, a vehicle classification module 203, and a road condition analysis result acquisition module 204, where:

[0127] The target intersection determination module 201 is used to obtain the position information of the current vehicle and determine at least one target intersection within the preset range of the current vehicle based on the position information;

[0128] The image acquisition module 202 is used to acquire the real-time image, lane driving direction indication image, and road signal light image of each target intersection, splice the real-time image, lane driving direction indication image, and road signal light image according to the arrangement order and position of each lane of the target intersection to obtain a corresponding spliced image, and then split the spliced image according to each lane of the target intersection to obtain a split image corresponding to each lane;

[0129] The vehicle classification module 203 is used to input the spliced image corresponding to the target intersection into a trained vehicle classification model to output a first classification result map, and input the split images corresponding to each lane of the target intersection into a trained vehicle classification model to output a second classification result map corresponding to each lane. Among them, the specific type of vehicle is marked in both the first classification result map and the second classification result map, and the specific type of vehicle is a vehicle that will cause intersection congestion determined in advance;

[0130] The road condition analysis result acquisition module 204 is used to fuse the first classification result map and the second classification result map to obtain a fused classification result map, and obtain the road condition analysis result of the target intersection based on the fused classification result map.

[0131] The solution provided by the present invention splices the real-time image, lane driving direction indication image, and road signal light image of the target intersection according to the lane to obtain a corresponding spliced image, and then splits the spliced image according to the lane to obtain a split image corresponding to each lane. Then, the trained vehicle classification model is used to process the spliced image and the split images of each lane respectively, so as to obtain a first classification result map including the specific type of vehicle in the spliced image, and a second classification result map including the specific type of vehicle in the split images of each lane. The two classification result maps are fused to obtain a corresponding fused classification result map, and the road condition analysis result can be obtained by using the fused result map. Since the global features and local features of the spliced image are utilized in the model processing process, the accuracy of vehicle classification is improved, and then the accuracy of road condition analysis is improved. At the same time, since the lane-level vehicle classification result is obtained, lane-level road condition analysis can be performed.

[0132] According to another road condition analysis device provided by the present invention, the image acquisition module is specifically used for:

[0133] For each image in the lane driving direction indication image and the road signal light image, the image is split according to the lane to obtain a corresponding first lane information sub-image;

[0134] The width of each lane information sub-image is scaled to be the same as the width of the corresponding lane in the real-time image to obtain a second lane information sub-image;

[0135] Splice each second-lane information sub-graph with the real-time image in the width direction according to the arrangement order and position of each lane of the target intersection to obtain the corresponding spliced image.

[0136] According to a traffic condition analysis device provided by the present invention, the first classification result graph is a spliced image of specific type vehicles marked by a first annotation box, and the second classification result graph is a split image of specific type vehicles marked by a second annotation box;

[0137] The traffic condition analysis result acquisition module is specifically used for:

[0138] Overlap the second classification result graphs corresponding to each lane with the lane corresponding to the first classification result graph respectively, and obtain a third annotation box based on the first annotation box and the second annotation box;

[0139] Obtain a fusion classification based on the overlapped image and each third annotation box.

[0140] According to a traffic condition analysis device provided by the present invention, the first annotation box and the second annotation box also carry the classification probabilities of the corresponding specific type vehicles;

[0141] The traffic condition analysis result acquisition module is further used for:

[0142] Take the non-overlapping first annotation box and second annotation box as the corresponding third annotation box respectively, and take the classification probabilities carried by the non-overlapping first annotation box and second annotation box as the classification probabilities carried by the corresponding third annotation box;

[0143] Take the union of the overlapping first annotation box and second annotation box to obtain the corresponding third annotation box, and weight the classification probabilities carried by the overlapping first annotation box and second annotation box according to a preset weight to obtain the classification probability carried by the corresponding third annotation box.

[0144] According to a traffic condition analysis device provided by the present invention, the specific type vehicles include at least one of the following:

[0145] Vehicles with incorrect driving lanes;

[0146] Vehicles with too slow driving speed;

[0147] Vehicles involved in traffic accidents.

[0148] According to a traffic condition analysis device provided by the present invention, the device further includes a training module for:

[0149] Obtain a preset number of real spliced images, and label the vehicles that cut in line to other lanes or whose turn signals do not match the driving direction of the lane in each real spliced image as vehicles with incorrect driving lanes, label the vehicles that are more than the preset distance away from the vehicle in front in each real spliced image as vehicles with slow driving, and label the vehicles with hazard lights on, doors not closed, or people around the vehicle in each real spliced image as vehicles involved in traffic accidents, to obtain a preset number of spliced image samples;

[0150] Split the preset number of spliced image samples by lane to obtain the preset number of split image samples corresponding to each lane;

[0151] Use the preset number of spliced image samples and the preset number of split image samples corresponding to each lane to train the initial vehicle classification model to obtain a trained vehicle classification model.

[0152] According to a road condition analysis device provided by the present invention, the training module is specifically used for:

[0153] Input any spliced image sample into the initial vehicle classification model to obtain a corresponding first loss value, and input the split image samples corresponding to each lane of any spliced image sample into the initial vehicle classification model to obtain corresponding second loss values;

[0154] Weight the first loss value and each second loss value based on a second preset weight to obtain a total loss value, and use the total loss value to adjust the network parameters of the initial vehicle classification model.

[0155] Figure 3 Illustrate a schematic physical structure diagram of an electronic device, such as Figure 3As shown in the figure, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communication bus 340. Among them, the processor 310, the communications interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logical instructions in the memory 330 to execute a road condition analysis method, which includes: obtaining the position information of the current vehicle, and determining at least one target intersection within the preset range of the current vehicle based on the position information; obtaining the real-time image, the lane driving direction indication image, and the road signal light image of each target intersection, splicing the real-time image, the lane driving direction indication image, and the road signal light image according to the arrangement order and arrangement position of the lanes of the target intersection to obtain a corresponding spliced image, and then splitting the spliced image according to the lanes of the target intersection to obtain split images corresponding to each lane; inputting the spliced image corresponding to the target intersection into a trained vehicle classification model to output a first classification result image, and inputting the split images corresponding to the lanes of the target intersection into the trained vehicle classification model to output second classification result images corresponding to each lane, where specific types of vehicles are marked in both the first classification result image and the second classification result images, and the specific types of vehicles are pre-determined vehicles that cause congestion at intersections; fusing the first classification result image and the second classification result images to obtain a fused classification result image, and obtaining a road condition analysis result of the target intersection based on the fused classification result image.

[0156] In addition, when the logical instructions in the above-mentioned memory 330 are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0157] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the road condition analysis method provided by each of the above methods. The method includes: obtaining the position information of the current vehicle, and determining at least one target intersection within the preset range of the current vehicle based on the position information; obtaining the real-time image, lane driving direction indication image, and road signal light image of each target intersection, splicing the real-time image, lane driving direction indication image, and road signal light image according to the arrangement order and arrangement position of each lane of the target intersection to obtain a corresponding spliced image, and then splitting the spliced image according to each lane of the target intersection to obtain split images corresponding to each lane; inputting the spliced image corresponding to the target intersection into a trained vehicle classification model to output a first classification result image, and inputting the split images corresponding to each lane of the target intersection into the trained vehicle classification model to output second classification result images corresponding to each lane, wherein specific types of vehicles are marked in both the first classification result image and the second classification result images, and the specific types of vehicles are pre-determined vehicles that cause intersection congestion; fusing the first classification result image and the second classification result images to obtain a fused classification result image, and obtaining a road condition analysis result of the target intersection based on the fused classification result image.

[0158] On another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the road condition analysis method provided by each of the above methods. The method includes: obtaining the position information of the current vehicle, and determining at least one target intersection within the preset range of the current vehicle based on the position information; obtaining the real-time image, lane driving direction indication image, and road signal light image of each target intersection, splicing the real-time image, lane driving direction indication image, and road signal light image according to the arrangement order and arrangement position of each lane of the target intersection to obtain a corresponding spliced image, and then splitting the spliced image according to each lane of the target intersection to obtain split images corresponding to each lane; inputting the spliced image corresponding to the target intersection into a trained vehicle classification model to output a first classification result image, and inputting the split images corresponding to each lane of the target intersection into the trained vehicle classification model to output second classification result images corresponding to each lane, wherein specific types of vehicles are marked in both the first classification result image and the second classification result images, and the specific types of vehicles are pre-determined vehicles that cause intersection congestion; fusing the first classification result image and the second classification result images to obtain a fused classification result image, and obtaining a road condition analysis result of the target intersection based on the fused classification result image.

[0159] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative work.

[0160] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A road condition analysis method, characterized in that, Including: Obtain the position information of the current vehicle, and determine at least one target intersection within the preset range of the current vehicle based on the position information; Obtain the real-time image, lane driving direction indication image, and road signal light image of each target intersection, splice the real-time image, the lane driving direction indication image, and the road signal light image according to the arrangement order and arrangement position of the lanes of the target intersection to obtain a corresponding spliced image, and then split the spliced image according to the lanes of the target intersection to obtain split images corresponding to the lanes; Input the spliced image corresponding to the target intersection into the trained vehicle classification model to output a first classification result image, and input the split images corresponding to the lanes of the target intersection into the trained vehicle classification model to output second classification result images corresponding to the lanes. Among them, specific type vehicles in the corresponding input images are marked in both the first classification result image and the second classification result image, and the specific type vehicles are vehicles that are pre-determined to cause intersection congestion; Fuse the first classification result image and the second classification result image to obtain a fused classification result image, and obtain the road condition analysis result of the target intersection based on the fused classification result image.

2. The method according to claim 1, wherein The step of splicing the real-time image, the lane driving direction indication image, and the road signal light image according to the arrangement order and arrangement position of the lanes of the target intersection to obtain a corresponding spliced image includes: For each of the lane driving direction indication image and the road signal light image, split the image by lane to obtain corresponding first lane information sub-images; Scale the width of each first lane information sub-image to be the same as the width of the corresponding lane in the real-time image to obtain a second lane information sub-image; Splice the second lane information sub-images in the width direction with the real-time image according to the arrangement order and arrangement position of the lanes of the target intersection to obtain a corresponding spliced image.

3. The method according to claim 1, wherein The first classification result image is a spliced image of specific type vehicles marked by a first annotation box, and the second classification result image is a split image of specific type vehicles marked by a second annotation box; The step of fusing the first classification result image and the second classification result image to obtain a fused classification result image includes: Overlap the second classification result images corresponding to the lanes with the lanes corresponding to the first classification result image respectively, and obtain a third annotation box based on the first annotation box and the second annotation box; Obtain the fused classification result image based on the overlapped image and each third annotation box.

4. The method according to claim 3, characterized in that, The first annotation box and the second annotation box also carry the classification probabilities of the corresponding specific type vehicles; The step of obtaining the third annotation box based on the first annotation box and the second annotation box includes: Take the non-overlapping first annotation box and second annotation box as the corresponding third annotation box respectively, and take the classification probabilities carried by the non-overlapping first annotation box and second annotation box as the classification probabilities carried by the corresponding third annotation box; Take the union of the overlapping first annotation box and the second annotation box to obtain the corresponding third annotation box, and weight the classification probabilities carried by the overlapping first annotation box and the second annotation box according to the first preset weight to obtain the classification probability carried by the corresponding third annotation box.

5. The method according to claim 1, wherein The specific type of vehicle includes at least one of the following: Vehicles with incorrect driving lanes; Vehicles with slow driving speed; Vehicles involved in traffic accidents.

6. The method according to claim 5, characterized in that, The trained vehicle classification model is obtained through the following method: Obtain a preset number of real spliced images, and label the vehicles that cut in line to other lanes or whose turn signals do not match the driving direction of the lane in each real spliced image as the vehicles with incorrect driving lanes, label the vehicles that are more than a preset distance away from the vehicle in front in each real spliced image as the vehicles with slow driving speed, and label the vehicles with hazard warning lights on, doors not closed or people around the vehicle in each real spliced image as the vehicles involved in traffic accidents, to obtain the preset number of spliced image samples; Split the preset number of spliced image samples by lane to obtain the preset number of split image samples corresponding to each lane; Use the preset number of spliced image samples and the preset number of split image samples corresponding to each lane to train the initial vehicle classification model to obtain the trained vehicle classification model.

7. The method according to claim 6, wherein Each training process includes: Input any spliced image sample into the initial vehicle classification model to obtain the corresponding first loss value, and input the split image samples corresponding to each lane of the any spliced image sample into the initial vehicle classification model to obtain the corresponding second loss values; Weight the first loss value and each second loss value based on the second preset weight to obtain the total loss value, and use the total loss value to adjust the network parameters of the initial vehicle classification model.

8. A road condition analysis device, characterized in that, It includes: A target intersection determination module, configured to obtain the position information of the current vehicle, and determine at least one target intersection within the preset range of the current vehicle based on the position information; An image acquisition module, configured to acquire the real-time image, lane driving direction indication image, and road signal lamp image of each target intersection, splice the real-time image, the lane driving direction indication image, and the road signal lamp image according to the arrangement order and arrangement position of each lane of the target intersection to obtain the corresponding spliced image, and then split the spliced image by each lane of the target intersection to obtain the split images corresponding to each lane; A vehicle classification module, configured to input the spliced image corresponding to the target intersection into the trained vehicle classification model to output a first classification result map, and input the split images corresponding to each lane of the target intersection into the trained vehicle classification model to output second classification result maps corresponding to each lane, wherein the specific type of vehicle is marked in both the first classification result map and the second classification result map, and the specific type of vehicle is a vehicle that is pre-determined to cause congestion at the intersection. The road condition analysis result acquisition module is used to fuse the first classification result map and the second classification result map to obtain a fused classification result map, and obtain the road condition analysis result of the target intersection based on the fused classification result map.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, the method described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the method described in any one of claims 1 to 7 is implemented.

Images