Road scene recognition method and device

Abstract

The present application relates to the field of satellite navigation technology, and discloses a road scene recognition method and a device thereof, the method comprising: collecting road condition video data and positioning log files, preprocessing the road condition video data to obtain image data aligned with the timestamp of the positioning log files, preprocessing the positioning log files and extracting the log data; using deep learning technology to perform scene recognition on the image data, and obtaining scene information corresponding to each frame of the image, including date, timestamp, scene category and confidence; performing map matching on each frame of the image based on the log data, and obtaining map matching information corresponding to each frame of the image, including date, timestamp and road type; using semantic segmentation to detect whether there are large vehicles and / or traffic signs in each frame of the image and obtaining adjacent road information, including the locations of large vehicles and traffic signs; fusing and outputting the scene information, map matching information and adjacent road information for evaluating positioning performance.

Description

Technical Field

[0001] The present application relates to the field of satellite navigation technology, and in particular to a road scene recognition method and device thereof. Background Art

[0002] At present, in the evaluation of the high-precision positioning performance of autonomous driving scenarios, being able to accurately analyze different usage scenarios and match different indicators has become an important topic. The traditional method is to use satellite information to judge the usage scenario, such as the number of satellites and altitude angles. This can achieve a certain distinction purpose, but because the distribution of satellites at different time points cannot be determined and unified, different scenes cannot be distinguished intuitively. Another solution is to determine an entire road section as the same scene. However, this is obviously not in line with the actual situation. The actual situation is much more complicated on a road. Therefore, in practical problems, an accurate scene recognition and distinction method is urgently needed to improve the accuracy of scene recognition while ensuring the distinction of as many scenes as possible, as a unified evaluation standard for the high-precision positioning performance of high-precision positioning scenarios.

[0003] Existing technologies often only perform scene recognition on a single image, that is, for a certain image, one or several scenes are obtained through deep learning feature extraction methods. However, for lane-level navigation and other requirements that require high positioning scenes and accuracy, it is impossible to output complete scene recognition. On the other hand, existing technologies often cannot solve the problem of distinguishing vehicles driving on and off elevated roads. Summary of the invention

[0004] The purpose of this application is to provide a road scene recognition method and device thereof, which adopts a scene recognition method that integrates deep learning scene recognition, map matching and semantic segmentation to achieve the purpose of outputting accurate scene recognition results.

[0005] In one embodiment of the present application, a road scene recognition method is disclosed, comprising:

[0006] Collecting road condition video data and positioning log files, preprocessing the road condition video data to obtain image data aligned with the timestamp of the positioning log files, preprocessing the positioning log files and extracting log data from the positioning log files, wherein the log data includes time information, coordinate information and vehicle speed information;

[0007] Using deep learning technology to perform scene recognition on the image data to obtain scene information corresponding to each frame of the image, the scene information including date, timestamp, scene category and confidence level;

[0008] Performing map matching on each frame of the image based on the log data to obtain map matching information corresponding to each frame of the image, the map matching information including date, timestamp and road type;

[0009] Using semantic segmentation to detect whether there is a large vehicle and / or a traffic sign in each frame of the image, and obtaining adjacent road information corresponding to each frame of the image, wherein the adjacent road information includes the positions of the large vehicle and the traffic sign;

[0010] The scene information, the map matching information and the adjacent road information are integrated and output for positioning performance evaluation.

[0011] In a preferred example, the step of preprocessing the traffic video data to obtain image data aligned with the timestamp of the positioning log file further includes:

[0012] The traffic video data is played frame by frame, and when the number of skipped frames is equal to the quotient of the acquisition frequency of the traffic video data and the acquisition frequency of the positioning log file, the current frame image is saved.

[0013] In a preferred embodiment, the step of preprocessing the positioning log file further includes:

[0014] Determining the integrity of each line of data in one or more positioning log files, deleting incomplete line data, and merging the one or more positioning log files;

[0015] The date, time stamp, coordinate information and vehicle speed information are extracted from the merged positioning log file.

[0016] In a preferred embodiment, the step of using semantic segmentation to detect whether there is a large vehicle and / or a traffic sign in each frame of the image, and obtaining the adjacent road information corresponding to each frame of the image, further comprises:

[0017] Using semantic segmentation to detect whether there are large vehicles and / or traffic signs in each frame of the image;

[0018] The positions of the large vehicle and the traffic sign are determined according to the pixel sizes of the large vehicle and the traffic sign, and the information of the large vehicles and traffic signs in front of the current position, in the left adjacent lane and in the right adjacent lane is retained.

[0019] In a preferred embodiment, the method further comprises:

[0020] Traversing the map matching information;

[0021] When the map matching information contains a certain number of consecutive road types of the same type, and 1 to 2 road types different from the same road types appear among the certain number of consecutive road types of the same type, the 1 to 2 road types are modified to be the same road types as the certain number of consecutive road types of the same type.

[0022] In a preferred embodiment, the method further comprises:

[0023] Traversing the scene information;

[0024] When the scene information contains a certain number of continuous scene categories, the number of scene categories with occlusion under viaduct in the certain number of continuous scene categories is greater than a predetermined threshold, and the road type at the time corresponding to the certain number of continuous scene categories is a highway or viaduct, the road type at the time corresponding to the certain number of continuous scene categories is modified to an ordinary urban road.

[0025] In a preferred example, the scene categories include any one or more of the following: bilateral occlusion, unilateral occlusion, temporary full occlusion, open space, traffic sign occlusion, bilateral mountain road occlusion, unilateral mountain road occlusion, tunnel, occlusion under an overpass, unilateral high-rise building occlusion, unilateral tree shade occlusion, bilateral high-rise building occlusion, bilateral tree shade occlusion, and next to an overpass.

[0026] In a preferred example, the road type includes any one or more of the following: expressway, elevated road, ordinary urban road and ramp.

[0027] In a preferred example, the map matching information also includes information related to road types, and the information related to road types includes any one or more of the following: whether it is in a service area, whether it is in a toll station, and whether it is next to a high-voltage transmission line.

[0028] In a preferred example, the road condition video data is collected by multiple driving recorders.

[0029] In another embodiment of the present application, a road scene recognition device is disclosed, comprising:

[0030] The acquisition module is used to collect traffic video data and locate log files;

[0031] A preprocessing module, used to preprocess the road condition video data to obtain image data aligned with the timestamp of the positioning log file, preprocess the positioning log file and extract log data from the positioning log file, wherein the log data includes time information, coordinate information and vehicle speed signal;

[0032] A scene recognition module, configured to perform scene recognition on the image data using deep learning technology to obtain scene information corresponding to each frame of the image, wherein the scene information includes a date, a timestamp, a scene category, and a confidence level;

[0033] A map matching module, configured to perform map matching on each frame of the image based on the log data, and obtain map matching information corresponding to each frame of the image, wherein the map matching information includes a date, a timestamp, and a road type;

[0034] A neighboring road information acquisition module, used to detect whether there is a large vehicle and / or a traffic sign in each frame of the image by using semantic segmentation, and acquire neighboring road information corresponding to each frame of the image, wherein the neighboring road information includes the positions of the large vehicle and the traffic sign;

[0035] The output module is used to fuse and output the scene information, the map matching information and the adjacent road information for positioning performance evaluation.

[0036] This patent proposes a scene recognition solution for high-precision positioning performance evaluation, which has at least the following beneficial effects:

[0037] 1) In the present invention, the scene recognition that integrates deep learning scene recognition, map matching and semantic segmentation technology is adopted, which can accurately identify the road network environment, reduce the misidentification of the results caused by interference factors, and thus improve the accuracy.

[0038] 2) After map matching, questions such as whether the vehicle is on an overpass or passing under the overpass may arise. Through the implementation of a post-processing solution, the impact of interference factors on the results is further reduced, thereby improving the accuracy of scene recognition results.

[0039] 3) For scene recognition for high-precision positioning performance evaluation, the solution integrates multiple recognition results such as deep learning scene recognition, map matching, and semantic segmentation. It can solve the problems of being unable to determine and unify evaluation standards using satellite information, being unable to intuitively distinguish different scenes, and avoiding identifying an entire road section as the same scene, and output more accurate fused scene recognition results.

[0040] A large number of technical features are recorded in the specification of this application, which are distributed in various technical solutions. If all possible combinations of technical features of this application (i.e., technical solutions) are to be listed, the specification will be too long. In order to avoid this problem, the various technical features disclosed in the above-mentioned invention content of this application, the various technical features disclosed in the various embodiments and examples below, and the various technical features disclosed in the accompanying drawings can be freely combined with each other to form various new technical solutions (these technical solutions are all deemed to have been recorded in this specification), unless the combination of such technical features is technically infeasible. For example, in one example, feature A+B+C is disclosed, and in another example, feature A+B+D+E is disclosed, and features C and D are equivalent technical means that play the same role. Technically, only one can be used, and it is impossible to use them at the same time. Feature E can be combined with feature C technically. Then, the solution of A+B+C+D should not be deemed to have been recorded because it is technically infeasible, and the solution of A+B+C+E should be deemed to have been recorded. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flowchart of a road scene recognition method according to the first implementation mode of the present application;

[0042] Figure 2 is a detailed flowchart of a road scene recognition method according to the first embodiment of the present application;

[0043] Figure 3 It is a schematic structural diagram of a road scene recognition device according to the second embodiment of the present application. DETAILED DESCRIPTION

[0044] In the following description, many technical details are provided to help readers better understand the present application. However, those skilled in the art can understand that the technical solution claimed in the present application can be implemented even without these technical details and various changes and modifications based on the following embodiments.

[0045] Description of some concepts:

[0046] Map matching technology: the process of associating the vehicle's location information with the road network of the electronic map and converting the coordinate downsampling sequence into a road network coordinate sequence to obtain the current road type.

[0047] Image semantic segmentation technology: Label the category to which each pixel in the image belongs and divide the image into several areas with similar properties to achieve the classification effect.

[0048] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0049] The first embodiment of the present application relates to a road scene recognition method, the process of which is as follows: Figure 1 As shown, the method comprises the following steps:

[0050] Step 101, collect road condition video data and positioning log files, preprocess the road condition video data to obtain image data aligned with the timestamp of the positioning log file, preprocess the positioning log file and extract log data from the positioning log file, the log data including time information, coordinate information and vehicle speed information.

[0051] In one embodiment, the road condition video data is collected by multiple driving recorders. For example, driving recorders are respectively set in front of the vehicle (for example, on the front windshield), on the left side (for example, on the left door), and on the right side (for example, on the right door), and the video data collected by each driving recorder is respectively obtained for preprocessing. In one embodiment, the positioning log file is obtained by a high-precision positioning module, and centimeter-level positioning results can be obtained.

[0052] In one embodiment, the step of preprocessing the road condition video data to obtain image data aligned with the timestamp of the positioning log file further includes: playing the road condition video data frame by frame, and saving the current frame image when the number of skipped frames is equal to the quotient of the acquisition frequency of the road condition video data and the acquisition frequency of the positioning log file.

[0053] In one embodiment, the step of preprocessing the positioning log file further includes:

[0054] Determine the integrity of each row of data in one or more positioning log files, delete the row data with incomplete row data, and merge the one or more positioning log files;

[0055] Log data is extracted from the merged positioning log file, including: date, time stamp, coordinate information and vehicle speed information.

[0056] Step 102: Use deep learning technology to perform scene recognition on the image data to obtain scene information corresponding to each frame of the image, the scene information including date, timestamp, scene category and confidence. Each frame of the image may be recognized as a variety of scene categories, the scene information records the possible multiple scene categories, and the possibility (confidence) size is recorded respectively.

[0057] In one embodiment, the scene categories include any one or more of the following: bilateral occlusion, unilateral occlusion, short-term full occlusion, open space, traffic sign occlusion, bilateral mountain road occlusion, unilateral mountain road occlusion, tunnel, occlusion under an elevated bridge, unilateral high-rise building occlusion, unilateral tree shade occlusion, bilateral high-rise building occlusion, bilateral tree shade occlusion, and next to an elevated bridge. It should be noted that the scene categories may include but are not limited to the above categories.

[0058] Step 103, map matching is performed on each frame of the image based on the log data to obtain map matching information corresponding to each frame of the image, wherein the map matching information includes a date, a timestamp, and a road type. In one embodiment, the road type includes any one or more of the following: expressway, elevated road, ordinary urban road, and ramp. It should be noted that the road type may include but is not limited to the above types.

[0059] In addition, the map matching information also includes information related to road types, and the information related to road types includes but is not limited to any one or more of the following: whether it is in a service area, whether it is at a toll station, and whether it is next to a high-voltage transmission line.

[0060] Step 104 , using semantic segmentation to detect whether there are large vehicles and / or traffic signs in each frame of the image, and obtaining adjacent road information corresponding to each frame of the image, wherein the adjacent road information includes the positions of the large vehicles and the traffic signs.

[0061] In one embodiment, the step of using semantic segmentation to detect whether there is a large vehicle and / or a traffic sign in each frame of the image and obtaining adjacent road information corresponding to each frame of the image further includes:

[0062] Using semantic segmentation to detect whether there are large vehicles and / or traffic signs in each frame of the image;

[0063] The positions of the large vehicle and the traffic sign are determined according to the pixel sizes of the large vehicle and the traffic sign, and the information of the large vehicles and traffic signs in front of the current position, in the left adjacent lane and in the right adjacent lane is retained.

[0064] It should be understood that semantic segmentation technology can detect the type of object in the image and mark the corresponding area with different colors. The size of the pixel area can indicate the distance of the corresponding object. The closer the distance, the larger the pixel area, and the farther the distance, the smaller the pixel area. Therefore, when the pixel area referring to a large vehicle is greater than a predetermined threshold, it can be considered that the large vehicle is closer to the current position. In addition, the location of the large vehicle is determined according to the installation position of the dash cam that collects the image. For example, the large vehicle that appears in the image collected by the dash cam on the left is located in the left adjacent lane, the large vehicle that appears in the image collected by the dash cam on the right is located in the right adjacent lane, and the large vehicle that appears in the image collected by the dash cam in front is determined to be located in front, left or right according to its position in the image. For example, the area in front of the position is divided into three equal parts according to the angle. The area on the left indicates that the large vehicle is located on the left, the area in the middle indicates that the large vehicle is located in front, and the area on the right indicates that the large vehicle is located on the right.

[0065] In one embodiment, the method further comprises:

[0066] Traversing the map matching information;

[0067] When the map matching information has a certain number of consecutive road types of the same type, and 1 to 2 road types different from the same number of consecutive road types appear among the certain number of consecutive road types of the same type, the 1 to 2 road types are modified to be the same road types as the certain number of consecutive road types of the same type. Furthermore, the 1 to 2 road types are the same road types.

[0068] In this embodiment, the scene category corresponding to the 1 to 2 road types may be occlusion under an elevated bridge. When the same road type is continuous, the road type matching may be wrong, which can be corrected in this embodiment. In addition, when the map matching information may contain multiple 1 to 2 road types, it is modified according to the above method.

[0069] It should be understood that the number of consecutive road types of a certain number can be set according to actual conditions, for example, greater than or equal to 10. In this embodiment, it is not limited to 1 to 2 different types of road types, but can be more, for example, 3 or a small number.

[0070] In one embodiment, the method further comprises:

[0071] Traversing the scene information;

[0072] When the scene information contains a certain number of continuous scene categories, the number of scene categories with occlusion under viaduct in the certain number of continuous scene categories is greater than a predetermined threshold, and the road type at the time corresponding to the certain number of continuous scene categories is a highway or viaduct, the road type at the time corresponding to the certain number of continuous scene categories is modified to an ordinary urban road.

[0073] In this embodiment, the number of scene categories that are continuous may be 20, and the predetermined threshold is 15.

[0074] Step 105: fuse and output the scene information, the map matching information and the adjacent road information for positioning performance evaluation.

[0075] In order to better understand the technical solution of this specification, a specific example is used for illustration below. The details listed in this example are mainly for ease of understanding and are not intended to limit the scope of protection of this application.

[0076] Process reference based on scene recognition method Figure 2 As shown, the main steps include:

[0077] (I) Data preprocessing: In order to meet the needs of subsequent automated scene recognition, the video data collected by the vehicle-mounted equipment and the positioning log files of the corresponding date are first preprocessed to obtain image data corresponding to the timestamp in the video, and the positioning log files are merged to extract the required log data of time information, coordinate information, vehicle speed information, etc.

[0078] (ii) Automated scene recognition: Using deep learning technology, map matching technology and semantic segmentation technology, we can obtain image-based scene recognition results (scene information), coordinate-based map matching results (map matching information) and image-based semantic segmentation results (neighboring road information).

[0079] (III) Map matching post-processing: based on the map matching results, filter and modify the incorrect matching information.

[0080] (IV) Output fusion recognition results.

[0081] Continue to refer Figure 2 As shown, the following are specific instructions for the steps of data preprocessing, automatic scene recognition, map matching post-processing, and outputting fusion recognition results:

[0082] 1. Data preprocessing

[0083] The data used for scene recognition includes image data and positioning log files. In order to align the two files using timestamps, the two types of data need to be preprocessed separately. The specific steps include:

[0084] 1) Video data analysis

[0085] Since the video data for scene recognition is collected at a frequency of 30Hz by a dash cam located on the front windshield, but the frequency of recording coordinates in the log file is 1Hz, it is first necessary to extract the video collected by the dash cam at a frequency of 1Hz. The specific operations are: prepare the video, play the video frame by frame, and when the number of skipped frames is equal to the acquisition frequency of the dash cam, save the image of the current frame and name the image with the current timestamp displayed in the image to facilitate subsequent alignment with the timestamp of the coordinate log file.

[0086] 2) Log file preprocessing

[0087] The log file contains a lot of data, and there may be multiple files. In order to facilitate subsequent map matching, the log file needs to be processed. The specific operation steps include: a) Dirty data processing. When the device is powered off, the data at the end of the file will be incomplete. First, the integrity of each line of data is judged and the incomplete data is deleted; b) Merge. After processing the dirty data, the data of multiple files are merged; c) Content extraction. Extract the date, time, coordinates, speed and other information from the merged data.

[0088] 2. Automated scene recognition

[0089] After completing data preprocessing, scene recognition, map matching and semantic segmentation based on deep learning technology are performed simultaneously. The specific steps are as follows:

[0090] 1) Scene recognition based on deep learning technology

[0091] Since the existing model is not trained for the scene recognition required by this patent, the model must first be trained using a classified scene data set. Before training, it is necessary to determine the scene category that needs to be identified. In the present invention, the scenes that need to be identified through deep learning include but are not limited to one or more of the following: bilateral occlusion, unilateral occlusion, short-term full occlusion, open space, traffic sign occlusion, bilateral mountain road occlusion, unilateral mountain road occlusion, tunnel, occlusion under the viaduct, unilateral high-rise building occlusion, unilateral tree shade occlusion, bilateral high-rise building occlusion, bilateral tree shade occlusion and next to the viaduct, etc. After a certain number of training times, this embodiment uses the ResNet-50 model to detect the scene corresponding to the image and record the result information, including: date, timestamp, scene category and confidence. Among them, the confidence indicates the possibility that the image belongs to a certain scene category (for example, 10%, 30%).

[0092] 2) Map matching

[0093] Since different algorithms are required for different road types, it is necessary to perform map matching based on the latitude and longitude data in the log file to obtain the corresponding road type on the electronic map data source, such as expressways, urban expressways, urban ordinary roads and ramps. At the same time, this patent can also match toll booths, service areas and high-voltage transmission lines from the electronic map data source. In the present invention, the national road network data is first divided according to provinces to reduce the amount of calculation of the algorithm. Then, the distance between the coordinates of each frame and the coordinates in a certain range of maps is calculated, and the nearest point is considered to be the location of the road on the electronic map where the current frame is located, and the map matching information is recorded, including: date, timestamp, road type, whether it is next to a high-voltage transmission line, whether it is a toll booth, whether it is a service area, and vehicle speed information.

[0094] During map matching, this embodiment can match the current road type through a hidden Markov algorithm.

[0095] 3) Semantic Segmentation

[0096] In the scene on the road, large vehicles and traffic signs are also one of the scenes that need attention. This patent adopts semantic segmentation technology and uses a pre-trained model to detect large vehicles. After segmenting the large vehicles, only the information of large vehicles in front of the current driving position, the left adjacent lane, and the right adjacent lane is retained for subsequent fusion. Similarly, the location information of traffic signs is extracted in a similar direction, and the traffic sign information in front of the current driving position, the left adjacent lane, and the right adjacent lane is retained.

[0097] When performing semantic segmentation, this embodiment may select target detection to perform recognition and detection on large vehicles and traffic signs.

[0098] (III) Map matching post-processing

[0099] After obtaining the map matching detection results, two problems will arise: 1) When passing under an overpass to block this scene category, the map matching may match the road above the current driving road, resulting in a road type matching error, which may last for 1 to 2 seconds; 2) When driving on an ordinary urban road under an overpass, the map matching may match the overpass above the current ordinary urban road, resulting in a road type matching error, which may last for a period of time. In response to these two problems, this patent also requires post-processing of the results of the map matching records. The specific steps are as follows:

[0100] In order to solve problem 1), this patent traverses the map matching results. If 1 to 2 different road types appear in the same continuous road type, and these 1 to 2 different road types are the same road type before and after, then these 1 to 2 different road types are determined to be false detections and are modified to road types consistent with the same continuous road type.

[0101] In order to solve problem 2), this patent uses sliding window technology to traverse the results of scene recording. If there are 15 scene categories blocked under viaducts in the scene categories of 20 consecutive data, and the road type in the road matching result of the corresponding timestamp is a highway or viaduct, it is determined that the current road is an ordinary urban road under the viaduct, and the road type of these 20 data is modified to an ordinary urban road.

[0102] (IV) Output fusion recognition results

[0103] Combine the three results obtained from the modified automatic scene recognition, fuse the results from different data sources, and output the fused scene recognition result corresponding to each frame of data. The specific steps are as follows:

[0104] 1) Read the timestamp of each frame of data from the map matching result as a benchmark, and align the timestamps in the deep learning scene recognition result and the semantic segmentation recognition result;

[0105] 2) Get data from each of the three files and record information including date, timestamp, and all identified scenes, including road types and all scenes obtained through map matching, deep learning scene recognition, and semantic segmentation.

[0106] 3) Output the fusion result of scene recognition.

[0107] The second embodiment of the present application relates to a road scene recognition device, the structure of which is as follows: Figure 3 As shown, the road scene recognition device includes: a collection module, a preprocessing module, a scene recognition module, a map matching module, a neighboring road information acquisition module and an output module, wherein:

[0108] The acquisition module is used to collect traffic video data and locate log files.

[0109] The preprocessing module is used to preprocess the road condition video data to obtain image data aligned with the timestamp of the positioning log file, preprocess the positioning log file and extract log data from the positioning log file, wherein the log data includes time information, coordinate information and vehicle speed information.

[0110] The scene recognition module is used to perform scene recognition on the image data using deep learning technology to obtain scene information corresponding to each frame of the image, where the scene information includes date, timestamp, scene category and confidence.

[0111] The map matching module is used to perform map matching on each frame of the image based on the log data to obtain map matching information corresponding to each frame of the image, wherein the map matching information includes a date, a timestamp, and a road type.

[0112] The adjacent road information acquisition module is used to use semantic segmentation to detect whether there are large vehicles and / or traffic signs in each frame of the image, and obtain the adjacent road information corresponding to each frame of the image, wherein the adjacent road information includes the positions of the large vehicles and the traffic signs.

[0113] The output module is used to fuse and output the scene information, the map matching information and the adjacent road information for positioning performance evaluation.

[0114] The first implementation manner is a method implementation manner corresponding to the present implementation manner. The technical details in the first implementation manner can be applied to the present implementation manner, and the technical details in the present implementation manner can also be applied to the first implementation manner.

[0115] It should be noted that those skilled in the art should understand that the implementation functions of each module shown in the above-mentioned embodiment of the road scene recognition device can be understood with reference to the relevant description of the above-mentioned road scene recognition method. The functions of each module shown in the above-mentioned embodiment of the road scene recognition device can be implemented by a program (executable instruction) running on a processor, or by a specific logic circuit. If the above-mentioned road scene recognition device in the embodiment of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM, Read Only Memory), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0116] It should be noted that in the application documents of this patent, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements. In the application documents of this patent, if it is mentioned that an action is performed according to an element, it means that the action is performed at least according to the element, which includes two situations: performing the action only according to the element, and performing the action according to the element and other elements. Expressions such as multiple, multiple, and multiple include 2, 2 times, 2 kinds, and more than 2, more than 2 times, and more than 2 kinds.

[0117] All documents mentioned in this application are considered to be included in the disclosure of this application as a whole, so that they can be used as the basis for modification when necessary. In addition, it should be understood that after reading the above disclosure of this application, those skilled in the art can make various changes or modifications to this application, and these equivalent forms also fall within the scope of protection claimed in this application.

Claims

1. A road scene recognition method, characterized in that: include: Collecting road condition video data and positioning log files, preprocessing the road condition video data to obtain image data aligned with the timestamp of the positioning log files, preprocessing the positioning log files and extracting log data from the positioning log files, wherein the log data includes time information, coordinate information and vehicle speed information; Using deep learning technology to perform scene recognition on the image data to obtain scene information corresponding to each frame of the image, the scene information including date, timestamp, scene category and confidence level; Performing map matching on each frame of the image based on the log data to obtain map matching information corresponding to each frame of the image, the map matching information including date, timestamp and road type; Using semantic segmentation to detect whether there is a large vehicle and / or a traffic sign in each frame of the image, and obtaining adjacent road information corresponding to each frame of the image, wherein the adjacent road information includes the location of the large vehicle and / or the traffic sign; The scene information, the map matching information and the adjacent road information are integrated and output for positioning performance evaluation, wherein the integration includes: reading the timestamp of each frame of data from the map matching result as a reference, and aligning the timestamps in the deep learning scene recognition result and the semantic segmentation recognition result; acquiring data from the scene information, the map matching information and the adjacent road information respectively, and recording information including: date, timestamp and all recognized scenes, including road types and all scenes obtained through map matching, deep learning scene recognition and semantic segmentation; and outputting the fusion result of scene recognition.

2. The road scene recognition method according to claim 1, characterized in that: The step of preprocessing the traffic video data to obtain image data aligned with the timestamp of the positioning log file further includes: The traffic video data is played frame by frame, and when the number of skipped frames is equal to the quotient of the acquisition frequency of the traffic video data and the acquisition frequency of the positioning log file, the current frame image is saved.

3. The road scene recognition method according to claim 1, characterized in that: The step of preprocessing the positioning log file further includes: Determine the integrity of each line of data in one or more positioning log files, delete incomplete line data, and merge the one or more positioning log files; Date, time stamp, coordinate information and vehicle speed information are extracted from the merged one or more positioning log files.

4. The road scene recognition method according to claim 1, characterized in that: The step of using semantic segmentation to detect whether there is a large vehicle and / or a traffic sign in each frame of the image, and obtaining adjacent road information corresponding to each frame of the image, further includes: Using semantic segmentation to detect whether there are large vehicles and / or traffic signs in each frame of the image; The positions of the large vehicle and / or the traffic sign are determined according to the pixel sizes of the large vehicle and / or the traffic sign, and the information of the large vehicles and / or traffic signs in front of the current position, in the adjacent lane on the left, and in the adjacent lane on the right is retained.

5. The road scene recognition method according to claim 1, characterized in that: Also includes: Traversing the map matching information; When the map matching information contains a certain number of consecutive road types of the same type, and 1 to 2 road types different from the same road types appear among the certain number of consecutive road types of the same type, the 1 to 2 road types are modified to be the same road types as the certain number of consecutive road types of the same type.

6. The road scene recognition method according to claim 1, characterized in that: Also includes: Traversing the scene information; When the scene information contains a certain number of continuous scene categories, the number of scene categories with occlusion under viaduct in the certain number of continuous scene categories is greater than a predetermined threshold, and the road type at the time corresponding to the certain number of continuous scene categories is a highway or viaduct, the road type at the time corresponding to the certain number of continuous scene categories is modified to an ordinary urban road.

7. The road scene recognition method according to claim 1, characterized in that: The scene categories include any one or more of the following: bilateral occlusion, unilateral occlusion, short-term full occlusion, open space, traffic sign occlusion, bilateral mountain road occlusion, unilateral mountain road occlusion, tunnel, occlusion under an overpass, unilateral high-rise building occlusion, unilateral tree shade occlusion, bilateral high-rise building occlusion, bilateral tree shade occlusion, and next to an overpass.

8. The road scene recognition method according to claim 1, characterized in that: The road type includes any one or more of the following: expressway, elevated road and urban ordinary road ramp.

9. The road scene recognition method according to claim 1, characterized in that: The map matching information also includes information related to road types, and the information related to road types includes any one or more of the following: whether the road is in a service area, whether the road is in a toll booth, and whether the road is next to a high-voltage transmission line.

10. The road scene recognition method according to claim 1, characterized in that: The road condition video data is collected through multiple driving recorders.

11. A road scene recognition device, characterized in that: include: The acquisition module is used to collect traffic video data and locate log files; A preprocessing module, used to preprocess the road condition video data to obtain image data aligned with the timestamp of the positioning log file, preprocess the positioning log file and extract log data from the positioning log file, wherein the log data includes time information, coordinate information and vehicle speed signal; A scene recognition module, configured to perform scene recognition on the image data using deep learning technology to obtain scene information corresponding to each frame of the image, wherein the scene information includes a date, a timestamp, a scene category, and a confidence level; A map matching module, configured to perform map matching on each frame of the image based on the log data, and obtain map matching information corresponding to each frame of the image, wherein the map matching information includes a date, a timestamp, and a road type; A neighboring road information acquisition module, used to detect whether there is a large vehicle and / or a traffic sign in each frame of the image by using semantic segmentation, and acquire neighboring road information corresponding to each frame of the image, wherein the neighboring road information includes the position of the large vehicle and / or the traffic sign; An output module is used to fuse and output the scene information, the map matching information and the adjacent road information for positioning performance evaluation, wherein the fusion includes: reading the timestamp of each frame of data from the map matching result as a reference, aligning the timestamps in the deep learning scene recognition result and the semantic segmentation recognition result; acquiring data from the scene information, the map matching information and the adjacent road information respectively, and recording information including: date, timestamp and all recognized scenes, including road types and all scenes obtained through map matching, deep learning scene recognition and semantic segmentation; and outputting the fusion result of scene recognition.

Images