A driving environment recognition method and system based on vehicle-mounted camera

Abstract

The present invention discloses a driving environment recognition method and system based on a vehicle-mounted camera, the method comprising: acquiring detection image data and preprocessing to obtain preprocessed image data; inputting the preprocessed image data into a preset driving environment recognition model to determine driving environment recognition data, recognition targets and risk levels of recognition targets; dividing a detection area into multiple sub-detection areas, adjusting the first attention score of the sub-detection area based on an attention mechanism, and analyzing the next detection image data according to the first attention score; comparing the detection image data of adjacent acquisition times to determine newly added targets and disappeared targets; extracting local image features of disappeared targets, adjusting the second attention score of the sub-detection area, and analyzing the next detection image data according to the second attention score. The present invention improves the recognition efficiency of the recognition targets in the driving environment by adjusting the attention score of each sub-recognition area.

Description

Technical Field

[0001] The present application relates to the field of intelligent recognition technology, and more specifically, to a method and system for recognizing a driving environment based on a vehicle-mounted camera. Background Art

[0002] With the rapid development of autonomous driving technology, accurate recognition of the driving environment has become the key to ensuring driving safety. As one of the important sensors of autonomous vehicles, the on-board camera can capture rich road information. However, its recognition effect is easily affected by factors such as lighting conditions, weather changes, and camera pollution. When dealing with complex and changeable driving environments, there are still problems such as low recognition accuracy and poor adaptability. In addition, the existing technology usually performs overall recognition of the image data obtained by the on-board camera to determine the driving environment recognition data, without giving priority to the recognition of movable targets such as pedestrians and vehicles. Considering the uncontrollability of movable targets such as pedestrians and vehicles to driving safety, it may result in a short time left for the central control system, and it is impossible to perform corresponding obstacle avoidance processing in time, which poses certain risks in autonomous driving.

[0003] Therefore, the prior art has defects and is in urgent need of improvement. Summary of the invention

[0004] In view of the above problems, the purpose of the present invention is to provide a driving environment recognition method and system based on a vehicle-mounted camera, which divides the recognition area into multiple sub-recognition areas, and improves the recognition efficiency of the recognition target in the driving environment by adjusting the attention score of each sub-recognition area.

[0005] A first aspect of the present invention provides a method for identifying a driving environment based on a vehicle-mounted camera, comprising:

[0006] Acquire detection image data of the detection area;

[0007] Preprocessing the detected image data to obtain preprocessed image data;

[0008] The pre-processed image data is input into a preset driving environment recognition model, and the driving environment recognition model extracts and classifies the pre-processed image data, outputs driving environment recognition data, and determines the recognition target; the driving environment recognition data includes but is not limited to lane line detection, traffic sign recognition, pedestrian detection, and vehicle detection; the recognition target includes but is not limited to pedestrian targets and vehicle targets;

[0009] Calculating a first coordinate distance between the identified target and the current vehicle to determine a risk level of the identified target;

[0010] Dividing the detection area into a plurality of sub-detection areas, and dividing the detection image data into a plurality of sub-detection image data based on the plurality of sub-detection areas;

[0011] Determine a vehicle moving route of a current vehicle and a target moving route of an identified target based on the detection image data of the continuous plurality of frames;

[0012] Adjusting the first attention score of the sub-detection area based on the attention mechanism, and analyzing the multiple sub-detection image data of the next detection image data in descending order of the first attention scores;

[0013] By comparing the detection image data of adjacent acquisition times, newly added targets and disappeared targets are determined;

[0014] When there is a disappeared target, the local image features of the disappeared target are extracted, and the second attention score of the sub-detection area at each data collection time is adjusted according to the vehicle movement route and the target movement route of the disappeared target. The multiple sub-detection image data of the next detection image data are analyzed in descending order of the second attention scores.

[0015] In this solution, the preprocessing of the detected image data to obtain preprocessed image data includes:

[0016] Performing data enhancement on the detection image data;

[0017] The detected image data is subjected to image preprocessing; the image preprocessing at least includes image denoising and color correction.

[0018] This plan also includes:

[0019] Obtain historical detection image data;

[0020] An analysis is performed based on the historical detection image data to establish a preset driving environment recognition model.

[0021] In this solution, the method of determining the vehicle moving route of the current vehicle and the target moving route of the identified target based on the detection image data of the continuous multiple frames includes:

[0022] By analyzing the detection image data of multiple consecutive frames, determining whether the identified target is a stationary target or a moving target;

[0023] When the identified target is a moving target, drawing a target moving route of the identified target based on the detection image data of the continuous multiple frames;

[0024] Get the vehicle movement route of the current vehicle;

[0025] Analyzing the vehicle moving route of the current vehicle and the target moving route of the identified target to determine the minimum coordinate distance between the current vehicle and the identified target within a preset time interval;

[0026] When the minimum coordinate distance is less than a preset distance threshold, an early warning is issued to the identified target.

[0027] In this solution, the first attention score of the sub-detection area is adjusted based on the attention mechanism, and multiple sub-detection image data of the next detection image data are analyzed in descending order of the first attention scores, including:

[0028] Analyze the vehicle moving route of the current vehicle and the target moving route of the identified target, predict the coordinate position of the identified target in the detection area in the next detection image data, and determine the first sub-detection area occupied by the identified target;

[0029] Improving a first attention score of the first sub-detection area based on an attention mechanism, and reducing first attention scores of other sub-detection areas other than the first sub-detection area;

[0030] Analyzing the sub-detection image data corresponding to each sub-detection area in order from large to small based on the first attention score and from left to right and from top to bottom, and determining the coordinate position of the identified target after movement through feature comparison;

[0031] The first sub-detection area occupied by the identification target is corrected according to the coordinate position of the identification target after the movement, and the first sub-detection area and other sub-detection areas that are not subjected to image analysis are adjusted.

[0032] In this solution, the newly added target and the disappeared target are determined by comparing the detection image data of adjacent acquisition times, including:

[0033] Compare the current detection image data with the previous detection image data, match the recognition targets of the current detection image data with the recognition targets of the previous detection image data in pairs according to the image features, and determine the newly added targets and the disappeared targets;

[0034] The reasons for adding the new target and the reasons for disappearing the disappeared target are determined based on the target movement routes.

[0035] In this solution, when there is a disappeared target, the local image features of the disappeared target are extracted, the second attention score of the sub-detection area at each data collection time is adjusted according to the vehicle movement route and the target movement route of the disappeared target, and multiple sub-detection image data of the next detection image data are analyzed in descending order of the second attention scores, including:

[0036] Extract the last detected image data P n-1one or more local image features of the disappeared target, and determining one or more first coordinates and corresponding feature scores corresponding to the one or more local image features;

[0037] According to the moving route of the vehicle and the moving route of the disappeared target, it is determined that the one or more local image features are in the detection image data P n+m One or more second coordinates corresponding to ; wherein m is a positive integer greater than 1;

[0038] Analyze each second coordinate in turn, and feature-mark a sub-detection area within a preset range of the second coordinate according to a local image feature corresponding to the second coordinate, to determine a second sub-detection area;

[0039] Determine the influence weight of the characteristic mark of the second sub-detection area according to the second coordinate distance from the center coordinate of the second sub-detection area to the corresponding second coordinate;

[0040] Performing weighted calculation on the feature score of the feature marker of the second sub-detection area and the corresponding influence weight to determine the second attention score and the average attention score of the second sub-detection area; the feature score of the feature marker is the feature score of the local image feature corresponding to the feature marker;

[0041] filtering the second sub-detection area whose average attention score is lower than a preset score threshold;

[0042] According to the second attention score from large to small, the detection image data P is n+m The sub-detection image data of each second sub-detection area is analyzed, and when the image features of the sub-detection image data match the local image features of the disappeared target, it is determined that the disappeared target reappears,

[0043] The disappeared object is framed according to the coordinate data of the sub-detection image data and the historical recognition data of the disappeared object, and the recognition data of the disappeared object is determined.

[0044] This plan also includes:

[0045] After the identification data of the disappeared target is determined, the feature marks of the local feature data of the disappeared target in each sub-detection area are cleared, the second attention score and the average attention score of each second sub-detection area are recalculated, and the identification of other disappeared targets continues.

[0046] A second aspect of the present invention provides a vehicle environment recognition system based on a vehicle-mounted camera, comprising:

[0047] A data acquisition module, used to acquire detection image data of the detection area;

[0048] A preprocessing module, used for preprocessing the detection image data to obtain preprocessed image data;

[0049] A driving environment recognition module is used to input the pre-processed image data into a preset driving environment recognition model, the driving environment recognition model performs feature extraction and classification on the pre-processed image data, outputs driving environment recognition data, and determines the recognition target; the driving environment recognition data includes but is not limited to lane line detection, traffic sign recognition, pedestrian detection and vehicle detection; the recognition target includes but is not limited to pedestrian targets and vehicle targets; the first coordinate distance between the recognition target and the current vehicle is calculated to determine the risk level of the recognition target;

[0050] A region segmentation module, used to divide the detection region into a plurality of sub-detection regions, and divide the detection image data into a plurality of sub-detection image data based on the plurality of sub-detection regions;

[0051] The target recognition monitoring module is used to determine the vehicle movement route of the current vehicle and the target movement route of the recognition target based on the detection image data of multiple consecutive frames; adjust the first attention score of the sub-detection area based on the attention mechanism, analyze multiple sub-detection image data of the next detection image data in descending order of the first attention score, and determine the newly added target and the disappeared target by comparing the detection image data of adjacent acquisition times;

[0052] The disappeared target monitoring module is used to extract the local image features of the disappeared target when there is a disappeared target, adjust the second attention score of the sub-detection area at each data collection time according to the vehicle movement route and the target movement route of the disappeared target, and analyze the multiple sub-detection image data of the next detection image data in descending order of the second attention scores.

[0053] The third aspect of the present invention provides a computer-readable storage medium, which includes a driving environment recognition method program based on a vehicle-mounted camera. When the driving environment recognition method program based on a vehicle-mounted camera is executed by a processor, the steps of the driving environment recognition method based on a vehicle-mounted camera as described above are implemented.

[0054] The present invention discloses a driving environment recognition method and system based on a vehicle-mounted camera, the method comprising: acquiring detection image data and preprocessing to obtain preprocessed image data; inputting the preprocessed image data into a preset driving environment recognition model to determine driving environment recognition data, recognition targets and risk levels of recognition targets; dividing a detection area into multiple sub-detection areas, adjusting the first attention score of the sub-detection area based on an attention mechanism, and analyzing the next detection image data according to the first attention score; comparing the detection image data of adjacent acquisition times to determine newly added targets and disappeared targets; extracting local image features of disappeared targets, adjusting the second attention score of the sub-detection area, and analyzing the next detection image data according to the second attention score. The present invention improves the recognition efficiency of the recognition targets in the driving environment by adjusting the attention score of each sub-recognition area. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A flow chart of a vehicle environment recognition method based on a vehicle-mounted camera provided by the present invention is shown;

[0056] Figure 2 A flow chart showing a method for adjusting a first attention score of a sub-detection area provided by the present invention is shown;

[0057] Figure 3 A flow chart showing a method for determining a newly added target and a disappeared target provided by the present invention is shown;

[0058] Figure 4 A block diagram of a vehicle environment recognition system based on a vehicle-mounted camera provided by the present invention is shown. DETAILED DESCRIPTION

[0059] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0060] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0061] Figure 1 A flow chart of a driving environment recognition method based on a vehicle-mounted camera provided by the present invention is shown.

[0062] like Figure 1 As shown, the present invention discloses a driving environment recognition method based on a vehicle-mounted camera, comprising:

[0063] S102, acquiring detection image data of the detection area;

[0064] S104, preprocessing the detected image data to obtain preprocessed image data;

[0065] S106, inputting the preprocessed image data into a preset driving environment recognition model, the driving environment recognition model extracts and classifies the preprocessed image data, outputs driving environment recognition data, and determines the recognition target; the driving environment recognition data includes but is not limited to lane line detection, traffic sign recognition, pedestrian detection, and vehicle detection; the recognition target includes but is not limited to pedestrian targets and vehicle targets;

[0066] S108, calculating the first coordinate distance between the identified target and the current vehicle to determine the risk level of the identified target;

[0067] S110, dividing the detection area into a plurality of sub-detection areas, and dividing the detection image data into a plurality of sub-detection image data based on the plurality of sub-detection areas;

[0068] S112, determining a vehicle moving route of the current vehicle and a target moving route of the identified target based on the detection image data of the continuous multiple frames;

[0069] S114, adjusting the first attention score of the sub-detection area based on the attention mechanism, and analyzing the multiple sub-detection image data of the next detection image data in descending order of the first attention scores;

[0070] S116, determining newly added targets and disappeared targets by comparing detection image data at adjacent acquisition times;

[0071] S118, when there is a disappeared target, extract the local image features of the disappeared target, adjust the second attention score of the sub-detection area at each data collection time according to the vehicle movement route and the target movement route of the disappeared target, and analyze the multiple sub-detection image data of the next detection image data in descending order of the second attention scores.

[0072] According to an embodiment of the present invention, according to the system set time interval, the image data obtained by the vehicle camera is extracted from the image data to obtain the detection image data corresponding to the vehicle camera, and the corresponding data enhancement and image preprocessing steps are performed on the obtained detection image data to obtain preprocessed image data, thereby improving the clarity and details of the detection image data and improving the accuracy of the driving environment recognition. The preprocessed image data is input into the preset driving environment recognition model, and the targets such as lane lines, traffic signs, pedestrians and vehicles are recognized through image feature extraction, comparison and classification, and the driving environment recognition data is output, and pedestrians and vehicles and other targets that may move are determined as recognition targets. The system sets a corresponding risk recognition strategy for each recognition target based on the recognition target type, and adjusts the risk recognition strategy of each recognition target according to the real-time speed of the current vehicle. The risk level of each recognition target is determined by the risk recognition strategy of each recognition target, and the number of risk levels and the first coordinate distance interval corresponding to each risk level are set and dynamically adjusted by the system. For example, when the current vehicle is traveling at a speed below 30km / h, the warning distance for pedestrians is 0.5-2 meters. When the distance between the pedestrian and the first coordinate of the current vehicle is less than 2 meters, the pedestrian enters the warning state and its risk level is determined to be the lowest risk level. When the distance between the pedestrian and the first coordinate of the current vehicle is less than 0.5 meters, the risk level of the pedestrian is determined to be the highest risk level. When the speed of the current vehicle is adjusted to 60km / h, the warning distance for pedestrians is adjusted to 2-8 meters.

[0073] When performing driving environment recognition, due to the uncertainty of the recognition target, the recognition target is recognized first. The detection area is divided into multiple sub-detection areas according to the system preset specifications (the length and width of each sub-detection area are the same), the vehicle movement route of the current vehicle is drawn according to the driving data of the current vehicle, and the target movement route of the recognition target is drawn according to the coordinate data of the recognition target in the continuous multi-frame detection image data. Combined with the system set time interval, the corresponding position of the recognition target in the detection image data obtained at the next acquisition time is predicted, and the first sub-detection area corresponding to the recognition target is determined. The first attention score of the first sub-detection area is increased, and the first attention scores of other sub-detection areas are reduced. The sub-detection image data where the recognition target may exist is analyzed first. Among them, the initial first attention score of each sub-detection area is the same. When the first attention scores of the sub-detection areas are the same, the sub-detection image data corresponding to each sub-detection area are analyzed in order from left to right and from top to bottom.

[0074] When the disappeared target is determined, the local image features of the disappeared target are extracted, combined with the vehicle movement route of the current vehicle and the target movement route of the disappeared target, the coordinate position of the local image features of the disappeared target in the next detection image data to be analyzed is predicted, and multiple second sub-detection areas where the local image features may appear are determined based on a preset range. On the basis of the first sub-detection area, the priority of the second sub-detection area is further improved, and the sub-detection area where the disappeared target may appear is preferentially identified to avoid the situation where the identified target suddenly appears and an early warning cannot be issued in time.

[0075] According to an embodiment of the present invention, preprocessing the detected image data to obtain preprocessed image data includes:

[0076] Perform data enhancement on the detection image data;

[0077] The detection image data is subjected to image preprocessing; the image preprocessing at least includes image denoising and color correction.

[0078] It should be noted that the on-board camera is installed outside the vehicle. Due to the influence of the environment (such as when covered by pollutants such as mud), the clarity of the captured image decreases and it is difficult to reflect the real road conditions. The detection image data is enhanced through adaptive histogram equalization, defogging algorithm and other technologies, and the appropriate image denoising method (such as Gaussian filtering, median filtering, etc.) is selected to remove noise from the detection image data. Color correction is performed by adjusting parameters such as image brightness, contrast and saturation to improve color balance and improve image data quality.

[0079] According to an embodiment of the present invention, it also includes:

[0080] Obtain historical detection image data;

[0081] Based on the analysis of historical detection image data, a preset driving environment recognition model is established.

[0082] It should be noted that the historical detection image data is the detection image data obtained during the historical use of the vehicle-mounted camera. The lane lines, traffic signs, pedestrians, vehicles and other targets in the historical detection image data are annotated through manual labeling and other methods. The annotated historical detection image data is input into the model for training, and the image features corresponding to different targets are extracted to establish a preset driving environment recognition model.

[0083] According to an embodiment of the present invention, determining a vehicle moving route of a current vehicle and a target moving route of an identified target based on a plurality of consecutive frames of detection image data includes:

[0084] By analyzing the detection image data of multiple consecutive frames, it is determined whether the identified target is a stationary target or a moving target;

[0085] When the identified target is a moving target, a target moving route of the identified target is drawn based on the detection image data of multiple consecutive frames;

[0086] Get the vehicle movement route of the current vehicle;

[0087] Analyze the vehicle movement route of the current vehicle and the target movement route of the identified target to determine the minimum coordinate distance between the current vehicle and the identified target within a preset time interval;

[0088] When the minimum coordinate distance is less than the preset distance threshold, an early warning is issued for the identified target.

[0089] It should be noted that, since the image data in the detection area will change as the vehicle travels, even the coordinate data of the stationary target in each detection image data will be different. By analyzing the detection image data of multiple consecutive frames, a known stationary target (such as a traffic sign) is used as a reference object, and the relative distance between the identification target and the reference object is compared to determine whether the identification target moves, thereby determining whether the identification target is a stationary target or a mobile target. This step is mainly for analyzing targets such as pedestrians and vehicles that have two states of movement and stillness. After determining the state of the identification target, the mobile target is analyzed, and the target movement route of the mobile target is drawn according to the relative coordinate data of the mobile target and the reference object in each detection image data, and the target movement route of the stationary target is determined as a fixed coordinate point. Through the vehicle navigation or the vehicle movement route of the current vehicle, the vehicle movement route of the current vehicle and the target movement route of the identification target are analyzed to determine the coordinate data of the current vehicle and the identification target at each acquisition time, and calculate the coordinate distance. The minimum coordinate distance between the current vehicle and the identified target within the preset time interval is selected, and compared with the preset distance threshold to determine whether the warning level of the identified target within the preset time interval will be changed to the highest warning level, so as to issue an early warning and remind the driver to avoid obstacles for the identified target in advance.

[0090] The preset time interval and the preset distance threshold are set by those skilled in the art according to actual needs.

[0091] Figure 2 A flow chart of the method for adjusting the first attention score of a sub-detection area provided by the present invention is shown.

[0092] like Figure 2 As shown, according to an embodiment of the present invention, the first attention score of the sub-detection area is adjusted based on the attention mechanism, and multiple sub-detection image data of the next detection image data are analyzed in descending order of the first attention scores, including:

[0093] S202, analyzing the vehicle moving route of the current vehicle and the target moving route of the identified target, predicting the coordinate position of the identified target in the detection area in the next detection image data, and determining a first sub-detection area occupied by the identified target;

[0094] S204, increasing a first attention score of the first sub-detection area based on the attention mechanism, and decreasing first attention scores of other sub-detection areas other than the first sub-detection area;

[0095] S206, analyzing the sub-detection image data corresponding to each sub-detection area in order from large to small and from left to right and from top to bottom based on the first attention score, and determining the coordinate position of the recognized target after movement through feature comparison;

[0096] S208, correcting the first sub-detection area occupied by the identified target according to the coordinate position of the identified target after movement, and adjusting the first sub-detection area and other sub-detection areas that have not been subjected to image analysis.

[0097] It should be noted that the coordinate position of the next detection image data acquired by the identification target at the next acquisition time can be predicted through the vehicle movement route of the current vehicle and the target movement route of the identification target, and the sub-detection area occupied by the identification target (that is, the sub-detection area containing the identification target) is determined as the first sub-detection area. Among them, affected by the actual movement and predicted movement errors, the first sub-detection area is a sub-detection area with a higher probability of the existence of the identification target. The first attention scores of the first sub-detection area and other sub-detection areas are adjusted through the attention mechanism. In the process of analyzing the detection image data, the first sub-detection area with a higher first attention score is analyzed first. By comparing the image features of the sub-detection image data of the first sub-detection area with the image features of the identification target, it is determined whether the whole or part of the image features of the identification target exist in the first sub-detection area, thereby determining the coordinate position of the identification target after movement.

[0098] After the coordinate position of the identified target after the move is determined, the first sub-detection area is corrected according to the sub-detection area actually occupied by the identified target, for example, the first attention score of the sub-detection area actually occupied by the identified target that has not been analyzed is reduced, and the first sub-detection area that does not match the actually occupied sub-detection area is adjusted to another sub-detection area, etc. In addition, the first attention score of the edge sub-detection area can also be increased to prioritize the identification of whether there is a new identified target.

[0099] Figure 3 The flowchart of the method for determining newly added targets and disappeared targets provided by the present invention is shown.

[0100] like Figure 3As shown, according to an embodiment of the present invention, by comparing the detection image data of adjacent acquisition times, the newly added target and the disappeared target are determined, including:

[0101] S302, comparing the current detection image data with the previous detection image data, matching the recognition targets of the current detection image data with the recognition targets of the previous detection image data in pairs according to image features, and determining newly added targets and disappeared targets;

[0102] S304: Determine the reasons for adding the new target and disappearing the target based on the target moving routes.

[0103] It should be noted that the identification target that exists in the current detection image data but does not exist in the previous detection image data is determined as a newly added target, and the identification target that does not exist in the current detection image data but exists in the previous detection image data is determined as a disappeared target. The reasons for the addition of the newly added target include addition and occlusion and reappearance, and the reasons for the disappearance of the disappeared target include departure from the detection area and occlusion. Priority is given to analyzing the newly added targets, and the reasons for the addition of the newly added targets that meet the conditions of no historical coordinate data or coordinate data being in the edge sub-detection area are determined as addition, and the reasons for the addition of the newly added targets that meet any of the conditions of the existence of historical coordinate data or coordinate data not being in the edge sub-detection area are determined as occlusion and reappearance. Then, the historical target movement routes of the disappeared targets are analyzed, and the reasons for the disappearance of the disappeared targets that move in the direction of departure from the detection area are determined as departure from the detection area, and the reasons for the disappearance of the disappeared targets that move in other directions are determined as target occlusion.

[0104] According to an embodiment of the present invention, when there is a disappeared target, local image features of the disappeared target are extracted, the second attention score of the sub-detection area at each data collection time is adjusted according to the vehicle moving route and the target moving route of the disappeared target, and multiple sub-detection image data of the next detection image data are analyzed in descending order of the second attention scores, including:

[0105] Extract the last detected image data P n-1 one or more local image features of the disappeared target, and determining one or more first coordinates corresponding to the one or more local image features and corresponding feature scores;

[0106] According to the moving route of the vehicle and the moving route of the disappeared target, one or more local image features are determined in the detection image data P n+m One or more second coordinates corresponding to ; wherein m is a positive integer greater than 1;

[0107] Analyze each second coordinate in turn, and feature-mark a sub-detection area within a preset range of the second coordinate according to the local image feature corresponding to the second coordinate, to determine a second sub-detection area;

[0108] Determine the influence weight of the feature mark of the second sub-detection area according to the second coordinate distance from the center coordinate of the second sub-detection area to the corresponding second coordinate;

[0109] Performing weighted calculation on the feature score of the feature marker of the second sub-detection area and the corresponding influence weight to determine the second attention score and the average attention score of the second sub-detection area; the feature score of the feature marker is the feature score of the local image feature corresponding to the feature marker;

[0110] filtering the second sub-detection area whose average attention score is lower than a preset score threshold;

[0111] According to the second attention score from large to small, the detection image data P is n+m The sub-detection image data of each second sub-detection area is analyzed, and when the image features of the sub-detection image data match the local image features of the disappeared target, it is determined that the disappeared target reappears,

[0112] The disappeared target is framed and selected according to the coordinate data of the sub-detection image data and the historical recognition data of the disappeared target, and the recognition data of the disappeared target is determined.

[0113] It should be noted that the local image feature refers to the local structure or texture information of the disappeared target in the detected image data that is unique, stable and distinguishable, reflecting the local characteristics of the disappeared target in the image, such as structural information such as line crossing, light and dark changes, etc. The disappeared target can be identified through the local image feature, and the first coordinate of each local image feature in the current detected image data is determined. According to the degree of correlation between each local image feature and the disappeared image, a corresponding feature score is assigned to each local image feature. The value range of the feature score is 0-1. The larger the feature score, the higher the accuracy of the disappeared image through the corresponding local image special identification.

[0114] For the next detection image data P n+1 When analyzing, the first coordinate is translated according to the vehicle moving route and the target moving route of the disappeared target to determine the local image feature in the next detection image data P n+1The second coordinate corresponding to the second coordinate is marked according to the preset range of the system, and the feature score of the local image feature is recorded in the feature mark. The sub-detection area with the feature mark is determined as the second sub-detection area, and the second coordinate distance of each second sub-detection area to the corresponding second coordinate is calculated respectively, and the influence weight of the feature mark in each second sub-detection area is determined. The value range of the influence weight is 0-1. The smaller the second coordinate distance of the second sub-detection area to the corresponding second coordinate is, the higher the corresponding influence weight is. When all local feature analyses are completed, each second sub-detection area is analyzed in turn, and the feature score of each feature mark in the second sub-detection area is multiplied by the corresponding influence weight respectively, and the calculation results are accumulated to determine the second attention score of the second sub-detection area, and the ratio of the second attention score to the number of feature marks is calculated to determine the average attention score of the second sub-detection area. Similarly, for the detection image data P n+m When performing the analysis, the local image features of the disappeared target are calculated according to the above steps in the detection image data P n+m The second coordinates in the image data P are determined, the second sub-detection areas are determined, the second attention scores and the average attention scores of each second sub-detection area are calculated, the second sub-detection areas whose average attention scores are lower than the preset score threshold are filtered, and the detection image data P is sequentially sorted according to the second attention scores from large to small. n+m The sub-detection image data in each second sub-detection area is analyzed. By giving priority to analyzing the second sub-detection area where the disappeared target may exist, the priority of the disappeared target recognition is increased to avoid the situation where the identified target suddenly appears and the warning cannot be issued in time.

[0115] In addition, the system can also set the maximum number of recognition times for disappeared targets. When the disappeared target does not appear in the detection image data of the maximum number of recognition times, it is determined that the disappeared target has left the detection area, and the sub-detection area where the disappeared target may appear will no longer be prioritized for recognition.

[0116] The preset score threshold is set by those skilled in the art according to actual needs.

[0117] According to an embodiment of the present invention, after the recognition data of the disappeared target is determined, the feature marks of the local feature data of the disappeared target in each sub-detection area are cleared, the second attention score and the average attention score of each second sub-detection area are recalculated, and the recognition of other disappeared targets continues.

[0118] It should be noted that after determining the identification data of the disappeared target, that is, determining the coordinate data where the disappeared target reappears, there is no need to identify the local feature data of the disappeared target. The feature mark corresponding to the local feature data of the disappeared target in the second detection area is canceled, and the second attention score and the average attention score of each second sub-detection area are recalculated according to the feature mark existing in each second detection area. The second sub-detection areas with an average attention score lower than the preset score threshold are filtered, and the sub-detection image data corresponding to the remaining second sub-detection areas that have not been analyzed are analyzed in descending order of the second attention scores, and other disappeared targets are identified.

[0119] When all the second detection areas are identified, the remaining unanalyzed sub-detection areas are analyzed according to the first attention scores.

[0120] Figure 4 A block diagram of a vehicle environment recognition system based on a vehicle-mounted camera provided by the present invention is shown.

[0121] like Figure 4 As shown, the second aspect of the present invention provides a driving environment recognition system based on a vehicle-mounted camera, comprising:

[0122] A data acquisition module, used to acquire detection image data of the detection area;

[0123] A preprocessing module, used for preprocessing the detection image data to obtain preprocessed image data;

[0124] The driving environment recognition module is used to input the pre-processed image data into a preset driving environment recognition model, which extracts and classifies the pre-processed image data, outputs driving environment recognition data, and determines the recognition target; the driving environment recognition data includes but is not limited to lane line detection, traffic sign recognition, pedestrian detection and vehicle detection; the recognition target includes but is not limited to pedestrian targets and vehicle targets; the first coordinate distance between the recognition target and the current vehicle is calculated to determine the risk level of the recognition target;

[0125] A region segmentation module, used to divide the detection region into a plurality of sub-detection regions, and divide the detection image data into a plurality of sub-detection image data based on the plurality of sub-detection regions;

[0126] The target recognition monitoring module is used to determine the vehicle movement route of the current vehicle and the target movement route of the recognition target based on the detection image data of multiple consecutive frames; adjust the first attention score of the sub-detection area based on the attention mechanism, analyze multiple sub-detection image data of the next detection image data in descending order of the first attention score, and determine the newly added target and the disappeared target by comparing the detection image data of adjacent acquisition times;

[0127] The disappeared target monitoring module is used to extract the local image features of the disappeared target when there is a disappeared target, adjust the second attention score of the sub-detection area at each data collection time according to the vehicle movement route and the target movement route of the disappeared target, and analyze the multiple sub-detection image data of the next detection image data in descending order of the second attention scores.

[0128] The third aspect of the present invention provides a computer-readable storage medium, which includes a driving environment recognition method program based on a vehicle-mounted camera. When the driving environment recognition method program based on a vehicle-mounted camera is executed by a processor, the steps of the above-mentioned driving environment recognition method based on a vehicle-mounted camera are implemented.

[0129] The information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the "detection image data" and "historical detection image data" involved in this disclosure are all obtained with full authorization.

[0130] The present invention discloses a driving environment recognition method and system based on a vehicle-mounted camera, the method comprising: acquiring detection image data and preprocessing to obtain preprocessed image data; inputting the preprocessed image data into a preset driving environment recognition model to determine driving environment recognition data, recognition targets and risk levels of recognition targets; dividing a detection area into multiple sub-detection areas, adjusting the first attention score of the sub-detection area based on an attention mechanism, and analyzing the next detection image data according to the first attention score; comparing the detection image data of adjacent acquisition times to determine newly added targets and disappeared targets; extracting local image features of disappeared targets, adjusting the second attention score of the sub-detection area, and analyzing the next detection image data according to the second attention score. The present invention improves the recognition efficiency of the recognition targets in the driving environment by adjusting the attention score of each sub-recognition area.

[0131] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0132] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0133] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0134] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: a mobile storage device, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0135] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

Claims

1. A driving environment recognition method based on a vehicle-mounted camera, characterized in that: include: Acquire detection image data of the detection area; Preprocessing the detected image data to obtain preprocessed image data; The pre-processed image data is input into a preset driving environment recognition model, the driving environment recognition model performs feature extraction and classification on the pre-processed image data, outputs driving environment recognition data, and determines the recognition target; the driving environment recognition data includes lane line detection, traffic sign recognition, pedestrian detection and vehicle detection; the recognition target includes pedestrian target and vehicle target; Calculating a first coordinate distance between the identified target and the current vehicle to determine a risk level of the identified target; Dividing the detection area into a plurality of sub-detection areas, and dividing the detection image data into a plurality of sub-detection image data based on the plurality of sub-detection areas; Determine a vehicle moving route of a current vehicle and a target moving route of an identified target based on the detection image data of the continuous plurality of frames; Adjusting the first attention score of the sub-detection area based on the attention mechanism, and analyzing the multiple sub-detection image data of the next detection image data in descending order of the first attention score; By comparing the detection image data of adjacent acquisition times, newly added targets and disappeared targets are determined; When there is a disappeared target, extract the local image features of the disappeared target, adjust the second attention score of the sub-detection area at each data collection time according to the vehicle movement route and the target movement route of the disappeared target, and analyze multiple sub-detection image data of the next detection image data in descending order of the second attention score; The first attention score of the sub-detection area is adjusted based on the attention mechanism, and the plurality of sub-detection image data of the next detection image data are analyzed in descending order of the first attention scores, including: Analyze the vehicle moving route of the current vehicle and the target moving route of the identified target, predict the coordinate position of the identified target in the detection area in the next detection image data, and determine the first sub-detection area occupied by the identified target; improving a first attention score of the first sub-detection area based on an attention mechanism, and reducing first attention scores of other sub-detection areas other than the first sub-detection area; Analyzing the sub-detection image data corresponding to each sub-detection area in order from large to small based on the first attention score and from left to right and from top to bottom, and determining the coordinate position of the identified target after movement through feature comparison; The first sub-detection area occupied by the identification target is corrected according to the coordinate position of the identification target after the movement, and the first sub-detection area and other sub-detection areas that are not subjected to image analysis are adjusted.

2. The method for identifying a driving environment based on a vehicle-mounted camera according to claim 1, characterized in that: The preprocessing of the detected image data to obtain preprocessed image data includes: Performing data enhancement on the detection image data; The detected image data is subjected to image preprocessing; the image preprocessing at least includes image denoising and color correction.

3. The method for identifying a driving environment based on a vehicle-mounted camera according to claim 1, characterized in that: Also includes: Obtain historical detection image data; An analysis is performed based on the historical detection image data to establish a preset driving environment recognition model.

4. The method for identifying a driving environment based on a vehicle-mounted camera according to claim 1, characterized in that: The method of determining the vehicle moving route of the current vehicle and the target moving route of the identified target based on the detection image data of the continuous multiple frames includes: By analyzing the detection image data of multiple consecutive frames, determining whether the identified target is a stationary target or a moving target; When the identified target is a moving target, drawing a target moving route of the identified target based on the detection image data of the continuous multiple frames; Get the vehicle movement route of the current vehicle; Analyzing the vehicle moving route of the current vehicle and the target moving route of the identified target to determine the minimum coordinate distance between the current vehicle and the identified target within a preset time interval; When the minimum coordinate distance is less than a preset distance threshold, an early warning is issued to the identified target.

5. The method for identifying a driving environment based on a vehicle-mounted camera according to claim 1, characterized in that: The step of comparing the detection image data at adjacent acquisition times to determine the newly added target and the disappeared target includes: Compare the current detection image data with the previous detection image data, match the recognition targets of the current detection image data with the recognition targets of the previous detection image data in pairs according to the image features, and determine the newly added targets and the disappeared targets; The reasons for adding the new target and the reasons for disappearing the disappeared target are determined based on the target movement routes.

6. The method for identifying a driving environment based on a vehicle-mounted camera according to claim 1, characterized in that: When there is a disappeared target, extract the local image features of the disappeared target, adjust the second attention score of the sub-detection area at each data collection time according to the vehicle movement route and the target movement route of the disappeared target, and analyze the multiple sub-detection image data of the next detection image data in descending order of the second attention scores, including: Extract the last detected image data P n-1 one or more local image features of the disappeared target, and determining one or more first coordinates and corresponding feature scores corresponding to the one or more local image features; According to the moving route of the vehicle and the moving route of the disappeared target, it is determined that the one or more local image features are in the detection image data P n+m One or more second coordinates corresponding to ; wherein m is a positive integer greater than 1; Analyze each second coordinate in turn, and feature-mark a sub-detection area within a preset range of the second coordinate according to a local image feature corresponding to the second coordinate, to determine a second sub-detection area; Determine the influence weight of the characteristic mark of the second sub-detection area according to the second coordinate distance from the center coordinate of the second sub-detection area to the corresponding second coordinate; Performing weighted calculation on the feature score of the feature mark of the second sub-detection area and the corresponding influence weight to determine the second attention score and the average attention score of the second sub-detection area; the feature score of the feature mark is the feature score of the local image feature corresponding to the feature mark; filtering the second sub-detection area whose average attention score is lower than a preset score threshold; According to the second attention score from large to small, the detection image data P is n+m The sub-detection image data of each second sub-detection area is analyzed, and when the image features of the sub-detection image data match the local image features of the disappeared target, it is determined that the disappeared target reappears, The disappeared object is framed according to the coordinate data of the sub-detection image data and the historical recognition data of the disappeared object, and the recognition data of the disappeared object is determined.

7. The method for identifying a driving environment based on a vehicle-mounted camera according to claim 6, characterized in that: Also includes: After the identification data of the disappeared target is determined, the feature marks of the local feature data of the disappeared target in each sub-detection area are cleared, the second attention score and the average attention score of each second sub-detection area are recalculated, and the identification of other disappeared targets continues.

8. A vehicle-mounted camera-based driving environment recognition system, used to implement the vehicle-mounted camera-based driving environment recognition method according to any one of claims 1 to 7, characterized in that: include: A data acquisition module, used to acquire detection image data of the detection area; A preprocessing module, used for preprocessing the detection image data to obtain preprocessed image data; A driving environment recognition module is used to input the pre-processed image data into a preset driving environment recognition model, the driving environment recognition model extracts and classifies the pre-processed image data, outputs driving environment recognition data, and determines the recognition target; the driving environment recognition data includes lane line detection, traffic sign recognition, pedestrian detection and vehicle detection; the recognition target includes pedestrian targets and vehicle targets; the first coordinate distance between the recognition target and the current vehicle is calculated to determine the risk level of the recognition target; A region segmentation module, used to divide the detection region into a plurality of sub-detection regions, and divide the detection image data into a plurality of sub-detection image data based on the plurality of sub-detection regions; A target recognition monitoring module is used to determine the vehicle movement route of the current vehicle and the target movement route of the recognition target based on the detection image data of multiple consecutive frames; Based on the attention mechanism, the first attention score of the sub-detection area is adjusted, and multiple sub-detection image data of the next detection image data are analyzed in descending order of the first attention score, and the newly added targets and the disappeared targets are determined by comparing the detection image data of adjacent acquisition times; A disappeared target monitoring module is used to extract local image features of the disappeared target when there is a disappeared target, adjust the second attention score of the sub-detection area at each data collection time according to the vehicle movement route and the target movement route of the disappeared target, and analyze multiple sub-detection image data of the next detection image data in descending order of the second attention scores; The first attention score of the sub-detection area is adjusted based on the attention mechanism, and the plurality of sub-detection image data of the next detection image data are analyzed in descending order of the first attention scores, including: Analyze the vehicle moving route of the current vehicle and the target moving route of the identified target, predict the coordinate position of the identified target in the detection area in the next detection image data, and determine the first sub-detection area occupied by the identified target; improving a first attention score of the first sub-detection area based on an attention mechanism, and reducing first attention scores of other sub-detection areas other than the first sub-detection area; Analyzing the sub-detection image data corresponding to each sub-detection area in order from large to small based on the first attention score and from left to right and from top to bottom, and determining the coordinate position of the identified target after movement through feature comparison; The first sub-detection area occupied by the identification target is corrected according to the coordinate position of the identification target after the movement, and the first sub-detection area and other sub-detection areas that are not subjected to image analysis are adjusted.

9. The vehicle environment recognition system based on a vehicle-mounted camera according to claim 8, characterized in that: The first attention score of the sub-detection area is adjusted based on the attention mechanism, and the plurality of sub-detection image data of the next detection image data are analyzed in descending order of the first attention scores, including: Analyze the vehicle moving route of the current vehicle and the target moving route of the identified target, predict the coordinate position of the identified target in the detection area in the next detection image data, and determine the first sub-detection area occupied by the identified target; improving a first attention score of the first sub-detection area based on an attention mechanism, and reducing first attention scores of other sub-detection areas other than the first sub-detection area; Analyzing the sub-detection image data corresponding to each sub-detection area in order from large to small based on the first attention score and from left to right and from top to bottom, and determining the coordinate position of the identified target after movement through feature comparison; The first sub-detection area occupied by the identification target is corrected according to the coordinate position of the identification target after the movement, and the first sub-detection area and other sub-detection areas that are not subjected to image analysis are adjusted.

Images