Night vision image processing method and device based on AI data analysis and storage medium
Through AI data analysis combined with the data of low light night vision instruments and infrared cameras, the global characteristics of night traffic are extracted, the problem of night traffic environment identification is solved, night driving safety warning is realized, and accident risk is reduced.
Patent Information
- Application Number
- CN202411062349.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The incidence of night traffic accidents is relatively high. The existing semiconductor laser night vision system may cause the environment to be too dark when reducing lighting energy, making it difficult to identify the night traffic environment, increasing the risk of accidents.
Using AI data analysis methods, night vision traffic images collected by low light night vision instruments and traffic thermal imaging videos collected by infrared cameras are extracted using deep learning technology to determine whether night driving safety warnings are issued to identify potential Dangerous situation.
The safety of driving at night is improved, and by comprehensively analyzing night vision and thermal imaging data, warning the driver is promptly issued to reduce the risk of accidents.
Smart Images

Figure CN119091398A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of night vision image processing, and more specifically, to a night vision image processing method, device and storage medium based on AI data analysis. Background Art
[0002] Traffic accidents refer to situations where vehicles collide or other unexpected events occur during road traffic, resulting in casualties or property damage. Traffic accidents that occur at night are particularly dangerous because visibility is low and drivers often have difficulty finding obstacles or other vehicles on the road in time, increasing the risk of accidents. Common problems in night driving include fatigue driving, drunk driving, limited vision, and insufficient road lighting, which can lead to serious traffic accidents, causing casualties and property damage.
[0003] With the advancement of night vision technology and the reduction of the cost of night vision equipment, semiconductor laser night vision system has been widely used as a night vision technology. This system uses semiconductor lasers as a lighting source to provide better night vision. However, due to the concentrated energy distribution of lasers, using lasers as lighting sources may cause strong saturation in the image. When the lighting energy is reduced, the environment may be too dark, making it difficult to identify the night traffic environment.
[0004] Therefore, a night vision image processing method, device and storage medium based on AI data analysis are desired. Summary of the invention
[0005] In order to solve the above technical problems, the present application is proposed. The embodiments of the present application provide a night vision image processing method, device and storage medium based on AI data analysis, which first obtains night vision traffic images at multiple predetermined time points collected by a low-light night vision device and traffic thermal imaging videos collected by an infrared camera, and then uses deep learning technology to perform feature extraction and correlation analysis on the two, and finally uses a classifier to determine whether to issue a night driving safety warning, so as to have a more comprehensive understanding of the traffic conditions on the road at night, including vehicles, pedestrians, etc., so as to accurately identify potential dangerous situations and issue warnings to drivers in time, thereby reducing the risk of accidents and improving the safety of night driving.
[0006] According to one aspect of the present application, a night vision image processing method based on AI data analysis is provided, which includes:
[0007] Acquire night vision traffic images at multiple predetermined time points collected by a low-light night vision device and traffic thermal imaging videos collected by an infrared camera;
[0008] Extracting a night vision traffic global feature vector and a traffic thermal imaging global feature vector from the night vision traffic images at a plurality of predetermined time points collected by the low-light night vision device and the traffic thermal imaging video collected by the infrared camera;
[0009] Based on the night vision traffic global feature vector and the traffic thermal imaging global feature vector, it is determined whether to issue a nighttime driving safety warning.
[0010] According to another aspect of the present application, a night vision image processing device based on AI data analysis is provided, comprising:
[0011] A traffic night vision data acquisition module is used to acquire night vision traffic images at multiple predetermined time points acquired by a low-light night vision device and traffic thermal imaging videos acquired by an infrared camera;
[0012] A traffic night vision data extraction module, used to extract night vision traffic global feature vectors and traffic thermal imaging global feature vectors from the night vision traffic images at multiple predetermined time points collected by the low-light night vision device and the traffic thermal imaging video collected by the infrared camera;
[0013] The nighttime driving safety warning module is used to determine whether to issue a nighttime driving safety warning based on the night vision traffic global feature vector and the traffic thermal imaging global feature vector.
[0014] According to another aspect of the present application, a computer-readable medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the night vision image processing device based on AI data analysis as described above.
[0015] Compared with the prior art, the present application provides a night vision image processing method, device and storage medium based on AI data analysis, which first obtains night vision traffic images at multiple predetermined time points collected by a low-light night vision device and traffic thermal imaging videos collected by an infrared camera, and then uses deep learning technology to perform feature extraction and correlation analysis on the two. Finally, a classifier is used to determine whether to issue a night driving safety warning, so as to have a more comprehensive understanding of the traffic conditions on the road at night, including vehicles, pedestrians, etc., so as to accurately identify potential dangerous situations and issue warnings to the driver in time, thereby reducing the risk of accidents and improving the safety of night driving. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.
[0017] Figure 1 It is a flowchart of a night vision image processing method based on AI data analysis according to an embodiment of the present application.
[0018] Figure 2 The present invention is a flowchart of extracting features from night vision traffic images at multiple predetermined time points collected by a low-light night vision device in a night vision image processing method based on AI data analysis according to an embodiment of the present application to obtain a global feature vector of night vision traffic.
[0019] Figure 3 The present invention is a flowchart of extracting features from the traffic thermal imaging video captured by an infrared camera in a night vision image processing method based on AI data analysis according to an embodiment of the present application to obtain the global feature vector of the traffic thermal imaging.
[0020] Figure 4 The present invention is a flowchart of extracting features from the multiple traffic thermal imaging key frames in the night vision image processing method based on AI data analysis according to an embodiment of the present application to obtain a traffic thermal imaging time attention feature map and a traffic thermal imaging space attention feature map.
[0021] Figure 5 It is a block diagram of a night vision image processing device based on AI data analysis according to an embodiment of the present application.
[0022] Figure 6 is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0023] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.
[0024] Exemplary Methods
[0025] Figure 1 FIG. 1 is a flow chart of a night vision image processing method based on AI data analysis according to an embodiment of the present application. Figure 1As shown, according to the night vision image processing method based on AI data analysis according to the embodiment of the present application, the method includes: S110, obtaining night vision traffic images at multiple predetermined time points collected by a low-light night vision device and traffic thermal imaging videos collected by an infrared camera; S120, extracting night vision traffic global feature vectors and traffic thermal imaging global feature vectors from the night vision traffic images at multiple predetermined time points collected by the low-light night vision device and the traffic thermal imaging videos collected by the infrared camera; S130, judging whether to issue a night driving safety warning based on the night vision traffic global feature vector and the traffic thermal imaging global feature vector.
[0026] In the above-mentioned night vision image processing method based on AI data analysis, the step S110 obtains night vision traffic images at multiple predetermined time points collected by the low-light night vision device and traffic thermal imaging videos collected by the infrared camera. It should be understood that in road traffic, a traffic accident refers to a situation in which a vehicle collides or other unexpected events occur during driving, causing casualties or property losses. Traffic accidents at night are particularly dangerous because visibility is low and drivers usually find it difficult to detect obstacles or other vehicles on the road in time, thereby increasing the risk of accidents. Common problems in night driving include fatigue driving, drunk driving, limited vision, insufficient road lighting and other factors, which may lead to serious traffic accidents and cause losses to people and property. With the continuous advancement of night vision technology and the reduction in the cost of night vision equipment, semiconductor laser night vision systems have been widely used as an advanced night vision technology. This system uses semiconductor lasers as lighting sources to provide clearer visual effects for night driving. However, due to the relatively concentrated energy distribution of the laser, the use of lasers as lighting sources may cause strong saturation in the image. To avoid this situation, while reducing the lighting energy, the environment may become too dim, making it difficult to accurately identify the nighttime traffic environment, thereby increasing driving risks. Therefore, the technical solution of this application aims to use night vision traffic images captured by a low-light night vision device at multiple predetermined time points and traffic thermal imaging videos acquired by an infrared camera, combined with deep learning technology for analysis. In this way, it can be determined whether a nighttime driving safety warning is needed to accurately identify potential dangerous situations and issue warnings to the driver in a timely manner. Through this method, the risk of accidents can be effectively reduced and the overall safety level of nighttime driving can be improved.
[0027] Specifically, low-light-level night vision devices can capture clear night vision images in low-light conditions through their sensitive optical sensors and image processing technology, providing important data support for nighttime traffic monitoring. These night vision images can not only help monitor traffic flow and vehicle driving conditions, but can also be used for accident investigation and safety assessment. At the same time, traffic thermal imaging videos from infrared cameras provide another important monitoring perspective. Infrared imaging technology can measure the heat distribution of objects and display the heat conditions of objects through colors of different temperatures. In traffic monitoring, this technology can be used to detect the heat distribution of vehicles, identify vehicles with abnormal engines or overheating, and even detect potential mechanical failures. By combining low-light-level night vision images and infrared thermal imaging videos, traffic management personnel can monitor traffic conditions in an all-round way, detect problems in a timely manner and take corresponding measures.
[0028] In the above-mentioned night vision image processing method based on AI data analysis, the step S120 extracts the night vision traffic global feature vector and the traffic thermal imaging global feature vector from the night vision traffic images at multiple predetermined time points collected by the low-light night vision device and the traffic thermal imaging video collected by the infrared camera. It should be understood that the comprehensively extracted night vision traffic global feature vector and traffic thermal imaging global feature vector can realize comprehensive analysis and monitoring of traffic scenes. This comprehensive analysis can not only help to better understand the traffic conditions, but also provide decision-making support for traffic management departments and improve traffic operation efficiency and safety. Therefore, extracting these global feature vectors is of great significance for optimizing traffic management and improving traffic safety levels.
[0029] In a specific embodiment of the present application, the step S120 includes: performing feature extraction on the night vision traffic images at multiple predetermined time points collected by the low-light night vision device to obtain a global feature vector of night vision traffic; performing feature extraction on the traffic thermal imaging video collected by the infrared camera to obtain the global feature vector of traffic thermal imaging.
[0030] It should be understood that night vision traffic images contain rich traffic information, such as the number of vehicles, vehicle types, speeds, lane usage, etc. By extracting features from these images, various visual features can be extracted, such as the outline, movement direction, size, etc. of the vehicle, and then the global feature vector of night vision traffic can be constructed. These global feature vectors can reflect the characteristics of the entire traffic scene, help understand traffic flow, congestion, and vehicle driving status, and provide important reference for traffic management decisions.
[0031] Furthermore, feature extraction is performed on the traffic thermal imaging videos collected by the infrared camera, where the traffic thermal imaging videos can reflect the heat distribution, engine temperature, and possible mechanical problems of the vehicle. By performing feature extraction on these videos, various heat distribution features can be extracted, such as vehicle hotspot locations, heat-intensive areas, etc., to construct global feature vectors of traffic thermal imaging. These feature vectors can help monitor vehicle status, identify abnormal conditions, and promptly detect vehicle failures or potential dangers, thereby improving traffic safety.
[0032] Figure 2 The present invention is a flowchart of extracting features from the night vision traffic images collected by the low-light night vision device at multiple predetermined time points in the night vision image processing method based on AI data analysis according to an embodiment of the present application to obtain a global feature vector of night vision traffic. Figure 2 As shown, in a specific embodiment of the present application, feature extraction is performed on the night vision traffic images at multiple predetermined time points collected by the low-light level night vision device to obtain a global feature vector of the night vision traffic, including: S210, passing the night vision traffic images at multiple predetermined time points collected by the low-light level night vision device through a night vision traffic target detection convolutional neural network to obtain multiple night vision traffic regions of interest; S220, arranging the multiple night vision traffic regions of interest to obtain a night vision traffic input tensor; S230, passing the night vision traffic input tensor through a night vision traffic low-light level feature encoder based on a convolutional neural network to obtain the night vision traffic global feature vector.
[0033] It should be understood that by processing the night vision traffic images at multiple predetermined time points collected by the low-light night vision device through the night vision traffic target detection convolutional neural network, multiple night vision traffic interest regions can be obtained. These interest regions are traffic targets that have been intelligently identified and located, and are of key significance to the development of traffic monitoring, safety management and intelligent transportation systems. Among them, the night vision traffic target detection convolutional neural network can effectively identify traffic targets in the image, such as vehicles, pedestrians, etc., and accurately calibrate their positions and bounding boxes. In this way, multiple interest regions, i.e., the location information of traffic targets, can be extracted from the night vision traffic image. Moreover, the multiple night vision traffic interest regions obtained by the night vision traffic target detection convolutional neural network can help to track and monitor traffic targets. Specifically, the night vision traffic images at multiple predetermined time points collected by the low-light night vision device are processed using the night vision traffic target detection convolutional neural network with the following target detection formula to obtain the multiple night vision traffic interest regions;
[0034] Among them, the target detection formula is:
[0035] ROI=H(ψ det ,B)-(cls(ψ det,B),Regr(ψ det , B))
[0036] Among them, ψ det are night vision traffic images collected by the low-light night vision device at multiple predetermined time points, B is the anchor frame, ROI is the region of interest for multiple night vision traffic, cls(ψ det ,B) indicates classification, Regr(ψ det ,B) indicates regression.
[0037] Furthermore, by arranging multiple night vision traffic regions of interest into a night vision traffic input tensor, the feature information of each region can be organized in an orderly manner to form a multi-dimensional data structure. This arrangement can better preserve the spatial position relationship and feature information of traffic targets, and improve the model's ability to identify and analyze traffic targets. Secondly, the construction of the night vision traffic input tensor can help achieve an overall understanding and analysis of traffic targets. By organically arranging multiple regions of interest together, the various targets in the traffic image and the associated information between them can be integrated into a unified data structure. This comprehensive data representation method helps deep learning models to learn and understand traffic scenes more comprehensively, and improve the accuracy and robustness of traffic target detection, tracking, and prediction.
[0038] Furthermore, by converting the night vision traffic input tensor into a night vision traffic global feature vector through a night vision traffic low-light feature encoder based on a convolutional neural network, it can help extract high-dimensional, abstract feature representations from complex image data. Among them, the night vision traffic low-light feature encoder based on a convolutional neural network can gradually extract local features and abstract features in the image through multi-layer convolution and pooling operations. This hierarchical feature extraction process can effectively capture various detailed information and structural features in traffic images, helping to better understand the complexity and diversity of traffic scenes. The night vision traffic global feature vector obtained by the night vision traffic low-light feature encoder integrates the local features in the image into a high-dimensional global feature representation. This global feature vector can better summarize the content and semantic information of the entire traffic image, and has stronger representation and generalization capabilities. Specifically, each layer of the night vision traffic low light feature encoder based on the convolutional neural network performs the following on the input data in the forward transfer of the layer: based on the convolution kernel, convolution processing is performed on the input data to generate a convolution feature map; the convolution feature map is subjected to global mean pooling processing based on the feature matrix to generate a pooled feature map; and the feature values at each position in the pooled feature map are nonlinearly activated to generate an activated feature addition map; wherein the output of the last layer of the night vision traffic low light feature encoder based on the convolutional neural network is the night vision traffic global feature vector, the input from the second layer to the last layer of the night vision traffic low light feature encoder based on the convolutional neural network is the output of the previous layer, and the input of the night vision traffic low light feature encoder based on the convolutional neural network is the night vision traffic input tensor.
[0039] Figure 3 The present invention is a flowchart of extracting features from the traffic thermal imaging video captured by the infrared camera to obtain the global feature vector of the traffic thermal imaging in the night vision image processing method based on AI data analysis according to an embodiment of the present application. Figure 3 As shown, in a specific embodiment of the present application, feature extraction is performed on the traffic thermal imaging video collected by the infrared camera to obtain the traffic thermal imaging global feature vector, including: S310, collecting key frames of the traffic thermal imaging video collected by the infrared camera at a predetermined sampling frequency to obtain multiple traffic thermal imaging key frames; S320, feature extraction is performed on the multiple traffic thermal imaging key frames to obtain a traffic thermal imaging time attention feature map and a traffic thermal imaging space attention feature map; S330, the traffic thermal imaging time attention feature map and the traffic thermal imaging space attention feature map are passed through a traffic thermal imaging convolutional feature encoder to obtain the traffic thermal imaging global feature vector.
[0040] It should be understood that key frame extraction is to select the most representative and information-rich frames from the video sequence in order to provide effective data support in subsequent analysis and processing. By acquiring the traffic thermal imaging video of the infrared camera at a predetermined sampling frequency, it can be ensured that the changes and dynamic information of the traffic scene are evenly captured in the video sequence. In this way, the extracted key frames can effectively represent the content and characteristics of the entire video sequence, avoid information duplication and unnecessary redundancy, and improve the efficiency of data utilization. Among them, extracting multiple traffic thermal imaging key frames helps to capture different states and key moments in the traffic scene. These key frames can reflect important information such as traffic flow, vehicle operation status, traffic accidents, etc., and provide more comprehensive data support for traffic monitoring, analysis and prediction.
[0041] Furthermore, feature extraction of multiple traffic thermal imaging key frames can help extract key temporal and spatial information from traffic thermal imaging data, thereby better understanding the dynamic changes and spatial distribution characteristics of traffic scenes. Here, by extracting features from multiple traffic thermal imaging key frames, rich temporal and spatial features can be extracted from each key frame. Among them, the temporal attention feature map reflects the importance and change trend of traffic thermal imaging data at different time points, helping to capture the temporal dynamic information in traffic scenes. The spatial attention feature map reflects the spatial distribution and correlation of traffic thermal imaging data, revealing the feature differences and connections between different areas in traffic scenes.
[0042] Furthermore, processing the time attention feature map and the space attention feature map of traffic thermal imaging through the traffic thermal imaging convolutional feature encoder can help combine the time and space information in the traffic thermal imaging data and extract higher-level global features, thereby achieving a comprehensive understanding and analysis of the entire traffic scene. Among them, the time attention feature map and the space attention feature map of traffic thermal imaging contain important time and space information in the traffic scene. The time attention feature map reflects the feature changes and dynamic trends of traffic thermal imaging data at different time points, and the space attention feature map reveals the feature distribution and correlation of different regions in the traffic scene. Through the processing of the traffic thermal imaging convolutional feature encoder, these time and space information can be effectively fused and extracted to obtain a more representative and comprehensive global feature vector. Here, the extraction of the global feature vector of traffic thermal imaging can help realize the global feature expression and analysis of the entire traffic scene. This global feature vector contains the key time and space features in the traffic thermal imaging data, which can better describe the overall characteristics and laws of the traffic scene. By analyzing and processing this global feature vector, the overall cognition of the traffic scene can be achieved, providing more comprehensive data support for traffic monitoring, traffic management and intelligent traffic decision-making. Specifically, each layer of the traffic thermal imaging convolutional feature encoder is used to perform convolution processing, mean pooling processing based on the local feature matrix and nonlinear activation processing on the input data in the forward pass of the layer so that the last layer of the traffic thermal imaging convolutional feature encoder outputs the traffic thermal imaging global feature vector, wherein the input of the traffic thermal imaging convolutional feature encoder is the traffic thermal imaging time attention feature map and the traffic thermal imaging spatial attention feature map.
[0043] Figure 4 The present invention is a flowchart of extracting features from the multiple traffic thermal imaging key frames in the night vision image processing method based on AI data analysis according to an embodiment of the present application to obtain a traffic thermal imaging time attention feature map and a traffic thermal imaging space attention feature map. Figure 4 As shown, in a specific embodiment of the present application, feature extraction is performed on the multiple traffic thermal imaging key frames to obtain a traffic thermal imaging time attention feature map and a traffic thermal imaging space attention feature map, including: S321, the multiple traffic thermal imaging key frames are passed through a traffic thermal imaging convolutional neural network model using a time attention mechanism to obtain the traffic thermal imaging time attention feature map; S322, the multiple traffic thermal imaging key frames are passed through a traffic thermal imaging convolutional neural network model using a space attention mechanism to obtain the traffic thermal imaging space attention feature map.
[0044] It should be understood that multiple traffic thermal imaging key frames contain characteristic information of traffic scenes at different time points. By using the traffic thermal imaging convolutional neural network model with a temporal attention mechanism, these key frames can be effectively modeled in time series, thereby extracting key temporal attention features. These temporal attention features reflect the importance and changing trends of traffic thermal imaging data at different time points, helping to better understand the temporal dynamic characteristics of traffic scenes. Specifically, an adjacent first thermal imaging key frame and a second thermal imaging key frame are extracted from the multiple traffic thermal imaging key frames; the first thermal imaging key frame and the second thermal imaging key frame are respectively passed through the first convolution layer and the second convolution layer of the traffic thermal imaging convolutional neural network model using the temporal attention mechanism to obtain a first convolutional feature map and a second convolutional feature map; the position point multiplication between the first convolutional feature map and the second convolutional feature map is calculated to obtain a temporal attention map; the temporal attention map is input into a Softmax activation function to obtain a temporal attention feature map; the second thermal imaging key frame is passed through the third convolution layer of the traffic thermal imaging convolutional neural network model using the temporal attention mechanism to obtain a third convolutional feature map; and, the position point multiplication between the third convolutional feature map and the temporal attention feature map is calculated to obtain the traffic thermal imaging temporal attention feature map.
[0045] Furthermore, processing multiple traffic thermal imaging keyframes by using the traffic thermal imaging convolutional neural network model with spatial attention mechanism can help extract spatial information from traffic thermal imaging data, reveal the characteristic distribution and correlation of different regions in the traffic scene, and provide strong support for the spatial analysis and target detection of traffic data. Here, multiple traffic thermal imaging keyframes contain the spatial characteristics and distribution information of different regions in the traffic scene. By processing these keyframes by using the traffic thermal imaging convolutional neural network model with spatial attention mechanism, the correlation and importance between different regions in the traffic scene can be effectively captured, and more representative and regionally perceptual spatial feature information can be extracted. The extraction of traffic thermal imaging spatial attention feature map can help realize the analysis of characteristic distribution and correlation of different regions in the traffic scene. This spatial attention feature map reflects the importance and characteristic expression of different regions in traffic thermal imaging data, which can help better understand the characteristics and changing laws of each region in the traffic scene. By extracting and analyzing these spatial features, spatial perception and target detection of traffic scenes can be realized, providing important support for the development and application of intelligent transportation systems. Specifically, the convolutional coding part of the traffic thermal imaging convolutional neural network model using the spatial attention mechanism is used to perform deep convolutional coding on the multiple traffic thermal imaging key frames to obtain an initial convolutional feature map; the initial convolutional feature map is input into the spatial attention part of the traffic thermal imaging convolutional neural network model using the spatial attention mechanism to obtain a spatial attention map; the spatial attention map is activated by a Softmax function to obtain a spatial attention feature map; and the spatial attention feature map and the initial convolutional feature map are multiplied by the position points to obtain the traffic thermal imaging spatial attention feature map. More specifically, the initial convolutional feature map is input into the spatial attention part of the convolutional neural network model to obtain the spatial attention map, including: performing average pooling and maximum pooling along the channel dimension on the initial convolutional feature map to obtain an average feature matrix and a maximum feature matrix; cascading and channel-adjusting the average feature matrix and the maximum feature matrix to obtain a channel feature matrix; and convolutionally coding the channel feature matrix using the convolutional layer of the spatial attention feature map to obtain the spatial attention map.
[0046] In the above-mentioned night vision image processing method based on AI data analysis, the step S130 determines whether to issue a night driving safety warning based on the night vision traffic global feature vector and the traffic thermal imaging global feature vector. It should be understood that judging whether to issue a night driving safety warning based on the night vision traffic global feature vector and the traffic thermal imaging global feature vector combines the global features of night vision images and traffic thermal imaging data, which can improve the monitoring and warning effects of night driving safety, provide more comprehensive information support for drivers and traffic management departments, and thus comprehensively consider the key information in night vision images and traffic thermal imaging data to improve the monitoring and warning effects of night driving safety. By comprehensively analyzing these two global feature vectors, the situation of night traffic scenes can be more comprehensively understood, traffic anomalies and safety hazards can be discovered in a timely manner, and timely and effective safety warning information can be provided to drivers and traffic management departments, thereby improving the safety and reliability of night driving.
[0047] In a specific embodiment of the present application, the step S130 includes: fusing the night vision traffic global feature vector and the traffic thermal imaging global feature vector to obtain a traffic night vision judgment classification feature vector; performing feature diffusion of a deep affine network on the traffic night vision judgment classification feature vector to obtain an optimized traffic night vision judgment classification feature vector; passing the optimized traffic night vision judgment classification feature vector through a classifier to obtain a classification result, and the classification result is used to determine whether to issue a night driving safety warning.
[0048] It should be understood that night vision images and traffic thermal imaging data provide different perspectives and information dimensions. Among them, night vision images can reflect the visual features of night traffic scenes, such as vehicle shape, light brightness, etc., while traffic thermal imaging data can show thermal features such as the heat distribution of traffic flow and vehicle operation status. By fusing the global feature vectors of these two different data sources, visual information and thermal features can be comprehensively considered to improve the understanding and analysis capabilities of night traffic scenes.
[0049] Furthermore, the feature diffusion of the deep affine network is performed on the traffic night vision judgment classification feature vector to obtain an optimized traffic night vision judgment classification feature vector, including: calculating the natural exponential function value of each position in the traffic night vision judgment classification feature vector to obtain a divergent traffic night vision judgment classification feature vector; accumulating and summing the divergent traffic night vision judgment classification feature vector to obtain the sum value of the divergent traffic night vision judgment classification feature vector; multiplying the sum value of the divergent traffic night vision judgment classification feature vector with the absolute value of the feature value of each position of the traffic night vision judgment classification feature vector and then calculating the logarithmic function value with base two to obtain the traffic night vision judgment classification feature vector after logarithmic change; multiplying the traffic night vision judgment classification feature vector after logarithmic change with the traffic night vision judgment classification feature vector by position to obtain the optimized traffic night vision judgment classification feature vector.
[0050] In particular, in the technical solution of the present application, by integrating the night vision traffic images collected by the low-light night vision device and the traffic thermal imaging video collected by the infrared camera, the traffic night vision judgment classification feature vector is calculated to determine whether to issue a night driving safety warning. Improving the information aggregation degree of the traffic night vision judgment classification feature vector is crucial for accurately judging the safety of night driving. Specifically, improving the information aggregation degree of the traffic night vision judgment classification feature vector can comprehensively consider the characteristics of the night vision traffic image and the traffic thermal imaging video. By integrating multimodal information, a more comprehensive understanding of night traffic conditions can be achieved, and the accuracy and reliability of the safety warning system can be improved. High information aggregation means that the global feature vector of night vision traffic and the global feature vector of traffic thermal imaging can be better integrated. By comprehensively considering the global features, the association between different information sources can be better captured, and the accuracy and robustness of the safety warning judgment can be improved. High information aggregation can improve the effect of feature fusion, which helps to more accurately integrate the global features of night vision traffic and the global features of traffic thermal imaging. Accurate feature fusion can improve the representation ability of the classification feature vector and provide a more reliable basis for night driving safety warning. By improving the information aggregation degree of the traffic night vision judgment classification feature vector, the judgment accuracy and timeliness of the safety warning system can be improved. Accurate classification results can help drivers and traffic management departments take timely measures to ensure the safety of night driving. Based on this, the feature diffusion of the deep affine network is performed on the traffic night vision judgment classification feature vector to obtain an optimized traffic night vision judgment classification feature vector.
[0051] Specifically, the feature diffusion of the deep affine network is performed on the traffic night vision judgment classification feature vector to obtain the optimized traffic night vision judgment classification feature vector, including: performing the feature diffusion of the deep affine network on the traffic night vision judgment classification feature vector according to the following optimization formula, wherein the optimization formula is:
[0052]
[0053] Among them, v i is the eigenvalue of the ith position of the traffic night vision judgment classification feature vector, and L is the length of the traffic night vision judgment classification feature vector V, log2 represents the logarithm with base 2, v' i It is the eigenvalue of the i-th position of the optimized traffic night vision judgment classification feature vector.
[0054] Here, in order to improve the information aggregation degree of the traffic night vision judgment classification feature vector, in the technical solution of the present application, the feature diffusion of the deep affine network is performed on the traffic night vision judgment classification feature vector. First, the natural exponential function value of each position in the traffic night vision judgment classification feature vector is calculated to diverge the feature information of the position, and the divergent information of all positions in the traffic night vision judgment classification feature vector is structurally aggregated using a wavelet-like function to obtain a global divergence factor. After the global divergence factor is applied to each position of the traffic night vision judgment classification feature vector, its logarithmic function value with base two is calculated to regress the divergent modulation information of the position. In this way, the feature values of each position of the traffic night vision judgment classification feature vector are feature expanded and concentrated through the mechanism of simulated divergence-regression. In this way, the information aggregation degree of the traffic night vision judgment classification feature vector is improved.
[0055] Furthermore, when the classifier divides the optimized traffic night vision judgment classification feature vector into two categories: safe and dangerous, the corresponding safety warning measures can be triggered according to the classification results. If the classification results indicate that there are safety hazards or abnormal conditions in the traffic scene, the warning system can promptly issue an alarm to the driver or traffic management department to help take necessary countermeasures and reduce the risk of accidents. By combining the application of feature extraction and classifier models, it is possible to achieve rapid and accurate analysis of nighttime traffic scenes, improve the monitoring effect of nighttime driving safety, and provide more reliable support for traffic safety management and driver driving.
[0056] In summary, the embodiment of the present application first obtains night vision traffic images at multiple predetermined time points collected by a low-light night vision device and traffic thermal imaging videos collected by an infrared camera, and then uses deep learning technology to perform feature extraction and correlation analysis on the two. Finally, a classifier is used to determine whether to issue a night driving safety warning, so as to have a more comprehensive understanding of the traffic conditions on the road at night, including vehicles, pedestrians, etc., so as to accurately identify potential dangerous situations and issue warnings to the driver in time, thereby reducing the risk of accidents and improving the safety of night driving.
[0057] Exemplary Devices
[0058] Figure 5FIG. 1 is a block diagram of a night vision image processing device based on AI data analysis according to an embodiment of the present application. Figure 5 As shown, according to the night vision image processing device 100 based on AI data analysis according to an embodiment of the present application, it includes: a traffic night vision data acquisition module 110, which is used to obtain night vision traffic images at multiple predetermined time points collected by a low-light night vision device and traffic thermal imaging videos collected by an infrared camera; a traffic night vision data extraction module 120, which is used to extract night vision traffic global feature vectors and traffic thermal imaging global feature vectors from the night vision traffic images at multiple predetermined time points collected by the low-light night vision device and the traffic thermal imaging videos collected by the infrared camera; a night driving safety warning module 130, which is used to determine whether to issue a night driving safety warning based on the night vision traffic global feature vector and the traffic thermal imaging global feature vector.
[0059] As described above, the night vision image processing device 100 based on AI data analysis according to the embodiment of the present application can be implemented in various terminal devices. In one example, the night vision image processing device 100 based on AI data analysis can be integrated into the terminal device as a software module and / or a hardware module. For example, the night vision image processing device 100 based on AI data analysis can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the night vision image processing device 100 based on AI data analysis can also be one of the many hardware modules of the terminal device.
[0060] Alternatively, in another example, the night vision image processing device 100 based on AI data analysis and the terminal device may also be separate devices, and the night vision image processing device 100 based on AI data analysis may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.
[0061] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned night vision image processing device based on AI data analysis have been referred to above. Figures 1 to 4 The description of the night vision image processing method based on AI data analysis has been introduced in detail, and therefore, its repeated description will be omitted.
[0062] Exemplary electronic devices and storage media
[0063] Figure 6 A structural diagram showing an exemplary hardware architecture of a computing device capable of implementing the night vision image processing method and apparatus based on AI data analysis according to an embodiment of the present application.
[0064] An embodiment of the present application provides an electronic device. Figure 6is a block diagram of an electronic device according to an embodiment of the present application.
[0065] like Figure 6 As shown, the electronic device 10 includes an input device 11, an input interface 12, a central processing unit 13, a memory 14, an output interface 15, an output device 16, and a bus 17. The input interface 12, the central processing unit 13, the memory 14, and the output interface 15 are connected to each other through the bus 17, and the input device 11 and the output device 16 are connected to the bus 17 through the input interface 12 and the output interface 15, respectively, and then connected to other components of the electronic device 10.
[0066] Specifically, the input device 11 receives input information from the outside and transmits the input information to the central processing unit 13 through the input interface 12; the central processing unit 13 processes the input information based on the computer executable instructions stored in the memory 14 to generate output information, temporarily or permanently stores the output information in the memory 14, and then transmits the output information to the output device 16 through the output interface 15; the output device 16 outputs the output information to the outside of the electronic device 10 for user use.
[0067] In one embodiment, Figure 6 The electronic device 10 shown can be implemented as a network device, which may include: a memory configured to store programs; a processor configured to run the programs stored in the memory to execute any one of the night vision image processing methods based on AI data analysis described in the above embodiments.
[0068] According to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network, and / or installed from a removable storage medium.
[0069] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0070] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present application, but the present application is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of the present application, and these modifications and improvements are also considered to be within the scope of protection of the present application.
Claims
1. A night vision image processing method based on AI data analysis, characterized in that: include: Acquire night vision traffic images at multiple predetermined time points collected by a low-light night vision device and traffic thermal imaging videos collected by an infrared camera; Extracting a night vision traffic global feature vector and a traffic thermal imaging global feature vector from the night vision traffic images at a plurality of predetermined time points collected by the low-light night vision device and the traffic thermal imaging video collected by the infrared camera; Based on the night vision traffic global feature vector and the traffic thermal imaging global feature vector, it is determined whether to issue a nighttime driving safety warning.
2. The night vision image processing method based on AI data analysis according to claim 1, characterized in that: Extracting a night vision traffic global feature vector and a traffic thermal imaging global feature vector from the night vision traffic images at a plurality of predetermined time points collected by the low-light night vision device and the traffic thermal imaging video collected by the infrared camera, including: Extracting features from the night vision traffic images collected by the low-light-level night vision device at a plurality of predetermined time points to obtain a global feature vector of the night vision traffic; Feature extraction is performed on the traffic thermal imaging video collected by the infrared camera to obtain the traffic thermal imaging global feature vector.
3. The night vision image processing method based on AI data analysis according to claim 2, characterized in that: Extracting features from the night vision traffic images at a plurality of predetermined time points collected by the low-light-level night vision device to obtain a global feature vector of night vision traffic includes: The night vision traffic images at multiple predetermined time points collected by the low-light night vision device are passed through a night vision traffic target detection convolutional neural network to obtain multiple night vision traffic interest areas; Arranging the plurality of night vision traffic regions of interest to obtain a night vision traffic input tensor; The night vision traffic input tensor is passed through a night vision traffic low-light feature encoder based on a convolutional neural network to obtain the night vision traffic global feature vector.
4. The night vision image processing method based on AI data analysis according to claim 3, characterized in that: Performing feature extraction on the traffic thermal imaging video collected by the infrared camera to obtain the traffic thermal imaging global feature vector includes: Acquiring key frames of the traffic thermal imaging video acquired by the infrared camera at a predetermined sampling frequency to obtain a plurality of traffic thermal imaging key frames; Performing feature extraction on the multiple traffic thermal imaging key frames to obtain a traffic thermal imaging time attention feature map and a traffic thermal imaging space attention feature map; The traffic thermal imaging time attention feature map and the traffic thermal imaging space attention feature map are passed through a traffic thermal imaging convolutional feature encoder to obtain the traffic thermal imaging global feature vector.
5. The night vision image processing method based on AI data analysis according to claim 4, characterized in that: Performing feature extraction on the multiple traffic thermal imaging key frames to obtain a traffic thermal imaging time attention feature map and a traffic thermal imaging space attention feature map, including: The plurality of traffic thermal imaging key frames are passed through a traffic thermal imaging convolutional neural network model using a temporal attention mechanism to obtain the traffic thermal imaging temporal attention feature map; The multiple traffic thermal imaging key frames are passed through a traffic thermal imaging convolutional neural network model using a spatial attention mechanism to obtain the traffic thermal imaging spatial attention feature map.
6. The night vision image processing method based on AI data analysis according to claim 5, characterized in that: Based on the night vision traffic global feature vector and the traffic thermal imaging global feature vector, determining whether to issue a night driving safety warning includes: The night vision traffic global feature vector and the traffic thermal imaging global feature vector are merged to obtain a traffic night vision judgment classification feature vector; Performing feature diffusion of a deep affine network on the traffic night vision judgment classification feature vector to obtain an optimized traffic night vision judgment classification feature vector; The optimized traffic night vision judgment classification feature vector is passed through a classifier to obtain a classification result, and the classification result is used to determine whether to issue a night driving safety warning.
7. The night vision image processing method based on AI data analysis according to claim 6, characterized in that: Performing feature diffusion of a deep affine network on the traffic night vision judgment classification feature vector to obtain an optimized traffic night vision judgment classification feature vector, including: Calculating the natural exponential function value of each position in the traffic night vision judgment classification feature vector to obtain a divergent traffic night vision judgment classification feature vector; Accumulating and summing the divergent traffic night vision judgment classification feature vectors to obtain a sum value of the divergent traffic night vision judgment classification feature vectors; The sum of the divergent traffic night vision judgment classification feature vectors is multiplied by the absolute value of the feature value of each position of the traffic night vision judgment classification feature vector, and then a logarithmic function value with base two is calculated to obtain a traffic night vision judgment classification feature vector after logarithmic change; The logarithmically changed traffic night vision judgment classification feature vector is multiplied by the traffic night vision judgment classification feature vector according to position to obtain the optimized traffic night vision judgment classification feature vector.
8. A night vision image processing device based on AI data analysis, characterized in that: include: A traffic night vision data acquisition module is used to acquire night vision traffic images at multiple predetermined time points acquired by a low-light night vision device and traffic thermal imaging videos acquired by an infrared camera; A traffic night vision data extraction module, used to extract night vision traffic global feature vectors and traffic thermal imaging global feature vectors from the night vision traffic images at multiple predetermined time points collected by the low-light night vision device and the traffic thermal imaging video collected by the infrared camera; The nighttime driving safety warning module is used to determine whether to issue a nighttime driving safety warning based on the night vision traffic global feature vector and the traffic thermal imaging global feature vector.
9. The night vision image processing device based on AI data analysis according to claim 8, characterized in that: The traffic night vision data extraction module comprises: Extracting features from the night vision traffic images collected by the low-light-level night vision device at a plurality of predetermined time points to obtain a global feature vector of the night vision traffic; Feature extraction is performed on the traffic thermal imaging video collected by the infrared camera to obtain the traffic thermal imaging global feature vector.
10. A readable storage medium, wherein a plurality of instructions are stored in the readable storage medium; the plurality of instructions are used for a processor to load and execute the night vision image processing method based on AI data analysis as described in any one of claims 1 to 7.
