Driving warning method and related equipment based on camera feature extraction

Abstract

The present invention provides a traffic warning method and related equipment based on camera feature extraction, wherein the method comprises: performing frame processing on the acquired multimodal video data, and then extracting features from the framed images through a deep residual network to obtain respective initial features. Next, the initial features corresponding to the RGB image, infrared image and depth image are optimized through feature interaction and feature shifting techniques to obtain optimized features. Finally, the optimized features are input into a preset pedestrian intention recognition model, and the model identifies the intention type of the target pedestrian in the camera data based on the input features. Based on the identified intention type and the current driving strategy of the vehicle-mounted terminal. This method improves the accuracy and real-time performance of traffic warnings by comprehensively utilizing multimodal data, especially its adaptability at night or in complex environments, and effectively improves the safety of road driving.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a vehicle warning method based on camera feature extraction and related equipment. Background Art

[0002] In modern transportation systems, ensuring driving safety has become an important issue, especially in the face of complex and changing road environments and unpredictable pedestrian behaviors. With the rapid development of intelligent transportation systems and autonomous driving technologies, using on-board smart cameras for driving warnings has become an effective way to improve road safety. However, although existing technologies can capture pedestrian behavior through cameras and perform a certain degree of behavior recognition, there are still certain limitations in terms of accuracy, real-time, and multi-environment adaptability.

[0003] Traditional traffic warning methods mainly rely on RGB video data captured by basic cameras, and warn of potential dangers by analyzing pedestrian behavior in video images. However, relying solely on RGB data is not ideal in environments with large lighting changes, obstructed vision, or at night. Summary of the invention

[0004] The main purpose of the present invention is to solve the technical problem that the existing method of relying on RGB data for driving warning is not ideal in the environment of large lighting changes, obstructed vision or night environment.

[0005] A first aspect of the present invention provides a vehicle driving warning method based on camera feature extraction, the vehicle driving warning method based on camera feature extraction comprising:

[0006] Obtain camera data of a smart camera in a vehicle-mounted terminal, wherein the smart camera integrates a basic camera, an infrared camera, and a depth camera, and the camera data includes RGB video data, infrared video data, and depth video data;

[0007] Respectively perform frame processing on the RGB video data, infrared video data and depth video data to obtain multiple frames of RGB images, infrared images and depth images;

[0008] Extracting features from the RGB images, infrared images and depth images of the plurality of frames respectively to obtain initial RGB features, initial infrared features and initial depth features;

[0009] Perform feature interaction and feature shifting between the initial RGB features, initial infrared features and initial depth features corresponding to each frame of RGB image, infrared image and depth image, and obtain optimized RGB features, optimized infrared features and optimized depth features respectively;

[0010] The optimized RGB features, the optimized infrared features, and the optimized depth features are input into a preset pedestrian intention recognition model, and the pedestrian intention recognition model recognizes the intention type of the target pedestrian in the camera data according to the input optimized RGB features, the optimized infrared features, and the optimized depth features;

[0011] Whether to issue a driving warning is determined based on the intention type and the current driving strategy of the vehicle-mounted terminal.

[0012] Optionally, in a first implementation of the first aspect of the present invention, the extracting features of the multiple frames of RGB images, infrared images, and depth images to obtain initial RGB features, initial infrared features, and initial depth features comprises:

[0013] Performing image preprocessing on multiple frames of RGB images, infrared images and depth images respectively, and inputting the preprocessed multiple frames of RGB images, infrared images and depth images into corresponding feature extraction networks respectively, wherein the feature extraction network is a deep residual network;

[0014] Extracting features of different levels from the input RGB image, infrared image or depth image through multiple residual blocks in the deep residual network to obtain corresponding deep feature representations;

[0015] Performing nonlinear mapping on the deep feature representation so that the deep feature representation is mapped to a high-dimensional space to obtain corresponding high-dimensional features;

[0016] The high-dimensional features are compressed by global average pooling in the deep residual network to obtain corresponding initial RGB features, initial infrared features or initial depth features.

[0017] Optionally, in a second implementation of the first aspect of the present invention, performing feature interaction and feature shifting between initial RGB features, initial infrared features, and initial depth features corresponding to each frame of RGB image, infrared image, and depth image to obtain optimized RGB features, optimized infrared features, and optimized depth features, respectively, includes:

[0018] Performing time synchronization processing on each frame of RGB image, infrared image and depth image, so that each frame of RGB image, infrared image and depth image is time-aligned;

[0019] The time-aligned RGB images, infrared images, and depth images are modally combined in pairs to obtain three groups of first combined images;

[0020] Performing feature interaction between initial RGB features, initial infrared features or initial depth features corresponding to any combined image in the three groups of combined images to obtain interactive optimization features corresponding to each frame of RGB image, infrared image and depth image;

[0021] Combining the RGB images, infrared images and depth images of each frame according to the corresponding cameras to obtain three groups of second combined images;

[0022] Feature shifting is performed between each frame of RGB image, infrared image or depth image in the second combined image and the RGB image, infrared image or depth image of the previous and next frames to obtain optimized RGB features, optimized infrared features and optimized depth features respectively.

[0023] Optionally, in a third implementation of the first aspect of the present invention, the feature interaction is performed between the initial RGB features, initial infrared features, or initial depth features corresponding to any combined image in the three groups of combined images to obtain the interactive optimization features corresponding to each frame of the RGB image, infrared image, and depth image, including:

[0024] Select the last k feature channels of the initial RGB features, initial infrared features or initial depth features corresponding to any combined image in the three groups of combined images, and use the feature channels as target channels, wherein k is a natural number not less than 1;

[0025] The target channels between the initial RGB features, the initial infrared features or the initial depth features in any of the three groups of combined images are exchanged to obtain interactive optimization features corresponding to each frame of the RGB image, the infrared image and the depth image.

[0026] Optionally, in a fourth implementation of the first aspect of the present invention, the performing feature shift between each frame of the RGB image, infrared image or depth image in the second combined image and the RGB image, infrared image or depth image of the previous and next frames to obtain optimized RGB features, optimized infrared features and optimized depth features respectively includes:

[0027] Determine a target frame in the second combined image, and determine a correspondence between an RGB image, an infrared image, or a depth image of the target frame and RGB images, infrared images, or depth images of a previous frame and a next frame of the target frame;

[0028] Selecting the first 2i feature channels in the interactive optimization features of the RGB image, infrared image or depth image of the target frame as forward shift target channels, where i is a predetermined number of feature channels;

[0029] Performing feature replacement on the forward shifted target channel of the target frame and the feature channel corresponding to the previous frame of the target frame;

[0030] Selecting the last 2i feature channels in the interactive optimization features of the RGB image, infrared image or depth image of the target frame as the backward shift target channels, where i is the predetermined number of feature channels;

[0031] Performing feature replacement on the backward shifted target channel of the target frame and the feature channel corresponding to the next frame of the target frame;

[0032] The interactive optimization features after feature replacement are aggregated to obtain optimized RGB features, optimized infrared features and optimized depth features respectively.

[0033] Optionally, in a fifth implementation of the first aspect of the present invention, the inputting the optimized RGB features, the optimized infrared features, and the optimized depth features into a preset pedestrian intention recognition model, and identifying the intention type of the target pedestrian in the camera data according to the input optimized RGB features, the optimized infrared features, and the optimized depth features by the pedestrian intention recognition model includes:

[0034] Inputting the optimized RGB features, optimized infrared features and optimized depth features into a preset pedestrian intention recognition model, wherein the pedestrian intention recognition model includes an input layer, an attention mechanism layer, a feature fusion layer, a classification layer and an output layer;

[0035] The input optimized RGB features, optimized infrared features, and optimized depth features are standardized through the output layer;

[0036] The optimized RGB features, the optimized infrared features and the optimized depth features after the standardized processing are subjected to feature fusion through the attention mechanism layer and the feature fusion layer to obtain a fused vector feature;

[0037] The classification layer performs nonlinear mapping according to the fusion vector features to obtain the intention type of the target pedestrian in the camera data, and the intention type is output through the output layer.

[0038] Optionally, in a sixth implementation of the first aspect of the present invention, judging whether to perform a driving warning according to the intention type and the current driving strategy of the vehicle-mounted terminal includes:

[0039] Calculating a potential impact value of the target pedestrian according to the intention type and the driving parameters of the current driving strategy;

[0040] Calculating a trigger warning threshold of the vehicle-mounted terminal according to the driving parameters, environmental parameters of the vehicle-mounted terminal corresponding to the environment of the vehicle, and a predetermined safety distance parameter;

[0041] Determining the security risk level of the vehicle-mounted terminal according to the potential impact value and the trigger warning threshold;

[0042] If the safety risk level exceeds a preset safety threshold, a driving warning is performed, otherwise no driving warning is performed.

[0043] A second aspect of the present invention provides a vehicle driving warning device based on camera feature extraction, the vehicle driving warning device based on camera feature extraction comprising:

[0044] A data acquisition module is used to acquire camera data of a smart camera in a vehicle-mounted terminal, wherein the smart camera integrates a basic camera, an infrared camera and a depth camera, and the camera data includes RGB video data, infrared video data and depth video data;

[0045] A framing module is used to perform frame processing on the RGB video data, infrared video data and depth video data respectively to obtain multiple frames of RGB images, infrared images and depth images;

[0046] A feature extraction module is used to extract features from multiple frames of RGB images, infrared images and depth images to obtain initial RGB features, initial infrared features and initial depth features;

[0047] The feature processing module is used to perform feature interaction and feature shift between the initial RGB features, initial infrared features and initial depth features corresponding to each frame of RGB image, infrared image and depth image, and obtain optimized RGB features, optimized infrared features and optimized depth features respectively;

[0048] A model input module, used to input the optimized RGB features, optimized infrared features and optimized depth features into a preset pedestrian intention recognition model, and identify the intention type of the target pedestrian in the camera data according to the input optimized RGB features, optimized infrared features and optimized depth features through the pedestrian intention recognition model;

[0049] The warning judgment module is used to judge whether to issue a driving warning based on the intention type and the current driving strategy of the vehicle terminal.

[0050] The third aspect of the present invention provides a traffic warning device based on camera feature extraction, comprising: a memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected through lines; the at least one processor calls the instructions in the memory to enable the traffic warning device based on camera feature extraction to perform the steps of the above-mentioned traffic warning method based on camera feature extraction.

[0051] The fourth aspect of the present invention provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps of the above-mentioned driving warning method based on camera feature extraction.

[0052] The above-mentioned traffic warning method and related equipment based on camera feature extraction perform frame processing on the acquired multimodal video data, and then extract features from the framed images through a deep residual network to obtain their respective initial features. Then, the initial features corresponding to the RGB image, infrared image and depth image are optimized through feature interaction and feature shifting technology to obtain optimized features. Finally, the optimized features are input into the preset pedestrian intention recognition model, and the model identifies the intention type of the target pedestrian in the camera data based on the input features. According to the identified intention type and the current driving strategy of the vehicle terminal. This method improves the accuracy and real-time performance of traffic warnings by comprehensively utilizing multimodal data, especially its adaptability at night or in complex environments, and effectively improves the safety of road driving.

[0053] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0054] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram of an embodiment of a vehicle warning method based on camera feature extraction in an embodiment of the present invention;

[0056] Figure 2 A schematic diagram of an embodiment of a vehicle warning device based on camera feature extraction in an embodiment of the present invention;

[0057] Figure 3 It is a schematic diagram of an embodiment of a traffic warning device based on camera feature extraction in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] The terms "including" and "having" and any variations thereof mentioned in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device end including a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or device ends.

[0060] To facilitate understanding of this embodiment, a vehicle warning method based on camera feature extraction disclosed in an embodiment of the present invention is first described in detail. Figure 1 As shown, the method comprises the following steps:

[0061] 101. Obtain camera data of a smart camera in a vehicle-mounted terminal, wherein the smart camera integrates a basic camera, an infrared camera, and a depth camera, and the camera data includes RGB video data, infrared video data, and depth video data;

[0062] In one embodiment of the present invention, during the operation of the vehicle terminal, three different types of cameras are activated simultaneously to capture real-time data from the vehicle's surroundings. The basic camera captures images under the visible light spectrum, namely RGB video data, which is the basis of most visual processing systems. The data captured by the infrared camera can reflect the heat distribution in the environment, which is particularly important for pedestrian detection at night or under conditions of limited vision. The depth camera measures the distance between the object and the camera by emitting and receiving reflected signals to generate depth video data, which helps to understand the three-dimensional structure of the scene. In the specific implementation steps, the data captured by each camera first undergoes a preprocessing stage to improve the data quality and prepare for subsequent analysis. For example, RGB data may be optimized by color balancing and noise filtering, infrared data is standardized by temperature correction, and depth data requires depth correction and smoothing to eliminate possible measurement noise.

[0063] 102. Perform frame processing on the RGB video data, the infrared video data and the depth video data respectively to obtain multiple frames of RGB images, infrared images and depth images;

[0064] In one embodiment of the present invention, the continuous video stream captured from the smart camera is first decoded, and the result of the decoding is a series of continuous image frames, each of which is an independent image and can be processed separately. Next, in order to extract useful information from the decoded video frames and perform frame processing, a frame selection algorithm is adopted. The goal of this algorithm is to reduce the amount of data that needs to be processed while ensuring that key information is not lost. A common method is fixed frame rate sampling, however, only using fixed frame rate sampling may not fully utilize the dynamic information in the video data. Therefore, a content-based dynamic frame selection method is further introduced. The basic idea of ​​this method is to dynamically adjust the frame selection frequency according to the changes in the video content, which can be achieved by calculating the difference between consecutive frames. By combining fixed frame rate sampling and content-based dynamic frame selection, representative frames can be effectively extracted from RGB video data, infrared video data, and depth video data.

[0065] 103. Perform feature extraction on multiple frames of RGB images, infrared images and depth images respectively to obtain initial RGB features, initial infrared features and initial depth features;

[0066] In one embodiment of the present invention, the feature extraction of the multiple frames of RGB images, infrared images and depth images respectively to obtain initial RGB features, initial infrared features and initial depth features includes: performing image preprocessing on the multiple frames of RGB images, infrared images and depth images respectively, and inputting the preprocessed multiple frames of RGB images, infrared images and depth images into corresponding feature extraction networks respectively, wherein the feature extraction network is a deep residual network; performing feature extraction at different levels on the input RGB image, infrared image or depth image through multiple residual blocks in the deep residual network to obtain corresponding deep feature representations; performing nonlinear mapping on the deep feature representation so that the deep feature representation is mapped to a high-dimensional space to obtain corresponding high-dimensional features; and performing feature compression on the high-dimensional features through global average pooling in the deep residual network to obtain corresponding initial RGB features, initial infrared features or initial depth features.

[0067] Specifically, in the image preprocessing stage, normalization, denoising, and enhancement techniques are used for RGB images, infrared images, and depth images. Normalization is achieved by subtracting the mean of the image and dividing it by its standard deviation, which helps to reduce the differences in brightness and contrast between different images. Then, these preprocessed images are fed into the deep residual network (ResNet) for feature extraction. The deep residual network solves the gradient vanishing problem in deep network training by introducing residual blocks, so that deep features can be effectively extracted. Subsequently, the deep feature representation extracted from the deep residual network is further converted to a high-dimensional space through non-linear mapping to enhance the expressive power of the features. Non-linear mapping is usually achieved through activation functions (such as ReLU or Sigmoid). Finally, the high-dimensional features are compressed through global average pooling (GAP) to obtain a compact feature representation. Global average pooling can be expressed as:

[0068] ;

[0069] Among them, R is the high-dimensional feature after the i-th feature mapping, N is the total number of features, and G is the feature vector after global average pooling. Through the above steps, representative features can be effectively extracted from RGB images, infrared images, and depth images. These features not only contain rich information of the original images, but also have been processed and optimized by the deep learning model, and have higher discrimination ability and stronger expression ability. For example, assuming that in a scene of night driving, due to poor lighting conditions, RGB images may not provide enough information, and infrared images and depth images are particularly important at this time. After preprocessing and feature extraction network processing, the deep feature representation of infrared and depth data under these conditions can be obtained, and these features are further processed through nonlinear mapping and global average pooling, and finally a compact feature vector that can accurately describe pedestrians and other objects in the night driving environment is obtained.

[0070] 104. Perform feature interaction and feature shifting on initial RGB features, initial infrared features, and initial depth features corresponding to each frame of RGB image, infrared image, and depth image to obtain optimized RGB features, optimized infrared features, and optimized depth features, respectively;

[0071] In one embodiment of the present invention, the feature interaction and feature shifting between the initial RGB features, initial infrared features and initial depth features corresponding to each frame of RGB image, infrared image and depth image to obtain optimized RGB features, optimized infrared features and optimized depth features respectively include: performing time synchronization processing on each frame of RGB image, infrared image and depth image so that each frame of RGB image, infrared image and depth image is time-aligned; performing two-by-two modal combination of the time-aligned RGB images, infrared images and depth images to obtain three groups of first combined images; and performing time synchronization processing on the three groups of combined images to obtain optimized RGB features, optimized infrared features and optimized depth features respectively. The method comprises the following steps: performing feature interaction between initial RGB features, initial infrared features or initial depth features corresponding to any combined image in the image to obtain interactive optimization features corresponding to each frame of RGB image, infrared image and depth image; combining the RGB images, infrared images and depth images according to the corresponding cameras to obtain three groups of second combined images; performing feature shift between each frame of RGB image, infrared image or depth image in the second combined image and the RGB images, infrared images or depth images of the previous and next frames to obtain optimized RGB features, optimized infrared features and optimized depth features respectively.

[0072] Specifically, before the time synchronization process, it is necessary to ensure that each frame of image or data has an accurate timestamp. These timestamps are usually automatically recorded by the corresponding camera when the image is captured. The data streams of different modalities are synchronized according to the timestamp. This can be achieved by finding the frame with the closest timestamp, that is, for a given RGB frame, find the infrared and depth frames whose timestamps are closest to the timestamp of the RGB frame. In some cases, there may not be a completely matching timestamp. At this time, interpolation techniques can be used to generate image frames at the missing time points. For example, if a modality is missing data at a specific time point, the image frame at that time point can be estimated by interpolating the data of adjacent frames.

[0073] After time synchronization, the features between different modalities are integrated by shifting the features in the modal dimension. The core of the DFS model is the dual feature shift mechanism, which includes the modal feature interaction module and the neighborhood feature propagation module. The modal feature interaction module integrates the features between different modalities by shifting the features in the modal dimension. The mode and T-frame video clips of different modalities are obtained by combining the last frame of a modality The feature channels of the modality are merged with the feature channels of the other modality to form updated features. Specifically, for the timestamp The next frame and , the modal feature interaction can be expressed as:

[0074]

[0075] in, represents the vector connection operation through the channel dimension. This operation does not involve multiplication calculations and effectively achieves efficient integration of features between different modalities. Feature shifting further shifts features in the time dimension of the single-modal frame by propagating information in both the forward and backward directions of the time dimension. T-frame video clips of each modality Timestamp The next frame The features can be obtained from the neighboring frames and Shift head and tail 2 feature channels to update:

[0076] ;

[0077] Furthermore, the feature interaction between the initial RGB features, initial infrared features or initial depth features corresponding to any combined image in the three groups of combined images to obtain the interactive optimization features corresponding to each frame of RGB image, infrared image and depth image includes: selecting the last k feature channels of the initial RGB features, initial infrared features or initial depth features corresponding to any combined image in the three groups of combined images, and using the feature channels as target channels, wherein k is a natural number not less than 1; feature channel exchange of the target channels between the initial RGB features, initial infrared features or initial depth features in any combined image in the three groups of combined images to obtain the interactive optimization features corresponding to each frame of RGB image, infrared image and depth image.

[0078] Specifically, first, for each frame of the image, the system extracts the initial features of the RGB, infrared, and depth images. These initial features are extracted by deep learning models, such as convolutional neural networks (CNNs), which can capture rich information and details of the image. After extracting the features, a parameter is defined , represents the number of target channels to be selected from each modal feature. is a natural number not less than 1, which determines the exact range of feature channels to be exchanged. In the specific implementation steps, for each modality (RGB, infrared, depth) the initial features extracted , ,and , each feature is a three-dimensional tensor with size ,in Represents the number of channels, and Represent the height and width of the image respectively. In order to perform feature exchange, first determine the final channels. This can be achieved through a simple slicing operation, that is, for the RGB modality, the target channel can be expressed as . Similarly, target channels can also be determined for infrared and depth features. Feature interaction operations are then performed on the target channels. In addition, for depth images, features can be exchanged with RGB or infrared images in the same way to further optimize feature representation. This feature channel exchange strategy not only promotes information fusion between different modalities, but also helps the model capture richer and more diverse behavioral features, thereby improving the accuracy and robustness of behavioral intention recognition.

[0079] Furthermore, the feature shifting between each frame of the RGB image, infrared image or depth image in the second combined image and the RGB image, infrared image or depth image of the previous and next frames to obtain the optimized RGB feature, optimized infrared feature and optimized depth feature respectively includes: determining the target frame in the second combined image, and determining the correspondence between the RGB image, infrared image or depth image of the target frame and the RGB image, infrared image or depth image of the previous frame and the next frame of the target frame; selecting the first 2i feature channels in the interactive optimization feature of the RGB image, infrared image or depth image of the target frame as the previous The method comprises the following steps: a forward shift target channel, wherein i is a predetermined number of feature channels; performing feature replacement on the forward shift target channel of the target frame and the feature channel corresponding to the previous frame of the target frame; selecting the last 2i feature channels in the interactive optimization features of the RGB image, infrared image or depth image of the target frame as the backward shift target channel, wherein i is a predetermined number of feature channels; performing feature replacement on the backward shift target channel of the target frame and the feature channel corresponding to the next frame of the target frame; aggregating the interactive optimization features after feature replacement to obtain optimized RGB features, optimized infrared features and optimized depth features respectively.

[0080] Specifically, determining the target frame and its corresponding relationship with the previous and next frames is the basis for realizing dynamic interaction of features. In a multimodal video sequence, each frame (i.e., the target frame) is accompanied by its specific timestamp. Through the timestamp, the previous frame and the next frame of the target frame can be accurately identified. Specifically, for RGB images, infrared images, or depth images, the interactive optimization features of these images can be represented as a high-dimensional feature vector, which contains a large number of channels, each of which carries certain feature information of the image. The predetermined number of feature channels, i, is used to select specific feature channels for forward and backward shift operations. Specifically, the first 2, i feature channels are selected from the interactive optimization features of the target frame as the forward shift target channels, and the last 2, i feature channels are selected as the backward shift target channels. This selection is based on experiments and experience, aiming to capture the feature information with the most significant dynamic changes. Then the feature shift operation is performed as above, and after the feature replacement is completed, all features are aggregated. This step is achieved through feature fusion technology, which takes into account the replaced features and the original features, and combines them through weighted sum or other advanced aggregation methods to form the final optimized features. In this way, optimized RGB features, infrared features and depth features can be obtained respectively, and each feature integrates the dynamic information in the time series.

[0081] 105. Inputting the optimized RGB features, the optimized infrared features, and the optimized depth features into a preset pedestrian intention recognition model, and using the pedestrian intention recognition model to recognize the intention type of the target pedestrian in the camera data according to the input optimized RGB features, the optimized infrared features, and the optimized depth features;

[0082] In one embodiment of the present invention, the step of inputting the optimized RGB features, optimized infrared features and optimized depth features into a preset pedestrian intention recognition model, and identifying the intention type of the target pedestrian in the camera data through the pedestrian intention recognition model according to the input optimized RGB features, optimized infrared features and optimized depth features includes: inputting the optimized RGB features, optimized infrared features and optimized depth features into a preset pedestrian intention recognition model, wherein the pedestrian intention recognition model includes an input layer, an attention mechanism layer, a feature fusion layer, a classification layer and an output layer; standardizing the input optimized RGB features, optimized infrared features and optimized depth features through the output layer; performing feature fusion on the standardized optimized RGB features, optimized infrared features and optimized depth features through the attention mechanism layer and the feature fusion layer to obtain fused vector features; performing nonlinear mapping through the classification layer according to the fused vector features to obtain the intention type of the target pedestrian in the camera data, and outputting the intention type through the output layer.

[0083] Specifically, first, the optimized RGB features, infrared features, and depth features are used as input data and input to the input layer of the pedestrian intention recognition model. In the input layer, these features are first normalized to eliminate the impact of different scales and dimensions on model training and prediction. Next, the normalized features are sent to the attention mechanism layer of the model. In this layer, the model assigns different weights to different features by learning the correlation between features to highlight the features that are more important for pedestrian intention recognition. After that, the weighted features are passed to the feature fusion layer, where features from different modalities are effectively integrated into a fused vector feature. Feature fusion can be achieved in a variety of ways, such as simple concatenation, weighted summation, or more complex nonlinear fusion strategies. The fused vector feature combines the information of RGB images, infrared images, and depth images, providing a comprehensive perspective for pedestrian intention recognition. The fused vector feature is then sent to the classification layer, which converts the fused vector feature into a predicted intent type through nonlinear mapping. Finally, the model outputs the predicted intent type through the output layer to provide decision support for downstream tasks. The output layer usually includes one or more fully connected layers and activation functions to ensure that the output intent type has clear classification boundaries.

[0084] 106. Determine whether to issue a driving warning based on the intention type and the current driving strategy of the vehicle terminal.

[0085] In one embodiment of the present invention, judging whether to perform a driving warning based on the intention type and the current driving strategy of the vehicle-mounted terminal includes: calculating the potential impact value of the target pedestrian based on the intention type and the driving parameters of the current driving strategy; calculating the triggering warning threshold of the vehicle-mounted terminal based on the driving parameters, the environmental parameters of the vehicle environment corresponding to the vehicle-mounted terminal and the predetermined safety distance parameters; determining the safety risk level of the vehicle-mounted terminal based on the potential impact value and the triggering warning threshold; if the safety risk level exceeds the preset safety threshold, a driving warning is performed, otherwise no driving warning is performed.

[0086] Specifically, the calculation of the potential impact value should not only consider the pedestrian's intention type, but also the vehicle's current driving conditions and reaction time. This can be achieved through a weighted multi-factor function that comprehensively considers factors such as the potential danger of pedestrian behavior, the impact of vehicle speed, and the driver's reaction time:

[0087]

[0088] in, It is a risk scoring function mapped according to the pedestrian intention type. The function can return a specific value according to the different intentions, indicating the potential risk level of the pedestrian behavior. The nonlinear characteristics of the impact of vehicle speed on safety are taken into account to ensure sensitivity to safety risks when driving at high speeds. It reflects the inverse relationship between the reaction distance and safety. is a small constant used to avoid division by zero. is the weight coefficient, which is used to adjust the relative importance of different factors in the calculation of the total impact value. : Potential impact value, which quantifies the potential impact of the target pedestrian’s behavior on vehicle safety. The pedestrian’s intention type, such as crossing the road, is predicted by the pedestrian intention recognition model. : The current speed of the vehicle, reflecting the driving status of the vehicle. Reaction distance refers to the distance required from detecting a potential hazard to the vehicle coming to a complete stop. : A small constant used to avoid division by zero in calculations.

[0089] Specifically, the threshold for triggering the warning takes into account the vehicle's driving parameters, the complexity of the environment, and the safety distance parameters, and can be calculated using the following formula:

[0090]

[0091] It is an environmental complexity scoring function calculated based on environmental parameters, which can comprehensively consider factors such as weather conditions, traffic conditions, and road types.

[0092] It represents the exponential decay effect of the safety distance parameter on the warning threshold, ensuring the sensitivity of the warning system at a smaller safety distance. is the weight coefficient, which is used to adjust the influence of different factors in calculating the threshold. It is the threshold for triggering an early warning, which is a safety parameter calculated based on the vehicle's current state and environmental conditions. It is an environmental parameter, including factors such as weather conditions, traffic conditions, and road types. is a predetermined safety distance parameter, which is the minimum distance required to maintain safety under specific environment and conditions. Finally, according to the potential impact value and the trigger warning threshold, the safety risk level of the vehicle terminal is determined, and it is determined whether to issue a warning.

[0093] In this embodiment, the acquired multimodal video data is framed, and then the features of the framed images are extracted through a deep residual network to obtain the respective initial features. Then, the initial features corresponding to the RGB image, infrared image and depth image are optimized through feature interaction and feature shifting technology to obtain optimized features. Finally, the optimized features are input into the preset pedestrian intention recognition model, and the model identifies the intention type of the target pedestrian in the camera data based on the input features. According to the identified intention type and the current driving strategy of the vehicle terminal. This method improves the accuracy and real-time performance of driving warnings by comprehensively utilizing multimodal data, especially its adaptability at night or in complex environments, and effectively improves the safety of road driving.

[0094] The above describes the vehicle warning method based on camera feature extraction in the embodiment of the present invention. The following describes the vehicle warning device based on camera feature extraction in the embodiment of the present invention. Figure 2 In one embodiment of the present invention, a vehicle warning device based on camera feature extraction includes:

[0095] A data acquisition module is used to acquire camera data of a smart camera in a vehicle-mounted terminal, wherein the smart camera integrates a basic camera, an infrared camera and a depth camera, and the camera data includes RGB video data, infrared video data and depth video data;

[0096] A framing module is used to perform frame processing on the RGB video data, infrared video data and depth video data respectively to obtain multiple frames of RGB images, infrared images and depth images;

[0097] A feature extraction module is used to extract features from multiple frames of RGB images, infrared images and depth images to obtain initial RGB features, initial infrared features and initial depth features;

[0098] The feature processing module is used to perform feature interaction and feature shift between the initial RGB features, initial infrared features and initial depth features corresponding to each frame of RGB image, infrared image and depth image, and obtain optimized RGB features, optimized infrared features and optimized depth features respectively;

[0099] A model input module, used to input the optimized RGB features, optimized infrared features and optimized depth features into a preset pedestrian intention recognition model, and identify the intention type of the target pedestrian in the camera data according to the input optimized RGB features, optimized infrared features and optimized depth features through the pedestrian intention recognition model;

[0100] The warning judgment module is used to judge whether to issue a driving warning based on the intention type and the current driving strategy of the vehicle terminal.

[0101] In an embodiment of the present invention, the driving warning device based on camera feature extraction runs the above-mentioned driving warning method based on camera feature extraction. The driving warning device based on camera feature extraction performs frame processing on the acquired multimodal video data, and then extracts features from the framed images through a deep residual network to obtain respective initial features. Then, the initial features corresponding to the RGB image, infrared image and depth image are optimized through feature interaction and feature shifting technology to obtain optimized features. Finally, the optimized features are input into a preset pedestrian intention recognition model, and the model identifies the intention type of the target pedestrian in the camera data based on the input features. According to the identified intention type and the current driving strategy of the vehicle-mounted terminal. This method improves the accuracy and real-time performance of driving warnings by comprehensively utilizing multimodal data, especially the adaptability at night or in complex environments, and effectively improves the safety of road driving.

[0102] above Figure 2 The traffic warning device based on camera feature extraction in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The traffic warning device based on camera feature extraction in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0103] Figure 3 1 is a schematic diagram of the structure of a traffic warning device based on camera feature extraction provided by an embodiment of the present invention. The traffic warning device 300 based on camera feature extraction may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 310 (for example, one or more processors) and a memory 320, and one or more storage media 330 (for example, one or more mass storage device terminals) storing application programs 333 or data 332. Among them, the memory 320 and the storage medium 330 may be short-term storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the traffic warning device 300 based on camera feature extraction. Furthermore, the processor 310 may be configured to communicate with the storage medium 330, and execute a series of instruction operations in the storage medium 330 on the traffic warning device 300 based on camera feature extraction to implement the steps of the above-mentioned traffic warning method based on camera feature extraction.

[0104] The driving warning device 300 based on camera feature extraction may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input and output interfaces 360, and / or one or more operating systems 331, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that Figure 3 The structure of the traffic warning device based on camera feature extraction shown does not constitute a limitation on the traffic warning device based on camera feature extraction provided by the present invention, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0105] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the driving warning method based on camera feature extraction.

[0106] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device, or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0107] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0108] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A driving warning method based on camera feature extraction, characterized in that: The driving warning method comprises: Obtain camera data of a smart camera in a vehicle-mounted terminal, wherein the smart camera integrates a basic camera, an infrared camera, and a depth camera, and the camera data includes RGB video data, infrared video data, and depth video data; Respectively perform frame processing on the RGB video data, infrared video data and depth video data to obtain multiple frames of RGB images, infrared images and depth images; Extracting features from the RGB images, infrared images and depth images of the plurality of frames respectively to obtain initial RGB features, initial infrared features and initial depth features; The invention relates to a method for obtaining an optimized RGB feature, an optimized infrared feature and an optimized depth feature by performing feature interaction and feature shifting on the initial RGB features, the initial infrared features and the initial depth features corresponding to each frame of the RGB image, the infrared image and the depth image, respectively; the method comprises: performing time synchronization processing on each frame of the RGB image, the infrared image and the depth image, so that each frame of the RGB image, the infrared image and the depth image is time-aligned; and performing time synchronization processing on the RGB image, the infrared image and the depth image after time alignment. The images and depth images are modally combined in pairs to obtain three groups of first combined images; feature interaction is performed between the initial RGB features, initial infrared features or initial depth features corresponding to any combined image in the three groups of combined images to obtain interactive optimization features corresponding to each frame of RGB image, infrared image and depth image; image combination is performed on each frame of RGB image, infrared image and depth image according to the corresponding camera to obtain three groups of second combined images; feature shift is performed between each frame of RGB image, infrared image or depth image in the second combined image and the RGB image, infrared image or depth image of the previous and next frames to obtain optimized RGB features, optimized infrared features and optimized depth features respectively; The optimized RGB features, the optimized infrared features, and the optimized depth features are input into a preset pedestrian intention recognition model, and the pedestrian intention recognition model recognizes the intention type of the target pedestrian in the camera data according to the input optimized RGB features, the optimized infrared features, and the optimized depth features; Whether to issue a driving warning is determined based on the intention type and the current driving strategy of the vehicle-mounted terminal.

2. The driving warning method according to claim 1, characterized in that: The extracting features of the multiple frames of RGB images, infrared images and depth images to obtain initial RGB features, initial infrared features and initial depth features comprises: Performing image preprocessing on multiple frames of RGB images, infrared images and depth images respectively, and inputting the preprocessed multiple frames of RGB images, infrared images and depth images into corresponding feature extraction networks respectively, wherein the feature extraction network is a deep residual network; Extracting features of different levels from the input RGB image, infrared image or depth image through multiple residual blocks in the deep residual network to obtain corresponding deep feature representations; Performing nonlinear mapping on the deep feature representation so that the deep feature representation is mapped to a high-dimensional space to obtain corresponding high-dimensional features; The high-dimensional features are compressed by global average pooling in the deep residual network to obtain corresponding initial RGB features, initial infrared features or initial depth features.

3. The driving warning method according to claim 1, characterized in that: The feature interaction between the initial RGB features, initial infrared features or initial depth features corresponding to any combination of the three groups of combined images to obtain the interactive optimization features corresponding to each frame of the RGB image, infrared image and depth image includes: Select the last k feature channels of the initial RGB features, initial infrared features or initial depth features corresponding to any combined image in the three groups of combined images, and use the feature channels as target channels, wherein k is a natural number not less than 1; The target channels between the initial RGB features, the initial infrared features or the initial depth features in any of the three groups of combined images are exchanged to obtain interactive optimization features corresponding to each frame of the RGB image, the infrared image and the depth image.

4. The driving warning method according to claim 3, characterized in that: The step of performing feature shifting between each frame of the RGB image, infrared image or depth image in the second combined image and the RGB image, infrared image or depth image in the previous and next frames to obtain optimized RGB features, optimized infrared features and optimized depth features respectively comprises: Determine a target frame in the second combined image, and determine a correspondence between an RGB image, an infrared image, or a depth image of the target frame and RGB images, infrared images, or depth images of a previous frame and a next frame of the target frame; Selecting the first 2i feature channels in the interactive optimization features of the RGB image, infrared image or depth image of the target frame as forward shift target channels, where i is a predetermined number of feature channels; Performing feature replacement on the forward shifted target channel of the target frame and the feature channel corresponding to the previous frame of the target frame; Selecting the last 2i feature channels in the interactive optimization features of the RGB image, infrared image or depth image of the target frame as the backward shift target channels, where i is the predetermined number of feature channels; Performing feature replacement on the backward shifted target channel of the target frame and the feature channel corresponding to the next frame of the target frame; The interactive optimization features after feature replacement are aggregated to obtain optimized RGB features, optimized infrared features and optimized depth features respectively.

5. The driving warning method according to claim 1, characterized in that: The step of inputting the optimized RGB features, the optimized infrared features, and the optimized depth features into a preset pedestrian intention recognition model, and identifying the intention type of the target pedestrian in the camera data according to the input optimized RGB features, the optimized infrared features, and the optimized depth features by the pedestrian intention recognition model includes: Inputting the optimized RGB features, optimized infrared features and optimized depth features into a preset pedestrian intention recognition model, wherein the pedestrian intention recognition model includes an input layer, an attention mechanism layer, a feature fusion layer, a classification layer and an output layer; The input optimized RGB features, optimized infrared features, and optimized depth features are standardized through the output layer; The optimized RGB features, the optimized infrared features and the optimized depth features after the standardized processing are subjected to feature fusion through the attention mechanism layer and the feature fusion layer to obtain a fused vector feature; The classification layer performs nonlinear mapping according to the fusion vector features to obtain the intention type of the target pedestrian in the camera data, and the intention type is output through the output layer.

6. The driving warning method according to claim 1, characterized in that: The determining whether to perform a driving warning according to the intention type and the current driving strategy of the vehicle-mounted terminal includes: Calculating a potential impact value of the target pedestrian according to the intention type and the driving parameters of the current driving strategy; Calculating a trigger warning threshold of the vehicle-mounted terminal according to the driving parameters, environmental parameters of the vehicle-mounted terminal corresponding to the environment of the vehicle, and a predetermined safety distance parameter; Determining the security risk level of the vehicle-mounted terminal according to the potential impact value and the trigger warning threshold; If the safety risk level exceeds a preset safety threshold, a driving warning is performed, otherwise no driving warning is performed.

7. A vehicle warning device based on camera feature extraction, characterized in that: The driving warning device based on camera feature extraction includes: A data acquisition module is used to acquire camera data of a smart camera in a vehicle-mounted terminal, wherein the smart camera integrates a basic camera, an infrared camera and a depth camera, and the camera data includes RGB video data, infrared video data and depth video data; A framing module is used to perform frame processing on the RGB video data, infrared video data and depth video data respectively to obtain multiple frames of RGB images, infrared images and depth images; A feature extraction module is used to extract features from multiple frames of RGB images, infrared images and depth images to obtain initial RGB features, initial infrared features and initial depth features; The feature processing module is used to perform feature interaction and feature shift between the initial RGB features, initial infrared features and initial depth features corresponding to each frame of RGB image, infrared image and depth image, and obtain optimized RGB features, optimized infrared features and optimized depth features respectively; the feature interaction and feature shift between the initial RGB features, initial infrared features and initial depth features corresponding to each frame of RGB image, infrared image and depth image, and obtain optimized RGB features, optimized infrared features and optimized depth features respectively include: performing time synchronization processing on each frame of RGB image, infrared image and depth image, so that each frame of RGB image, infrared image and depth image is time-aligned; processing the time-aligned RGB images The image, infrared image and depth image are modally combined in pairs to obtain three groups of first combined images; the initial RGB features, initial infrared features or initial depth features corresponding to any combined image in the three groups of combined images are interactively interacted to obtain interactive optimization features corresponding to each frame of RGB image, infrared image and depth image; the RGB image, infrared image and depth image are combined according to the corresponding cameras to obtain three groups of second combined images; the features of each frame of RGB image, infrared image or depth image in the second combined image are shifted with the RGB images, infrared images or depth images of the previous and next frames to obtain optimized RGB features, optimized infrared features and optimized depth features respectively; A model input module, used to input the optimized RGB features, optimized infrared features and optimized depth features into a preset pedestrian intention recognition model, and identify the intention type of the target pedestrian in the camera data according to the input optimized RGB features, optimized infrared features and optimized depth features through the pedestrian intention recognition model; The warning judgment module is used to judge whether to issue a driving warning based on the intention type and the current driving strategy of the vehicle terminal.

8. A traffic warning device based on camera feature extraction, characterized in that: The vehicle warning device based on camera feature extraction includes: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory so that the traffic warning device based on camera feature extraction performs the steps of the traffic warning method based on camera feature extraction as described in any one of claims 1-6.

9. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instruction is executed by the processor, the steps of the driving warning method based on camera feature extraction as described in any one of claims 1-6 are implemented.

Images