Vehicle blind area monitoring method and device based on electronic rearview mirror and storage medium
By using the YOLOv5 algorithm and attention module in the electronic rearview mirror system, the target detection is optimized, and the problem of insufficient real-time and accuracy of the blind spot monitoring function is solved, the system complexity and cost are reduced, and the driver's safety is improved. Product competitiveness.
Patent Information
- Application Number
- CN202411065370.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-12-06
AI Technical Summary
The existing electronic rearview mirror system has blind spots in driving state, resulting in driver safety hazards. The existing technology blind spot monitoring function is insufficient in real time and accuracy under the influence of factors such as changes in lighting conditions, target occlusion and movement speed. The system High complexity and cost.
Using the object detection algorithm based on YOLOv5, combining the channel attention module and the spatial attention module, optimize the object detection model, obtain environmental information through high-definition cameras, perform preprocessing and feature extraction, generate channel and spatial attention feature maps, and improve object detection Accuracy and real-timeness reduce blind spots.
It improves the real-time and accuracy of the blind spot monitoring function of the electronic rearview mirror system, reduces system complexity and cost, enhances driver's driving safety, and enhances product competitiveness and highlights.
Smart Images

Figure CN119091416A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of new energy vehicle technology, and in particular to a vehicle blind spot monitoring method, device and storage medium based on an electronic rearview mirror. Background Art
[0002] At present, car rearview mirrors are an important tool for drivers to observe the situation behind them, and they play a vital role in driving. However, traditional rearview mirrors have a narrow field of view, especially when the vehicle changes lanes or turns, blind spots are prone to appear, which poses a safety hazard to the driver. In order to solve this problem, many manufacturers have launched electronic rearview mirror systems (Camera Monitor System, CMS) equipped with cameras, which expand the field of view through image transmission technology. However, the blind spot monitoring function of these systems is still limited and cannot fully guarantee driving safety.
[0003] With the development of science and technology, computer vision and target detection technology have been widely used in various fields, especially in the fields of autonomous driving, intelligent monitoring, and intelligent transportation. As an important part of the car, CMS's main function is to obtain environmental information around the vehicle through the camera so that the driver can make the right driving decision. However, due to factors such as the body structure, there are still blind spots when driving.
[0004] Existing solutions mainly include the following methods: 1. Improve camera design: Optimize the camera design to expand the field of view and reduce blind spots. 2. Joint observation with multiple cameras: Use multiple cameras for joint observation to achieve monitoring of a larger range. 3. Image processing and computer vision technology: Identify potential dangerous targets by processing the images obtained by the camera.
[0005] Although these solutions have improved the blind spot problem to a certain extent, the following problems still exist in practical applications: Improving camera design and using multi-camera solutions can reduce blind spots, but it will increase the complexity and cost of the system. In actual road applications, the existing image processing and computer vision technologies are affected by factors such as changing lighting conditions, target occlusion, and target movement speed, and the real-time and accuracy of target detection still need to be improved. Summary of the invention
[0006] In view of this, the embodiments of the present application provide a vehicle blind spot monitoring method, device and storage medium based on an electronic rearview mirror to solve the problems of high system complexity and cost, as well as insufficient real-time and accuracy of target detection in the prior art.
[0007] According to a first aspect of an embodiment of the present application, a vehicle blind spot monitoring method based on an electronic rearview mirror is provided, comprising: using a camera installed on the vehicle's electronic rearview mirror to collect environmental information around the vehicle, and preprocessing the environmental information to extract target information; inputting the target information into a target detection model, wherein the target detection model comprises a channel attention module and a spatial attention module, the channel attention module is used to compress the feature map in the spatial dimension and generate a channel attention feature map, and the spatial attention module is used to compress the feature map in the channel dimension and generate a spatial attention feature map; using the target detection model to detect and identify the target information, output the detection result, and perform blind spot monitoring based on the detection result.
[0008] According to a second aspect of an embodiment of the present application, a vehicle blind spot monitoring device based on an electronic rearview mirror is provided, comprising: an acquisition module, configured to use a camera installed on the vehicle's electronic rearview mirror to collect environmental information around the vehicle, and pre-process the environmental information to extract target information; an input module, configured to input the target information into a target detection model, wherein the target detection model includes a channel attention module and a spatial attention module, the channel attention module is used to compress the feature map in the spatial dimension and generate a channel attention feature map, and the spatial attention module is used to compress the feature map in the channel dimension and generate a spatial attention feature map; an output module, configured to detect and identify the target information using the target detection model, output the detection result, and perform blind spot monitoring based on the detection result.
[0009] According to a third aspect of an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the computer program.
[0010] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.
[0011] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:
[0012] The environmental information around the vehicle is collected by using a camera installed on the electronic rearview mirror of the vehicle, and the environmental information is preprocessed to extract the target information; the target information is input into the target detection model, wherein the target detection model includes a channel attention module and a spatial attention module, the channel attention module is used to compress the feature map in the spatial dimension and generate the channel attention feature map, and the spatial attention module is used to compress the feature map in the channel dimension and generate the spatial attention feature map; the target information is detected and identified using the target detection model, the detection results are output, and blind spot monitoring is performed based on the detection results. This application can reduce system complexity and cost, improve the real-time performance and accuracy of target detection, and thus improve the effect of vehicle blind spot monitoring. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0014] Figure 1 It is a schematic diagram of a driving blind spot in an actual scenario provided by an embodiment of the present application;
[0015] Figure 2 It is a flow chart of a vehicle blind spot monitoring method based on an electronic rearview mirror provided in an embodiment of the present application;
[0016] Figure 3 It is a schematic diagram of the overall architecture of the rearward blind spot monitoring algorithm of the electronic rearview mirror system provided in an embodiment of the present application;
[0017] Figure 4 It is a schematic diagram of the overall structure of the convolutional neural network CNN provided in the embodiment of the present application;
[0018] Figure 5 It is a schematic diagram of the overall structure of the convolutional attention module provided in the embodiment of the present application;
[0019] Figure 6 is a structural schematic diagram of a vehicle blind spot monitoring device based on an electronic rearview mirror provided in an embodiment of the present application;
[0020] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0021] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0022] At present, car rearview mirrors are an important tool for drivers to observe the situation behind them, and they play a vital role in driving. However, traditional rearview mirrors have a narrow field of view, especially when the vehicle changes lanes or turns, blind spots are prone to appear, which poses a safety hazard to the driver. In order to solve this problem, many manufacturers have launched electronic rearview mirror systems (Camera Monitor System, CMS) equipped with cameras, which expand the field of view through image transmission technology. However, the blind spot monitoring function of these systems is still limited and cannot fully guarantee driving safety.
[0023] With the development of science and technology, computer vision and target detection technology have been widely used in various fields, especially in the fields of autonomous driving, intelligent monitoring, and intelligent transportation. Among them, CMS, as an important part of the car, has the main function of obtaining environmental information around the vehicle through the camera so that the driver can make the right driving decision. Due to factors such as the body structure, there are blind spots when driving. Figure 1 is a schematic diagram of a driving blind spot in an actual scenario provided by an embodiment of the present application, such as Figure 1 As shown in the figure, when the rear-coming vehicle enters the blind spot, it will bring certain risks to driving safety.
[0024] To solve this problem, existing solutions mainly improve the design of cameras or use multiple cameras for joint observation to expand the camera's observation range and reduce blind spots. In addition, some solutions use image processing and computer vision technology to process the images obtained by the camera to identify potential dangerous targets.
[0025] However, there are still some problems with the existing technology in practical applications. First, although the existing camera design and multi-camera joint observation scheme can reduce blind spots, it will increase the complexity and cost of the system. Secondly, although the existing image processing and computer vision technology can identify potential dangerous targets to a certain extent, in actual road conditions, due to factors such as changes in lighting conditions, target occlusion, and target movement speed, the real-time and accuracy of target detection needs to be improved. Therefore, how to improve the blind spot monitoring function of CMS while ensuring real-time and accuracy is a problem that needs to be solved urgently.
[0026] Therefore, this application mainly solves the following technical problems: how to improve the blind spot monitoring function of CMS while ensuring real-time and accuracy; how to improve the accuracy and reliability of target detection by optimizing the target detection algorithm; how to effectively reduce the blind spot of the electronic rearview mirror system without increasing the complexity and cost of the system.
[0027] In view of the problems existing in the prior art, the present application provides a vehicle blind spot monitoring method based on an electronic rearview mirror. The technical solution of the present application mainly includes the following technical means:
[0028] 1. Use the camera of the electronic rearview mirror CMS to obtain target information: The camera of the electronic rearview mirror CMS installed on the vehicle can obtain the environmental information around the vehicle in real time, including target information such as other vehicles, pedestrians, obstacles, etc.
[0029] 2. Optimize the target detection algorithm based on YOLOv5: Input the acquired target information into the target detection algorithm based on YOL Ov5, which can detect and identify the target in real time and improve the accuracy and reliability of target detection. Among them, YOLOv5 is a target detection algorithm based on deep learning, with high detection speed and accuracy.
[0030] 3. Meet the real-time and accuracy requirements: By optimizing the target detection algorithm, the blind spot monitoring function of the electronic rearview mirror system (CMS) can meet the real-time and accuracy requirements of the detection target under actual road conditions.
[0031] 4. Reduce the blind spots of the electronic rearview mirror system: By optimizing the target detection algorithm, the blind spots of the electronic rearview mirror system can be effectively reduced, allowing the driver to obtain more accurate information and increase driving safety.
[0032] 5. Enhance product highlights and selling points: By optimizing the target detection algorithm and improving the blind spot monitoring function of the electronic rearview mirror system (CMS), the product will be more competitive and the product highlights and selling points will be enhanced.
[0033] The contents of the technical solution of the present application are described in detail below with reference to the accompanying drawings and specific embodiments.
[0034] Figure 2 It is a flow chart of a vehicle blind spot monitoring method based on an electronic rearview mirror provided in an embodiment of the present application. Figure 2 The vehicle blind spot monitoring method based on the electronic rearview mirror can be executed by the on-board chip of the new energy vehicle. Figure 2 As shown, the vehicle blind spot monitoring method based on the electronic rearview mirror may specifically include:
[0035] S201, collecting environmental information around the vehicle using a camera installed on the electronic rearview mirror of the vehicle, and preprocessing the environmental information to extract target information;
[0036] S202, inputting the target information into the target detection model, wherein the target detection model includes a channel attention module and a spatial attention module, the channel attention module is used to compress the feature map in the spatial dimension and generate the channel attention feature map, and the spatial attention module is used to compress the feature map in the channel dimension and generate the spatial attention feature map;
[0037] S203, using the target detection model to detect and identify the target information, output the detection result, and perform blind spot monitoring based on the detection result.
[0038] In some embodiments, a camera installed on the electronic rearview mirror of the vehicle is used to collect environmental information around the vehicle, and the environmental information is pre-processed to extract target information, including:
[0039] A camera installed on the vehicle's electronic rearview mirror is used to obtain environmental information around the vehicle in real time, and the obtained environmental information is preprocessed to extract preset target information from the preprocessed environmental information. The camera has a preset resolution and frame rate, and the preprocessing includes denoising, enhancement and format conversion. The target information includes other vehicles, pedestrians and obstacles around the vehicle.
[0040] Specifically, first, a high-definition camera installed on the electronic rearview mirror of the vehicle is used to obtain environmental information around the vehicle in real time. The high-definition camera has a preset resolution and frame rate. For example, two high-definition cameras with a resolution of 1920x1080 are used in this embodiment. The cameras are installed on the electronic rearview mirror bracket of the vehicle and can provide high-quality image data. The frame rate of the camera is 60 frames per second. Such a high frame rate can ensure that target information such as other vehicles, pedestrians, obstacles, etc. around the vehicle can still be captured in real time when driving at high speed.
[0041] Next, the acquired environmental information is preprocessed, and the preprocessing steps include denoising, enhancement, and format conversion. Denoising is to eliminate noise interference in the image, enhancement is to improve the contrast and clarity of the image, and format conversion is to convert the image data into a format suitable for subsequent processing.
[0042] The pre-processed environmental information will be further processed to extract the preset target information. The process of extracting target information includes identifying and separating target objects such as other vehicles, pedestrians and obstacles in the image. This information is important input data for the rear blind spot monitoring function.
[0043] In one example, Figure 3is a schematic diagram of the overall architecture of the rearward blind spot monitoring algorithm of the electronic rearview mirror system provided in the embodiment of the present application, such as Figure 3 As shown in Figure 1, the algorithm design architecture of the rear blind spot monitoring function of the electronic rearview mirror system (CMS) includes three parts: perception, decision-making, and execution. In the perception part, the pre-processed environmental information is analyzed and processed by the target signal processing module to generate target information output and vehicle status information.
[0044] In the decision-making part, the system first performs target selection to determine which target objects need further processing. Then, a safety distance model is constructed to judge potential dangers based on the distance between the target object and the vehicle. Next, a graded warning system is designed to generate corresponding warning levels according to different danger levels and output warning signals.
[0045] In the execution part, the system performs safety warning control and human-machine interaction control according to the warning signal. The safety warning control module can trigger the first-level image warning and the second-level audio and video alarm to remind the driver of potential dangers. The human-machine interaction control module combines the vehicle driving status and the driver's operation intention to provide reasonable driving suggestions, such as adjusting the speed and changing the driving trajectory.
[0046] Through the method of the above-mentioned embodiment, a high-definition camera installed on the vehicle's electronic rearview mirror is used to obtain and preprocess the environmental information around the vehicle in real time. Combined with the rear blind spot monitoring algorithm, the accuracy and real-time performance of the vehicle's blind spot monitoring are effectively improved, thereby enhancing driving safety.
[0047] In some embodiments, the target detection model further includes:
[0048] A global pooling layer and a fully connected layer are provided in the target detection model; wherein the global pooling layer is used to perform a global pooling operation on each feature layer to generate a feature vector; the fully connected layer is used to receive the feature vector after global pooling and determine the connection between the feature vectors to generate a weight division;
[0049] Both the channel attention module and the spatial attention module adopt the self-attention mechanism; among them, the self-attention mechanism is used to calculate the relationship between each feature unit, including the relationship between channels and channels and between pixels, and introduce weights in the channel dimension and spatial dimension so as to enhance the relationship between feature units by using weights.
[0050] Specifically, the Attention Mechanism (AM) is a computing resource allocation scheme for current neural network models. It can prioritize computing resources to more important tasks when computing power is limited. Generally, the more parameters a model has, the stronger its expressive power is, and the more information it stores, but this will lead to the problem of information overload. By introducing AM, we can focus on the more critical information in this task among a lot of input information, reduce the attention to other information, and even filter out irrelevant information, which can solve the problem of information overload and improve the efficiency and accuracy of the model.
[0051] In one example, X = (x1, x2, x3, ..., x n ,) represents n input information. Given a query vector q, define an attention variable z∈[1,N] to represent the index position of the selected information. Then the attention distribution α of the i-th input information i It is formula (1).
[0052]
[0053] In formula (1), s(x i ,q) is the attention scoring function, which usually uses the scoring function in the form of scaled dot product, that is, d is the dimension of the input information.
[0054] Furthermore, the Self-Attention Mechanism (SAM) introduces weights at the channel and space levels by calculating the values between each unit channel and between pixels, thereby strengthening the connection between each other and improving the feature extraction capability. Global Average Pooling (GAP) and Fully Connected Layer (FCL) are based on this, by globally pooling each feature layer, and then finding the connection between features in the fully connected layer, and finally obtaining the weight division.
[0055] Compared with traditional convolutional neural networks (CNN) and recurrent neural networks (RNN), SAM and GAP can better represent the long-range correlation of input information. Although gating mechanisms such as long short-term memory (LSTM) can introduce long-range correlation, their expressive power is still limited. On the other hand, fully connected networks can model long-range dependencies, but cannot handle variable-length input sequences, and the connection weights of different input lengths are also different. Therefore, SAM and GAP can be used to "dynamically" generate weights for different connections, so that the weights are related to the importance of the data itself, further optimizing the network's representation capabilities.
[0056] In SAM, assume that the input sequence is X = (x1, x2, x3, ..., x n ,), the output sequence is H = (h1,h2,h3,…,h n ,), firstly, three sets of vector sequences are obtained through linear transformation, as shown in formula (2).
[0057]
[0058] In formula (2), Q, K, and V are the query vector sequence, key vector sequence, and value vector sequence, respectively, all of which are obtained by linear transformation of X. The weight matrix W Q , W K , W V It can be adjusted dynamically through AM.
[0059] The output vector is obtained as formula (3).
[0060]
[0061] Written in matrix form as formula (4).
[0062]
[0063] SAM can be used as a layer in a neural network, or it can be used to replace a convolutional layer or a recurrent layer, or it can be cross-stacked with a convolutional layer or a recurrent layer.
[0064] Furthermore, the present application also uses the convolutional attention module to optimize the traditional YOLOv5 algorithm. CNN is a commonly used algorithm in deep learning, and has achieved amazing performance in many fields such as computer vision, speech recognition, and natural language processing. CNN is composed of multiple convolutional layers and pooling layers. Its structure is as follows Figure 4 As shown, Figure 4It is a schematic diagram of the overall structure of the convolutional neural network CNN provided in the embodiment of the present application. The input layer of CNN is usually preprocessed image data; the convolution layer is used to extract the features of the input data; the pooling layer can generally be divided into average pooling, maximum pooling and minimum pooling, and the convolution layer and the pooling layer can jointly realize the feature extraction function in traditional machine learning; the fully connected layer is usually placed at the tail of CNN to realize the connection between neurons at different levels; the output layer is used for the final result output. The convolution layer contains multiple convolution kernels, and each element of the convolution kernel corresponds to a weight coefficient and a bias. The area covered by the convolution kernel is the receptive field.
[0065] In some embodiments, the channel attention module is used to compress the feature map in the spatial dimension and generate a channel attention feature map, including:
[0066] In the channel attention module, global average pooling and maximum pooling operations are performed on the input feature map in the spatial dimension to generate a one-dimensional feature vector;
[0067] Input the one-dimensional feature vector into a shared multi-layer perceptron, which includes two fully connected layers, the first fully connected layer is used to output the compressed feature vector, and the second fully connected layer is used to output the initial channel attention feature map;
[0068] The initial channel attention feature map is element-wise multiplied with the input feature map and processed through the activation function to generate the final channel attention feature map.
[0069] Specifically, first, global average pooling and maximum pooling operations are performed on the input feature map in the spatial dimension. Global average pooling generates a one-dimensional feature vector by calculating the average value of all pixels in the feature map, while maximum pooling generates another set of one-dimensional feature vectors by selecting the maximum value of each pooling area in the feature map. These two operations generate two different feature vectors, which can aggregate the spatial information of the feature map.
[0070] Next, the one-dimensional feature vector generated above is input into a shared multi-layer perceptron (MLP). The multi-layer perceptron consists of two fully connected layers. The first fully connected layer compresses the input one-dimensional feature vector and generates a new feature vector. This step is mainly used to reduce the dimension of the feature vector and extract more compact features. Then, the second fully connected layer receives the compressed feature vector, further processes it and outputs the initial channel attention feature map.
[0071] Furthermore, after obtaining the initial channel attention feature map, the initial feature map is multiplied element-by-element with the input original feature map. The purpose of this step is to weight the original feature map through the channel attention feature map so that important features can be paid more attention in subsequent processing. Finally, the final channel attention feature map is generated through activation function processing. The activation function usually uses Sigmoid function or other nonlinear functions to enhance the expressiveness of the feature map.
[0072] In one example, the channel attention module of the convolutional attention module (CBA) focuses on the importance of features. By compressing in the spatial dimension, average pooling and maximum pooling operations can effectively aggregate the spatial information of feature maps and process them through a shared multi-layer perceptron. Average pooling has feedback for every pixel on the feature map, while maximum pooling only has gradient feedback for the largest response in the feature map during gradient back propagation calculation. This design can make full use of different spatial information to generate more accurate channel attention feature maps.
[0073] For example, in a specific application scenario, the feature map processed by the channel attention module can more accurately identify pedestrians and obstacles around the vehicle, thereby improving the overall performance of the target detection model. This module is particularly important in the electronic rearview mirror system because it can effectively reduce blind spots, provide more reliable environmental perception information, and ensure driving safety.
[0074] In a specific example, the input feature map F is subjected to global average pooling and maximum pooling respectively; then, it is sent to the Multi-Layer Perception (MLP) respectively, the number of neurons in the first layer is C / r, the activation function is Relu, the number of neurons in the second layer is C, and the weights of the neural network of the two layers are shared; finally, the features output by the MLP are summed up, and after the sigmoid activation function, M is finally generated. c (F), that is, the attention of the input feature map F on the channel, the calculation formula is as shown in formula (5).
[0075]
[0076] In formula (5), σ represents the Sigmoid activation function.
[0077] The code implementation is as follows:
[0078] out = self.conv2(out)
[0079] out=self.bn2(out)#[1,64,56,56]
[0080] ca_tmp=self.ca(out)#[1,64,1,1]
[0081] out=self.ca(out)*out#[1,64,56,56]
[0082] In some embodiments, the spatial attention module is used to compress the feature map in the channel dimension and generate a spatial attention feature map, including:
[0083] In the spatial attention module, global average pooling and maximum pooling operations are performed on the input feature map in the channel dimension to generate a two-dimensional feature map;
[0084] The two-dimensional feature maps are concatenated to form a feature map containing multiple channels, and a convolution layer is used to perform a convolution operation on the concatenated feature map to generate a weight map;
[0085] The weight map is processed through an activation function to generate an initial spatial attention feature map, which is element-wise multiplied with the input feature map to generate the final spatial attention feature map.
[0086] Specifically, first, in the spatial attention module, global average pooling and maximum pooling operations are performed on the input feature map in the channel dimension. Global average pooling generates a two-dimensional feature map by calculating the average value of all pixels in the feature map, while maximum pooling generates another set of two-dimensional feature maps by selecting the maximum value of each pooling area in the feature map. These two operations generate two different feature maps, which can aggregate the spatial information of the feature map.
[0087] Next, the two-dimensional feature maps generated above are concatenated to form a feature map containing multiple channels. This step combines the results of average pooling and maximum pooling, making full use of the information advantages brought by different pooling methods.
[0088] Then, the convolution layer is used to perform a convolution operation on the concatenated feature map to generate a weight map. The convolution operation extracts the feature information with spatial correlation in the feature map by sliding the convolution kernel on the feature map. In this process, a 7x7 convolution kernel is usually used, which has a better application effect than a 3x3 convolution kernel and can better capture spatial information.
[0089] Next, the weight map is processed through the activation function to generate the initial spatial attention feature map. The activation function usually uses the Sigmoid function, which enhances the expressiveness of the feature map by mapping the result of the convolution operation to a range of 0 to 1.
[0090] Finally, the initial spatial attention feature map is element-wise multiplied with the input feature map to generate the final spatial attention feature map. This step weights the input feature map through the weight map so that the spatially important features are strengthened, thereby improving the accuracy and reliability of feature extraction.
[0091] In one example, CBA's spatial attention module focuses on the importance of spatial locations. By compressing in the channel dimension, average pooling and maximum pooling operations can effectively aggregate the spatial information of feature maps and process them through convolutional layers and activation functions. Average pooling and maximum pooling extract the average and maximum values on each channel, respectively, and merge these results into a feature map containing multiple channels. Through convolution operations and activation function processing, a weight map is generated for weighted processing of the input feature map.
[0092] For example, in a specific application scenario, the feature map processed by the spatial attention module can more accurately identify pedestrians and obstacles around the vehicle, thereby improving the overall performance of the target detection model. This module is particularly important in the electronic rearview mirror system because it can effectively reduce blind spots, provide more reliable environmental perception information, and ensure driving safety.
[0093] In a specific example, since the CBA module adopts a residual structure, the input feature map of the spatial attention is the output of the channel attention The original feature map, as shown in formula (6), sets the input feature map of spatial attention to First, the maximum pooling and average pooling operations are used to aggregate the channel information and the results are connected; then, a 7×7 convolution layer and a sigmoid function are used to obtain the weight corresponding to each point, and the edge of the image matrix is padded with Padding=3. The output result of the spatial attention is M s As shown in formula (7), the final output of the CBA module is It is obtained through the calculation of the second residual structure, as shown in formula (8).
[0094]
[0095] In the formula, σ represents the Sigmoid activation function; 7×7 represents the size of the convolution kernel, and its application effect is better than that of the 3×3 convolution kernel.
[0096] The code implementation is as follows:
[0097]
[0098] It should be noted that the two modules of channel attention and spatial attention in the embodiment of the present application can be combined together in parallel or sequentially. The present application combines the two modules sequentially and puts the channel attention in front, which can achieve better results.
[0099] In some embodiments, the method further comprises:
[0100] The channel attention module and the spatial attention module are set in the convolutional attention module of the object detection model;
[0101] The convolutional attention module is used to generate attention regions along the channel and space order, where the input feature map is sequentially input into the channel attention module and the spatial attention module;
[0102] Generate a channel attention feature map using the channel attention module, and generate a spatial attention feature map using the spatial attention module;
[0103] Multiply the channel attention feature map and the spatial attention feature map by the input feature map element by element to generate a weighted feature map;
[0104] The weighted feature maps are fused using residual connections to generate the final feature map.
[0105] Specifically, in order to ensure real-time detection of the movement status of the vehicle behind while driving on the road, especially when the target vehicle is located in the adjacent lanes on the left and right sides of the vehicle and approaches quickly, the detection system is required to provide a rapid early warning response, which puts higher requirements on the performance of the target detection algorithm.
[0106] To this end, this application introduces a convolutional block attention (CBA) module to generate attention areas along the channel and spatial order before outputting the model's prediction results, further improving the model's feature extraction capabilities and improving network performance.
[0107] like Figure 5 As shown, Figure 5 It is a schematic diagram of the overall structure of the convolutional attention module provided in the embodiment of the present application. CBA is a lightweight attention module for feedforward convolutional neural networks, which consists of a channel attention module and a spatial attention module, and can generate attention areas along the channel and space sequence. Specifically, after the feature map is input into the CBA module, the channel attention feature map and the spatial attention feature map are calculated in sequence. The two sequential modules are used to refine the feature map passed through CBA, and residual connections are used. represents the attention multiplication operator, which enables the channel attention feature map to be mapped along the space.
[0108] Furthermore, the channel attention module of CBA focuses on the importance of features. The channel attention mechanism compresses the feature map in the spatial dimension and obtains a one-dimensional vector before operating. When compressing in the spatial dimension, not only average pooling (Average Pooling) but also maximum pooling (Max Pooling) is considered. Average pooling and maximum pooling can be used to aggregate the spatial information of feature maps, send them to a shared network, compress the spatial dimension of the input feature map, and sum and merge them element by element to generate a channel attention map. For a single image, channel attention focuses on which content on this image is important. Average pooling has feedback for every pixel on the feature map, while maximum pooling only has gradient feedback for the place with the largest response in the feature map when performing gradient backpropagation calculation.
[0109] Furthermore, CBA's spatial attention module focuses on the importance of spatial position. The spatial attention mechanism compresses the channel and performs average pooling and maximum pooling in the channel dimension. The operation of MaxPool is to extract the maximum value on the channel, and the number of extractions is height multiplied by width; the operation of AvgPool is to extract the average value on the channel, and the number of extractions is also height multiplied by width; then the previously extracted feature maps (the number of channels is 1) are merged to obtain a 2-channel feature map. Since the CBA module uses a residual structure, the input feature map of the spatial attention is the output of the channel attention. Original feature map, set the input feature map of spatial attention to First, use the maximum pooling and average pooling operations to aggregate channel information and connect the results; then, use a 7×7 convolution layer and a sigmoid function to obtain the weight corresponding to each point, and perform a padding operation of Padding=3 on the edge of the image matrix.
[0110] The improved YOLOv5 algorithm (CBA-YOLOv5) provided in the embodiment of the present application is based on YOLOv5, and its network structure is further studied and improved to make it better adapted to the blind spot detection system designed by the present application. In addition, as the core of the detection function in the data processing module, YOLOv5 can also design an automatic calibration algorithm for the camera model, relying on the faster detection speed of YOLOv5 to achieve the rapid establishment of the 2D pixel coordinate system to the 3D world coordinate system, and complete the initialization of the ranging function of the detection system. Using channels and spatial attention to extract deeper information, adaptively refine features to improve network performance, can help the network resist chaotic information during feature extraction, generate attention areas in up and down sampling, focus on useful target features, and further enhance the overall performance of the CBA-YOLOv5 network. CBA-YOLOv5 is a simple convolutional neural network with strong feature extraction capabilities. This network incorporates SAM into a variety of computer vision tasks, including image classification, target detection and instance segmentation. The present application inputs the acquired target information into the algorithm, and the algorithm automatically detects and identifies the target in real time.
[0111] It can be seen that the CBA-YOLOv5 algorithm proposed in this application is an improvement on the YOLOv5 algorithm, mainly by optimizing the network structure to achieve the real-time and accuracy requirements of target detection under actual road conditions. This application provides a technical solution for integrating blind spot monitoring expansion functions in an electronic rearview mirror system, which achieves accurate detection and timely warning of blind spots through the optimization of the target detection algorithm perceived by the camera and the blind spot monitoring algorithm, greatly improving driving safety, making the product more competitive, and enhancing the highlights and selling points of the product.
[0112] In some embodiments, blind spot monitoring is performed based on the detection results, including:
[0113] The detection results output by the target detection model are transmitted to the blind spot monitoring and control algorithm, wherein the detection results include obstacle category, location and size;
[0114] Identify obstacles in the blind spot and assess the threat level of obstacles based on obstacle type, location, size and preset control strategy;
[0115] When an obstacle is detected in the blind spot, a first-level image warning and / or a second-level audio and video warning is generated according to the threat level of the obstacle;
[0116] The warning signal is sent to the vehicle's safety control system so that the safety control system can execute corresponding safety warning control measures.
[0117] Specifically, an embodiment of the present application describes an optimization study of a target detection algorithm for the blind spot monitoring function of an electronic rearview mirror system based on YOLOv5, aiming to improve the accuracy and real-time performance of the vehicle's recognition of blind spot obstacles during driving.
[0118] Furthermore, the optimized YOLOv5 algorithm is applied to the electronic rearview mirror system to detect obstacles in the blind spot in real time, and output the detection results (including obstacle type, location, size and other information) to the blind spot monitoring and control algorithm, and formulate corresponding control strategies based on the detection results. When an obstacle is detected in the blind spot, a first-level image warning and a second-level audio and video alarm are issued in time to remind the driver to pay attention to safety. Combined with factors such as the vehicle's driving status and the driver's operating intention, reasonable driving suggestions are provided to the driver, such as adjusting the vehicle speed and changing the driving trajectory.
[0119] Overall, the target detection algorithm of the blind spot monitoring function of the electronic rearview mirror system based on YOLOv5 has been significantly improved in accuracy, real-time and stability, providing drivers with a safer and more convenient driving experience.
[0120] According to the technical solution provided in the embodiments of the present application, the technical solution of the present application has at least the following advantages:
[0121] 1. Optimize the target detection algorithm: This application adopts the target detection algorithm based on YOLOv5, which can improve the accuracy of target detection while ensuring real-time performance compared to existing image processing and computer vision technologies. In particular, it can effectively identify potential dangerous targets in actual road conditions, thereby improving the blind spot monitoring function of the electronic rearview mirror system (CMS).
[0122] 2. Reduce system complexity and cost: This application can effectively reduce the blind spots of the electronic rearview mirror system by optimizing the target detection algorithm without improving the camera design or using multiple cameras for joint observation, thereby reducing the complexity and cost of the system.
[0123] 3. Improve driving safety: This application optimizes the target detection algorithm so that the blind spot monitoring function of the electronic rearview mirror system (CMS) can provide the driver with more accurate information, thereby increasing driving safety. This is also the greatest advantage of this application over the existing technology.
[0124] 4. Increase product highlights and selling points: This application optimizes the target detection algorithm to make the blind spot monitoring function of the electronic rearview mirror system (CMS) more accurate, which will greatly enhance the competitiveness of the product and increase the product highlights and selling points.
[0125] Due to the advanced nature of this application, it can be widely used in application fields such as autonomous driving, intelligent monitoring, and intelligent transportation. First, this application can improve the safety of the autonomous driving system. Through the optimized target detection algorithm, it can detect potential dangerous targets in real time and accurately, thereby avoiding the occurrence of traffic accidents. Secondly, this application can also be applied to the field of intelligent monitoring. Through real-time analysis of monitoring videos, abnormal events can be effectively detected and monitoring efficiency can be improved. Finally, this application can also be applied to the field of intelligent transportation. Through real-time monitoring and analysis of traffic flow, traffic signal control can be optimized and traffic operation efficiency can be improved. At the same time, with the continuous development of the automobile market and the increasing demand of consumers for driving safety, this application also has broad market demand and good application prospects.
[0126] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.
[0127] Figure 6 Schematic diagram of the structure of the vehicle blind spot monitoring device based on the electronic rearview mirror provided in the embodiment of the present application. Figure 6 As shown, the vehicle blind spot monitoring device based on the electronic rearview mirror includes:
[0128] The acquisition module 601 is configured to collect environmental information around the vehicle using a camera installed on the electronic rearview mirror of the vehicle, and pre-process the environmental information to extract target information;
[0129] An input module 602 is configured to input the target information into the target detection model, wherein the target detection model includes a channel attention module and a spatial attention module, the channel attention module is used to compress the feature map in the spatial dimension and generate the channel attention feature map, and the spatial attention module is used to compress the feature map in the channel dimension and generate the spatial attention feature map;
[0130] The output module 603 is configured to detect and identify target information using a target detection model, output a detection result, and perform blind spot monitoring based on the detection result.
[0131] In some embodiments, Figure 6 The acquisition module 601 uses a camera installed on the vehicle's electronic rearview mirror to obtain environmental information around the vehicle in real time, pre-processes the obtained environmental information, and extracts preset target information from the pre-processed environmental information, wherein the camera has a preset resolution and frame rate, the pre-processing includes denoising, enhancement and format conversion, and the target information includes other vehicles, pedestrians and obstacles around the vehicle.
[0132] In some embodiments, Figure 6The input module 602 is provided with a global pooling layer and a fully connected layer in the target detection model; wherein the global pooling layer is used to perform a global pooling operation on each feature layer to generate a feature vector; the fully connected layer is used to receive the feature vector after global pooling, and determine the connection between the feature vectors to generate a weight division; the channel attention module and the spatial attention module both adopt a self-attention mechanism; wherein the self-attention mechanism is used to calculate the relationship between each feature unit, the relationship including the relationship between channels and between pixels, and introduces weights in the channel dimension and the spatial dimension so as to enhance the relationship between the feature units by using the weights.
[0133] In some embodiments, Figure 6 The input module 602 performs global average pooling and maximum pooling operations on the input feature map in the spatial dimension in the channel attention module to generate a one-dimensional feature vector; the one-dimensional feature vector is input into a shared multi-layer perceptron, which includes two fully connected layers, the first fully connected layer is used to output the compressed feature vector, and the second fully connected layer is used to output the initial channel attention feature map; the initial channel attention feature map is element-wise multiplied with the input feature map, and processed through an activation function to generate a final channel attention feature map.
[0134] In some embodiments, Figure 6 The input module 602 in the spatial attention module performs global average pooling and maximum pooling operations on the input feature map in the channel dimension to generate a two-dimensional feature map; the two-dimensional feature map is spliced to form a feature map containing multiple channels, and the spliced feature map is convolved using a convolution layer to generate a weight map; the weight map is processed by an activation function to generate an initial spatial attention feature map, and the initial spatial attention feature map is multiplied element-by-element with the input feature map to generate a final spatial attention feature map.
[0135] In some embodiments, Figure 6 The channel attention module and the spatial attention module in the input module 602 are set in the convolutional attention module of the target detection model; the convolutional attention module is used to generate attention areas along the channel and space sequence, wherein the input feature map is sequentially input into the channel attention module and the spatial attention module; the channel attention feature map is generated by using the channel attention module, and the spatial attention feature map is generated by using the spatial attention module; the channel attention feature map and the spatial attention feature map are respectively element-wise multiplied with the input feature map to generate a weighted feature map; the weighted feature map is fused by using the residual connection to generate the final feature map.
[0136] In some embodiments, Figure 6The output module 603 transmits the detection results output by the target detection model to the blind spot monitoring control algorithm, wherein the detection results include obstacle category, position and size; according to the obstacle category, position, size and preset control strategy, the obstacles in the blind spot are identified, and the threat level of the obstacles is evaluated; when an obstacle is detected in the blind spot, a first-level image warning and / or a second-level audio and video warning is generated according to the threat level corresponding to the obstacle; and the warning signal is sent to the vehicle's safety control system so that the safety control system executes corresponding safety warning control measures.
[0137] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0138] Figure 7 Schematic diagram of the structure of the electronic device 7 provided in the embodiment of the present application. Figure 7 As shown, the electronic device 7 of this embodiment includes: a processor 701, a memory 702, and a computer program 703 stored in the memory 702 and executable on the processor 701. When the processor 701 executes the computer program 703, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 701 executes the computer program 703, the functions of the modules / units in the above-mentioned device embodiments are implemented.
[0139] Exemplarily, the computer program 703 may be divided into one or more modules / units, which are stored in the memory 702 and executed by the processor 701 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 703 in the electronic device 7.
[0140] The electronic device 7 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 7 may include, but is not limited to, a processor 701 and a memory 702. Those skilled in the art will appreciate that Figure 7 It is only an example of the electronic device 7 and does not constitute a limitation of the electronic device 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.
[0141] The processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
[0142] The memory 702 may be an internal storage unit of the electronic device 7, for example, a hard disk or memory of the electronic device 7. The memory 702 may also be an external storage device of the electronic device 7, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 7. Further, the memory 702 may also include both an internal storage unit of the electronic device 7 and an external storage device. The memory 702 is used to store computer programs and other programs and data required by the electronic device. The memory 702 may also be used to temporarily store data that has been output or is to be output.
[0143] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0144] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0145] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0146] In the embodiments provided in the present application, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. Multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0147] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0148] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0149] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.
[0150] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A vehicle blind spot monitoring method based on electronic rearview mirror, It is characterized in that include: Using a camera installed on the electronic rearview mirror of the vehicle to collect environmental information around the vehicle, and preprocessing the environmental information to extract target information; Inputting the target information into a target detection model, wherein the target detection model includes a channel attention module and a spatial attention module, wherein the channel attention module is used to compress the feature map in the spatial dimension and generate the channel attention feature map, and the spatial attention module is used to compress the feature map in the channel dimension and generate the spatial attention feature map; The target information is detected and identified using the target detection model, a detection result is output, and blind spot monitoring is performed based on the detection result.
2. The method according to claim 1, It is characterized in that The method of collecting environmental information around the vehicle using a camera installed on the electronic rearview mirror of the vehicle and preprocessing the environmental information to extract target information includes: A camera installed on a vehicle's electronic rearview mirror is used to obtain environmental information around the vehicle in real time, and the obtained environmental information is preprocessed to extract preset target information from the preprocessed environmental information, wherein the camera has a preset resolution and frame rate, the preprocessing includes denoising, enhancement and format conversion, and the target information includes other vehicles, pedestrians and obstacles around the vehicle.
3. The method according to claim 1, It is characterized in that The target detection model also includes: A global pooling layer and a fully connected layer are provided in the target detection model; wherein the global pooling layer is used to perform a global pooling operation on each feature layer to generate a feature vector; and the fully connected layer is used to receive the feature vector after global pooling and determine the connection between the feature vectors to generate a weight division; The channel attention module and the spatial attention module both adopt a self-attention mechanism; wherein the self-attention mechanism is used to calculate the relationship between each feature unit, the relationship including the relationship between channels and between pixels, and introduce weights in the channel dimension and the spatial dimension so as to utilize the weights to enhance the relationship between the feature units.
4. The method according to claim 1, It is characterized in that The channel attention module is used to compress the feature map in the spatial dimension and generate a channel attention feature map, including: In the channel attention module, global average pooling and maximum pooling operations are performed on the input feature map in the spatial dimension to generate a one-dimensional feature vector; Inputting the one-dimensional feature vector into a shared multi-layer perceptron, the multi-layer perceptron comprising two fully connected layers, the first fully connected layer being used to output a compressed feature vector, and the second fully connected layer being used to output an initial channel attention feature map; The initial channel attention feature map is element-wise multiplied with the input feature map and processed through an activation function to generate a final channel attention feature map.
5. The method according to claim 1, It is characterized in that The spatial attention module is used to compress the feature map in the channel dimension and generate a spatial attention feature map, including: In the spatial attention module, global average pooling and maximum pooling operations are performed on the input feature map in the channel dimension to generate a two-dimensional feature map; The two-dimensional feature maps are spliced to form a feature map containing multiple channels, and a convolution operation is performed on the spliced feature map using a convolution layer to generate a weight map; The weight map is processed by an activation function to generate an initial spatial attention feature map, and the initial spatial attention feature map is element-wise multiplied with the input feature map to generate a final spatial attention feature map.
6. The method according to any one of claims 1 to 5, It is characterized in that The method further comprises: The channel attention module and the spatial attention module are arranged in a convolutional attention module of the target detection model; The convolutional attention module is used to generate attention regions along the channel and space order, wherein the input feature map is sequentially input into the channel attention module and the spatial attention module; Generating a channel attention feature map using the channel attention module, and generating a spatial attention feature map using the spatial attention module; Multiplying the channel attention feature map and the spatial attention feature map by the input feature map element by element to generate a weighted feature map; The weighted feature maps are fused using residual connections to generate the final feature map.
7. The method according to claim 1, It is characterized in that The performing blind spot monitoring based on the detection result includes: Transmitting the detection results output by the target detection model to the blind spot monitoring control algorithm, wherein the detection results include obstacle type, location and size; According to the obstacle type, location, size and preset control strategy, the obstacles in the blind spot are identified and the threat level of the obstacles is evaluated; When an obstacle is detected in the blind spot, a first-level image warning and / or a second-level audio and video warning is generated according to the threat level corresponding to the obstacle; The warning signal is sent to the safety control system of the vehicle so that the safety control system executes corresponding safety warning control measures.
8. A vehicle blind spot monitoring device based on an electronic rearview mirror, It is characterized in that include: A collection module is configured to collect environmental information around the vehicle using a camera installed on the electronic rearview mirror of the vehicle, and pre-process the environmental information to extract target information; An input module is configured to input the target information into a target detection model, wherein the target detection model includes a channel attention module and a spatial attention module, wherein the channel attention module is used to compress the feature map in the spatial dimension and generate the channel attention feature map, and the spatial attention module is used to compress the feature map in the channel dimension and generate the spatial attention feature map; The output module is configured to detect and identify the target information using the target detection model, output the detection result, and perform blind spot monitoring based on the detection result.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
