Automobile night vision image segmentation method and device, equipment and storage medium

Abstract

The invention relates to the technical field of image processing, and discloses an automobile night vision image segmentation method, device and equipment and a storage medium, and the method comprises the steps: obtaining an object thermal signal collected by a vehicle-mounted infrared night vision system, and generating a night vision grayscale image according to the object thermal signal; according to a residual network, obtaining a preliminary feature of the grayscale image; acquiring a feature matrix corresponding to the preliminary features according to an embedded feature integration module and a space attention module; obtaining a segmentation result according to the night vision grayscale image after binary classification of the feature matrix; based on the deep learning technology, the invention provides a multi-scale context attention image segmentation network, and the network greatly enhances the extraction capability of image features through an embedded feature integration module and a space attention module, so that the network can also have better expression capability under the scenes of insufficient illumination at night, blurred images and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method, device, equipment and storage medium for segmenting automobile night vision images. Background Art

[0002] In the existing field of vehicle autonomous driving environment perception, image semantic segmentation, as a key technology, can classify target objects such as traffic participants, road edges, and front obstacles in traffic road scene images pixel by pixel, and further identify target types based on the segmentation results, obtain obstacle information in front of the vehicle, and plan the passable range of vehicles in the passable area, etc., thereby providing relevant environmental perception results for the autonomous driving system.

[0003] In nighttime conditions, due to insufficient light, target features are not obvious. Existing image semantic segmentation methods have problems of poor segmentation effect and low accuracy, and require manual extraction of image features. They are only applicable to relatively simple image scenes and are not suitable for processing complex and high-dimensional image data. Summary of the invention

[0004] The present invention aims to provide a method, device, equipment and storage medium for segmenting automobile night vision images, aiming to solve the technical problem that the accuracy of nighttime driving image features is not high.

[0005] To achieve the above object, the present invention proposes a vehicle night vision image segmentation method, the vehicle night vision image segmentation method comprising:

[0006] Acquire a thermal signal of an object collected by a vehicle-mounted infrared night vision system, and generate a night vision grayscale image according to the thermal signal of the object;

[0007] Obtaining preliminary features of the grayscale image according to a residual network;

[0008] Acquire a feature matrix corresponding to the preliminary features according to an embedded feature integration module and a spatial attention module;

[0009] The segmentation result is obtained by performing binary classification on the night vision grayscale image according to the feature matrix.

[0010] In one embodiment, the step of obtaining a feature matrix corresponding to the preliminary features according to the embedded feature integration module and the spatial attention module further includes:

[0011] capturing multi-scale contextual feature information in the preliminary features according to the embedded feature integration module;

[0012] Acquire the target area position information in the multi-scale context feature information according to the spatial attention module;

[0013] The feature matrix is ​​acquired according to the multi-scale context feature information and the target area position information.

[0014] In one embodiment, the capturing of multi-scale context feature information in the preliminary features according to the embedded feature integration module further includes:

[0015] Acquire height information and width information of the night vision grayscale image;

[0016] Get the expansion rate of different dilated convolution branches;

[0017] Obtaining a first separation value after depth-separable convolution of the preliminary feature;

[0018] Obtaining a global average pooling layer output value according to the height information, the width information, the expansion rate and the first separation value;

[0019] The multi-scale context feature information is obtained according to the global average pooling layer output value.

[0020] In one embodiment, the obtaining the multi-scale context feature information according to the global average pooling layer output value further includes:

[0021] Obtain a first weight of the first fully connected layer and a second weight of the second fully connected layer;

[0022] Obtain a first proportional coefficient according to the global average pooling layer output value, the first weight and the second weight;

[0023] Acquire a first channel output value according to the first separation value and the first proportional coefficient;

[0024] Obtain a first parameter of the first convolutional layer and a second parameter of the second convolutional layer;

[0025] The multi-scale context feature information is acquired according to the first channel output value, the first parameter and the second parameter.

[0026] In one embodiment, the acquiring the target area position information in the multi-scale context feature information according to the spatial attention module further includes:

[0027] Obtaining a maximum pooling value after global maximum pooling and an average pooling value after global average pooling of the multi-scale context feature information;

[0028] The target area location information is obtained according to the maximum pooling value and the average pooling value.

[0029] In one embodiment, the obtaining preliminary features of the grayscale image according to the residual network further includes:

[0030] Performing dilated convolution on the grayscale image according to the residual network to obtain a dilated convolution result of the grayscale image;

[0031] The preliminary features are extracted from the dilated convolution results.

[0032] In one embodiment, the step of obtaining a segmentation result of the night vision grayscale image after binary classification according to the feature matrix further includes:

[0033] Perform binary classification according to the feature matrix to obtain the probability of a preset category;

[0034] A segmentation result corresponding to the probability is obtained according to the night vision grayscale image.

[0035] In addition, to achieve the above-mentioned purpose, the present invention also proposes a vehicle night vision image segmentation device, the device comprising:

[0036] An image generation module, used to obtain a thermal signal of an object collected by a vehicle-mounted infrared night vision system, and generate a night vision grayscale image according to the thermal signal of the object;

[0037] A feature acquisition module, used for acquiring preliminary features of the grayscale image according to a residual network;

[0038] A matrix generation module, used for obtaining a feature matrix corresponding to the preliminary features according to the embedded feature integration module and the spatial attention module;

[0039] The result acquisition module is used to obtain the segmentation result of the night vision grayscale image after binary classification according to the feature matrix.

[0040] In addition, to achieve the above-mentioned purpose, the present invention also proposes a car night vision image segmentation device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the car night vision image segmentation method as described above.

[0041] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the above-mentioned automobile night vision image segmentation method are implemented.

[0042] The present invention proposes a method for segmenting automobile night vision images, which comprises: obtaining a thermal signal of an object collected by an on-board infrared night vision system, and generating a night vision grayscale image according to the thermal signal of the object; obtaining preliminary features of the grayscale image according to a residual network; obtaining a feature matrix corresponding to the preliminary features according to an embedded feature integration module and a spatial attention module; and obtaining a segmentation result of the night vision grayscale image after binary classification according to the feature matrix. Based on deep learning technology, the present invention proposes a multi-scale contextual attention image segmentation network, which greatly enhances the ability to extract image features through an embedded feature integration module and a spatial attention module, and thus has better performance in scenes such as insufficient light at night and blurred images. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of a flow chart provided for the first embodiment of the vehicle night vision image segmentation method of the present invention;

[0044] Figure 2 A first flow chart of a second embodiment of the vehicle night vision image segmentation method of the present invention;

[0045] Figure 3 A second flow chart of the second embodiment of the vehicle night vision image segmentation method of the present invention;

[0046] Figure 4 A third flow chart of the second embodiment of the vehicle night vision image segmentation method of the present invention;

[0047] Figure 5 A fourth flow chart of the second embodiment of the vehicle night vision image segmentation method of the present invention;

[0048] Figure 6 A first flow chart of a third embodiment of the vehicle night vision image segmentation method of the present invention;

[0049] Figure 7 A second flow chart of the third embodiment of the vehicle night vision image segmentation method of the present invention;

[0050] Figure 8 This is a schematic diagram of the module structure of the vehicle night vision image segmentation device according to an embodiment of the present invention;

[0051] Fig. 9 The present invention is a schematic diagram of the structure of a vehicle night vision image segmentation device in the hardware operating environment of the vehicle night vision image segmentation method according to the embodiment of the present invention.

[0052] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present invention and are not used to limit the present invention.

[0054] In order to better understand the technical solution of the present invention, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0055] The main solution of the embodiment of the present invention is: obtaining the thermal signal of the object collected by the vehicle-mounted infrared night vision system, and generating a night vision grayscale image based on the thermal signal of the object; obtaining the preliminary features of the grayscale image based on the residual network; obtaining the feature matrix corresponding to the preliminary features based on the embedded feature integration module and the spatial attention module; and obtaining the segmentation result of the night vision grayscale image after binary classification based on the feature matrix.

[0056] In nighttime conditions, due to insufficient light, target features are not obvious. Existing image semantic segmentation methods have problems of poor segmentation effect and low accuracy, and require manual extraction of image features. They are only applicable to relatively simple image scenes and are not suitable for processing complex and high-dimensional image data.

[0057] The present invention provides a solution. Based on deep learning technology, a multi-scale contextual attention image segmentation network is proposed. The network greatly enhances the ability to extract image features through an embedded feature integration module and a spatial attention module, and thus has better performance even in scenes such as insufficient lighting at night and blurred images.

[0058] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a car night vision image segmentation device that can realize the above functions. The following takes the car night vision image segmentation device as an example to illustrate this embodiment and the following embodiments.

[0059] Based on this, the embodiment of the present invention provides a method for segmenting a night vision image of an automobile, referring to Figure 1 , which is a flow chart of the first embodiment of the vehicle night vision image segmentation method of the present invention.

[0060] In this embodiment, the vehicle night vision image segmentation method includes steps S10 to S40:

[0061] Step S10: Acquire the thermal signal of the object collected by the vehicle-mounted infrared night vision system, and generate a night vision grayscale image according to the thermal signal of the object.

[0062] It should be noted that the vehicle-mounted infrared night vision system captures the infrared radiation emitted by objects through its infrared sensors. These radiations are generated because the temperature of the objects is higher than absolute zero. The infrared detectors in the infrared sensors convert the received infrared radiation into electrical signals, which are then converted into digital signals for further processing and analysis.

[0063] It should be understood that the processing includes steps such as signal amplification, noise filtering, and non-uniformity correction to improve image quality; the processed digital signal is converted into a grayscale image through grayscale transformation, and the grayscale value represents the temperature information of the object surface; in this process, objects of different temperatures appear as different grayscale levels in the image, thus forming a thermal image; the generated night vision grayscale image is able to highlight objects with higher temperatures, such as pedestrians, animals or other vehicles, which can be clearly seen even in a completely dark environment; this image is crucial to night driving safety because it helps drivers identify potential obstacles and hazards in low visibility conditions.

[0064] Step S20: Obtain preliminary features of the grayscale image according to the residual network.

[0065] It should be noted that the residual network (ResNet) is a deep learning model that introduces the residual learning method to enable the network to better adapt to complex data distribution during training, especially in tasks such as image recognition and target detection. The core idea of ​​the residual network is to learn features by introducing residual blocks. Each residual block contains two parts: cross-layer connection and identity mapping. This allows information to be directly passed to the subsequent layers, thereby improving the accuracy and generalization ability of the network.

[0066] It should be understood that when using the residual network for feature extraction, the residual block can extract some high-level features, and the cross-layer connection allows these features to be directly passed to the subsequent layers; this structural design not only increases the depth of the network, but also effectively avoids the gradient vanishing and gradient exploding problems, thereby improving the performance of the network.

[0067] It should be noted that for the preliminary feature extraction of grayscale images, the residual network can capture the details and texture information of the image through its deep structure. This information is very important for subsequent image processing tasks, such as image segmentation and target detection. Through training, the residual network can learn the intrinsic representation of the image, which can reflect the key features of the image and provide strong support for further image analysis.

[0068] Step S30: Obtain a feature matrix corresponding to the preliminary features according to the embedded feature integration module and the spatial attention module.

[0069] It should be understood that the embedded feature integration module combines multi-scale dilated convolutions and channel attention mechanisms, which plays a key role in enhancing the network receptive field, capturing global contextual information and correlation features between channels; considering the complexity of night vision images of cars and the irregularity of pedestrians and other obstacles, it is important to increase the receptive field of feature points in the center of the network while ensuring the capture of detail information; some of the most advanced deep learning models have proved that dilated convolutions can be used as an ideal replacement for pooling layers; the embedded feature integration module combines dilated convolutions with depthwise separable convolutions, and uses five branches with dilation rates of 1, 2, 4, 8, and 16 in parallel to solve the grid effect in dilated convolutions and the problem that remote information may be irrelevant.

[0070] It should be noted that the spatial attention module aims to generate a spatial attention map using the spatial relationship of the feature map, thereby improving the network's ability to capture the location information of the target area. Since pedestrians and obstacles in car night vision images are often blurry, it is particularly important to effectively identify the target area and enhance the features of the target area. The spatial attention module adopts the method of applying maximum pooling and average pooling to the input feature map along the channel dimension, so that applying pooling operations along the channel dimension can effectively highlight the feature area.

[0071] Step S40: Obtaining a segmentation result of the night vision grayscale image after binary classification according to the feature matrix.

[0072] It should be understood that when performing binary classification and segmentation of night vision grayscale images, it is necessary to extract features that can represent the image content from the night vision images. These features may include texture, shape, color, etc.; the constructed feature matrix will contain the quantitative representation of these features for subsequent classification and segmentation tasks; in the binary classification stage, the classifier will learn to distinguish different image categories based on the information in the feature matrix; this step usually involves machine learning algorithms, such as support vector machines (SVM), random forests or neural networks, which can learn classification rules from training data.

[0073] It should be noted that the segmentation stage is based on binary classification, which uses the classification results to determine the ownership of each area in the image. In night vision images, the segmentation algorithm needs to be able to handle noise and contrast problems under low light conditions; Common segmentation techniques include threshold segmentation, region growing, clustering algorithms (such as K-means) and graph-based methods; The evaluation of segmentation results can use indicators such as pixel accuracy, intersection over union (IoU), and Dice coefficient to measure the accuracy of segmentation. In addition, due to the particularity of night vision images, specific pre-processing and post-processing steps may be required, such as denoising, contrast enhancement, and morphological operations, to improve the robustness and accuracy of segmentation.

[0074] The present embodiment proposes a method for automobile night vision image segmentation, which includes: obtaining a thermal signal of an object collected by an on-board infrared night vision system, and generating a night vision grayscale image according to the thermal signal of the object; obtaining preliminary features of the grayscale image according to a residual network; obtaining a feature matrix corresponding to the preliminary features according to an embedded feature integration module and a spatial attention module; and obtaining a segmentation result of the night vision grayscale image after binary classification according to the feature matrix. Based on deep learning technology, the present invention proposes a multi-scale contextual attention image segmentation network, which greatly enhances the ability to extract image features through an embedded feature integration module and a spatial attention module, and thus has better performance in scenes such as insufficient light at night and blurred images.

[0075] Based on the first embodiment of the present invention, in the second embodiment of the present invention, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be described in detail later. Figure 2 , Figure 2 The first flow chart of the second embodiment of the vehicle night vision image segmentation method of the present invention is provided. In step S30, the vehicle night vision image segmentation method further includes:

[0076] Step S3100: Capturing multi-scale context feature information in the preliminary features according to the embedded feature integration module.

[0077] It should be understood that embedded systems usually have limitations in computing resources and power consumption, so the integrated modules need to be optimized to adapt to these limitations while maintaining sensitivity and accuracy to multi-scale features; usually feature extraction, multi-scale analysis, context information fusion (involving pyramid construction), real-time processing and resource optimization are performed. Through these steps, the integrated modules can effectively capture and process multi-scale context feature information in embedded systems to support various computer vision tasks such as target detection, image segmentation and scene understanding.

[0078] Step S3200: Obtain target area position information in the multi-scale context feature information according to the spatial attention module.

[0079] It should be noted that the spatial attention module can help the model focus on the key areas in the image, thereby improving the performance and accuracy of the model; in the process of obtaining multi-scale contextual feature information, the spatial attention module calculates the importance of each position in the feature map and generates a spatial weight map, which is used to adjust the model's attention to different areas.

[0080] It should be understood that in multi-scale feature extraction, the spatial attention module is able to capture target features at different scales, which is crucial for understanding different levels of abstraction of images; for example, in the improvement of YOLOv5, the MSDA (Multi-Scale Dilated Attention) module is introduced, which obtains the corresponding query, key, and value of the feature map through linear projection, and performs multi-scale SWDA (self-attention) with different dilation rates in different heads; in this way, MSDA can effectively aggregate semantic information of different scales within the attended receptive field, and effectively reduce the redundancy of the self-attention mechanism without the need for complex operations and additional computational costs.

[0081] Step S3300: Acquire the feature matrix according to the multi-scale context feature information and the target area position information.

[0082] It should be noted that in practical applications, this feature matrix can be used to train classifiers or regressors to achieve accurate target recognition and positioning; the process of constructing the feature matrix involves feature selection, extraction and alignment to ensure that the features of the target area are consistent at different scales and can accurately reflect the characteristics of the target; when expanding the feature matrix, it is necessary to consider how to effectively integrate multi-scale information and how to adjust the spatial layout of features according to the location information of the target area. It is necessary to design a complex network architecture and training strategy to ensure that the model can learn powerful feature representations and achieve high performance in various visual tasks.

[0083] Please refer to Figure 3 , Figure 3 The second flow chart of the second embodiment of the vehicle night vision image segmentation method of the present invention is provided, and step S3100 further includes:

[0084] Step S3101: Acquire height information and width information of the night vision grayscale image;

[0085] Step S3102: Obtain expansion rates of different dilated convolution branches;

[0086] Step S3103: Obtaining a first separation value after performing depth-separable convolution on the preliminary feature;

[0087] Step S3104: obtaining a global average pooling layer output value according to the height information, the width information, the expansion rate and the first separation value;

[0088] Step S3105: Obtain the multi-scale context feature information according to the output value of the global average pooling layer.

[0089] It should be noted that the specific calculation process is as follows:

[0090]

[0091] Where X input It is the input of the embedded feature integration module. H and W represent the height information and width information of the night vision grayscale image respectively. DepSep represents the output of the depthwise separable convolution. Rate is the dilation rate of the dilated convolution. After the input image passes through the depthwise separable convolution with the dilation rate, global average pooling is performed. z represents the output value of the global average pooling layer.

[0092] Please refer to Figure 4 , Figure 4 The third flow chart of the second embodiment of the vehicle night vision image segmentation method of the present invention is provided, and step S3105 further includes:

[0093] Step S3106: Obtain a first weight of the first fully connected layer and a second weight of the second fully connected layer;

[0094] Step S3107: Obtain a first proportional coefficient according to the global average pooling layer output value, the first weight and the second weight;

[0095] Step S3108: obtaining a first channel output value according to the first separation value and the first proportional coefficient;

[0096] Step S3109: Obtain a first parameter of the first convolutional layer and a second parameter of the second convolutional layer;

[0097] Step S3110: Acquire the multi-scale context feature information according to the first channel output value, the first parameter and the second parameter.

[0098] It should be understood that W1 and W2 represent the weights of the two fully connected layers, σ and δ represent the Swish activation function and the sigmoid function respectively, and s is the ratio coefficient between the channels learned by the two fully connected layers after back propagation; similarly, s i represents the proportional coefficient of the i-th channel, is the output of the ith channel, X is the output after all channels are fused; W3 and W4 represent the parameters of the two convolutional layers, and U is the output of the entire embedded feature integration module, as shown in equations (2), (3), (4) and (5):

[0099] s=σ(Dropout(W2·Dropout(δ(W1·z)))) (2)

[0100]

[0101] U=Dropout(W4·(DepSep+W3·X)) (5)

[0102] Please refer to Figure 5 , Figure 5 The fourth flow chart of the second embodiment of the vehicle night vision image segmentation method of the present invention is provided, and step S3200 further includes:

[0103] Step S3201: obtaining a maximum pooling value after global maximum pooling and an average pooling value after global average pooling of the multi-scale context feature information;

[0104] Step S3202: Obtain the target area location information according to the maximum pooling value and the average pooling value.

[0105] The spatial attention module uses the method of applying the maximum pooling (MaxPool) and average pooling (AvgPool) to the input feature map along the channel dimension. In this way, applying the pooling operation along the channel dimension can effectively highlight the feature area. The specific calculation process is shown in formula (6):

[0106] SA(x)=σ(f 7×7 ([MaxPool(x);AvgPool(x)]))=σ(f 7×7 ([F max ; F avg ])) (6)

[0107] Where x represents the input feature map, SA(x) represents the output of the spatial attention module, and F max ∈R 1×H×W and F avg ∈R 1 ×H×W are the outputs of the maximum pooling layer and the average pooling layer, σ represents the sigmoid function, and f 7×7 Represents a 7×7 convolutional layer.

[0108] In this embodiment, the embedded feature integration module and the spatial attention module are combined, and the preliminary features are used as the input of the void spatial convolution pooling module. The embedded feature integration module is composed of multiple parallel convolution branches, each branch has a different expansion rate, and the expansion rates are set to 1, 2, 4, 8, and 16 respectively. Finally, a global average pooling operation is also included. The spatial attention module is connected after the void convolution with the expansion rate, and the dependency between channels is flexibly captured, thereby enhancing the feature representation ability of the model; then splicing is performed along the channel dimension, and processed through the convolution layer to generate the spatial attention weight; finally, the sigmoid function is connected and weighted processing is performed, and then the output features of the channel attention module and the spatial attention module are multiplied element by element to obtain the final attention enhancement feature as the subsequent network input; the network greatly enhances the ability to extract image features through the embedded feature integration module and the spatial attention module, and thus has a good performance in scenes such as insufficient light at night and blurred images, thereby improving the accuracy of night driving image features.

[0109] Based on the first embodiment and / or the second embodiment of the present invention, in the third embodiment of the present invention, the same or similar contents as those in the first and second embodiments can be referred to the above description, and will not be described in detail later. Figure 6 , Figure 6 The first flow chart of the third embodiment of the vehicle night vision image segmentation method of the present invention is provided. In step S20, the vehicle night vision image segmentation method further includes:

[0110] Step S2100: performing dilated convolution on the grayscale image according to the residual network to obtain a dilated convolution result of the grayscale image.

[0111] It should be noted that dilated convolution, also known as hole convolution, is a special convolution operation that increases the receptive field, that is, the size of the area covered by the convolution kernel, by introducing holes in the standard convolution kernel.

[0112] It should be understood that by combining the residual network with the dilated convolution, a dilated residual network (DRN) can be constructed. This network structure can expand the network's receptive field while maintaining feature resolution, thereby improving the model's ability to understand image details. In tasks such as image segmentation, the dilated residual network can better handle small targets and complex scenes, providing more accurate segmentation results.

[0113] Step S2200: extracting the preliminary features from the dilated convolution results.

[0114] Please refer to Figure 7 , Figure 7The first flow chart of the third embodiment of the vehicle night vision image segmentation method of the present invention is provided, wherein the segmentation result of the night vision grayscale image obtained after binary classification according to the feature matrix further includes:

[0115] Step S4100: performing binary classification according to the feature matrix to obtain the probability of a preset category;

[0116] Step S4200: Obtain a segmentation result corresponding to the probability according to the night vision grayscale image.

[0117] It should be noted that the feature matrix is ​​a mathematical representation of the data set, in which each row represents a sample and each column represents a feature. In the binary classification problem, the feature matrix is ​​usually used to train classification models, such as logistic regression, support vector machine (SVM) or neural network. After the classification model is trained, the model's predicted probability of these samples can be obtained by inputting test samples in the feature matrix. These probabilities reflect the model's confidence that the sample belongs to the positive class (usually represented as 1) or the negative class (usually represented as 0).

[0118] In this embodiment, by performing dilated convolution on the grayscale image according to the residual network, the dilated convolution result of the grayscale image is obtained, the preliminary features are extracted from the dilated convolution result, and the network's receptive field is expanded, thereby improving the model's ability to understand image details. In tasks such as image segmentation, the dilated residual network can better handle small targets and complex scenes and provide more accurate segmentation results. Binary classification is performed according to the feature matrix to obtain the probability of a preset category, and then the corresponding segmentation result in the probability is obtained according to the night vision grayscale image, thereby improving the accuracy and reliability of the model.

[0119] The present invention also provides a vehicle night vision image segmentation device, please refer to Figure 8 , the automobile night vision image segmentation device comprises:

[0120] The image generation module 10 is used to obtain the thermal signal of the object collected by the vehicle-mounted infrared night vision system, and generate a night vision grayscale image according to the thermal signal of the object;

[0121] A feature acquisition module 20, used for acquiring preliminary features of the grayscale image according to a residual network;

[0122] A matrix generation module 30, used for obtaining a feature matrix corresponding to the preliminary features according to the embedded feature integration module and the spatial attention module;

[0123] The result acquisition module 40 is used to obtain the segmentation result of the night vision grayscale image after binary classification according to the feature matrix.

[0124] Optionally, the matrix generation module 30 is also used to capture multi-scale contextual feature information in the preliminary features according to the embedded feature integration module; obtain target area position information in the multi-scale contextual feature information according to the spatial attention module; and obtain the feature matrix according to the multi-scale contextual feature information and the target area position information.

[0125] Optionally, the matrix generation module 30 is also used to obtain the height information and width information of the night vision grayscale image; obtain the expansion rate of different hole convolution branches; obtain the first separation value after the preliminary feature is subjected to depth-separable convolution; obtain the global average pooling layer output value according to the height information, the width information, the expansion rate and the first separation value; obtain the multi-scale context feature information according to the global average pooling layer output value.

[0126] Optionally, the matrix generation module 30 is also used to obtain the first weight of the first fully connected layer and the second weight of the second fully connected layer; obtain the first proportional coefficient according to the global average pooling layer output value, the first weight and the second weight; obtain the first channel output value according to the first separation value and the first proportional coefficient; obtain the first parameter of the first convolutional layer and the second parameter of the second convolutional layer; obtain the multi-scale context feature information according to the first channel output value, the first parameter and the second parameter.

[0127] Optionally, the matrix generation module 30 is also used to obtain the maximum pooling value after global maximum pooling of the multi-scale context feature information, and the average pooling value after global average pooling; and obtain the target area position information according to the maximum pooling value and the average pooling value.

[0128] Optionally, the feature acquisition module 20 is further used to perform dilated convolution on the grayscale image according to the residual network to obtain a dilated convolution result of the grayscale image; and extract the preliminary features from the dilated convolution result.

[0129] Optionally, the result acquisition module 40 is further used to obtain the probability of a preset category by performing binary classification according to the feature matrix; and obtain the segmentation result corresponding to the probability according to the night vision grayscale image.

[0130] The automobile night vision image segmentation device provided by the present invention adopts the automobile night vision image segmentation method in the above embodiment, which can solve the technical problem that the accuracy of night driving image features is not high. Compared with the prior art, the automobile night vision image segmentation device provided by the present invention has the same beneficial effects as the automobile night vision image segmentation method provided by the above embodiment, and other technical features in the automobile night vision image segmentation device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0131] The present invention provides an automobile night vision image segmentation device, which includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the automobile night vision image segmentation method in the above-mentioned embodiment one.

[0132] Reference below Fig. 9 , which shows a schematic diagram of the structure of a vehicle night vision image segmentation device suitable for implementing an embodiment of the present invention. The vehicle night vision image segmentation device in the embodiment of the present invention may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The automobile night vision image segmentation device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0133] like Fig. 9As shown, the automobile night vision image segmentation device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the automobile night vision image segmentation device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an automobile night vision image segmentation device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0134] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present invention are executed.

[0135] The automobile night vision image segmentation device provided by the present invention adopts the automobile night vision image segmentation method in the above embodiment, which can solve the technical problem that the accuracy of night driving image features is not high. Compared with the prior art, the automobile night vision image segmentation device provided by the present invention has the same beneficial effects as the automobile night vision image segmentation method provided by the above embodiment, and other technical features in the automobile night vision image segmentation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0136] It should be understood that the various parts disclosed in the present invention can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0137] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0138] The present invention provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the automobile night vision image segmentation method in the above-mentioned embodiment.

[0139] The computer-readable storage medium provided by the present invention may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0140] The computer-readable storage medium may be included in the automobile night vision image segmentation device; or may exist independently without being assembled into the automobile night vision image segmentation device.

[0141] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0142] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0143] The modules involved in the embodiments of the present invention may be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0144] The readable storage medium provided by the present invention is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned automobile night vision image segmentation method, and can solve the technical problem that the accuracy of night driving image features is not high. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present invention are the same as the beneficial effects of the automobile night vision image segmentation method provided by the above-mentioned embodiment, and will not be repeated here.

[0145] The above descriptions are only some embodiments of the present invention, and are not intended to limit the patent scope of the present invention. All equivalent structural changes made using the contents of the present invention's specification and drawings under the technical concept of the present invention, or directly / indirectly applied in other related technical fields, are included in the patent protection scope of the present invention.

Claims

1. A method for segmenting automobile night vision images, characterized in that: The automobile night vision image segmentation method comprises: Acquire a thermal signal of an object collected by a vehicle-mounted infrared night vision system, and generate a night vision grayscale image according to the thermal signal of the object; Obtaining preliminary features of the grayscale image according to a residual network; Acquire a feature matrix corresponding to the preliminary features according to an embedded feature integration module and a spatial attention module; The segmentation result is obtained by performing binary classification on the night vision grayscale image according to the feature matrix.

2. The vehicle night vision image segmentation method according to claim 1, characterized in that: The step of obtaining a feature matrix corresponding to the preliminary features according to the embedded feature integration module and the spatial attention module also includes: capturing multi-scale contextual feature information in the preliminary features according to the embedded feature integration module; Acquire the target area position information in the multi-scale context feature information according to the spatial attention module; The feature matrix is ​​acquired according to the multi-scale context feature information and the target area position information.

3. The vehicle night vision image segmentation method according to claim 2, characterized in that: The capturing of multi-scale context feature information in the preliminary features according to the embedded feature integration module also includes: Acquire height information and width information of the night vision grayscale image; Get the expansion rate of different dilated convolution branches; Obtaining a first separation value after depth-separable convolution of the preliminary feature; Obtaining a global average pooling layer output value according to the height information, the width information, the expansion rate and the first separation value; The multi-scale context feature information is obtained according to the global average pooling layer output value.

4. The vehicle night vision image segmentation method according to claim 3, characterized in that: The acquiring the multi-scale context feature information according to the output value of the global average pooling layer also includes: Obtain a first weight of the first fully connected layer and a second weight of the second fully connected layer; Obtain a first proportional coefficient according to the global average pooling layer output value, the first weight and the second weight; Acquire a first channel output value according to the first separation value and the first proportional coefficient; Obtain a first parameter of the first convolutional layer and a second parameter of the second convolutional layer; The multi-scale context feature information is acquired according to the first channel output value, the first parameter and the second parameter.

5. The vehicle night vision image segmentation method according to claim 2, characterized in that: The step of obtaining the target area position information in the multi-scale context feature information according to the spatial attention module also includes: Obtaining a maximum pooling value after global maximum pooling and an average pooling value after global average pooling of the multi-scale context feature information; The target area location information is obtained according to the maximum pooling value and the average pooling value.

6. The vehicle night vision image segmentation method according to claim 1, characterized in that: The step of obtaining preliminary features of the grayscale image according to the residual network further includes: Performing dilated convolution on the grayscale image according to the residual network to obtain a dilated convolution result of the grayscale image; The preliminary features are extracted from the dilated convolution results.

7. The vehicle night vision image segmentation method according to claim 1, characterized in that: The step of obtaining a segmentation result of the night vision grayscale image after binary classification according to the feature matrix also includes: Perform binary classification according to the feature matrix to obtain the probability of a preset category; A segmentation result corresponding to the probability is obtained according to the night vision grayscale image.

8. A vehicle night vision image segmentation device, characterized in that: The device comprises: An image generation module, used to obtain a thermal signal of an object collected by a vehicle-mounted infrared night vision system, and generate a night vision grayscale image according to the thermal signal of the object; A feature acquisition module, used for acquiring preliminary features of the grayscale image according to a residual network; A matrix generation module, used for obtaining a feature matrix corresponding to the preliminary features according to the embedded feature integration module and the spatial attention module; The result acquisition module is used to obtain the segmentation result of the night vision grayscale image after binary classification according to the feature matrix.

9. A vehicle night vision image segmentation device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the automobile night vision image segmentation method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the automobile night vision image segmentation method according to any one of claims 1 to 7 are implemented.

Images