Image fusion segmentation method and device, equipment and storage medium

Abstract

The invention provides an image fusion segmentation method and device, equipment and a storage medium, and the method comprises the steps: obtaining an initial image, segmenting the initial image into a plurality of initial image channels, and determining a weight factor through calculating the cosine similarity of each pixel position, so as to carry out the feature fusion processing of each initial image channel, thereby achieving the image fusion segmentation. Obtaining a fused image channel and a comprehensive image channel based on the fusion process, taking the fused image channel as input again to carry out the next round of feature fusion processing until all the comprehensive image channels are obtained, and obtaining a fusion segmentation result of the initial image based on the integrated comprehensive image channel; according to the method, the accuracy and efficiency of image segmentation are effectively improved, a high-quality image segmentation effect is realized, and powerful data support is provided for environment information identification of the vehicle.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image fusion and segmentation method, device, equipment and storage medium. Background Art

[0002] Image segmentation is a key technology in the field of computer vision and is essential for application scenarios such as autonomous driving, smart medical care, and image search. Image segmentation technology is particularly important in military and special operations environments. For example, in a complex battlefield environment, accurately identifying enemy and friendly targets and distinguishing different types of obstacles and terrain features are crucial to improving operational efficiency and ensuring personnel safety. In addition, for reconnaissance and surveillance missions at night or in severe weather conditions, high-precision image segmentation can greatly enhance situational awareness.

[0003] Traditional image segmentation methods mainly rely on low-level features (such as color, texture) and hand-designed feature extraction algorithms. These methods show limitations in dealing with the complexity of image details and object differences (such as scale and posture changes), and it is difficult to achieve the required accuracy and generalization ability. Summary of the invention

[0004] In view of the above-mentioned shortcomings of the prior art, the present application provides an image fusion segmentation method, device, equipment and storage medium to solve the above-mentioned technical problems.

[0005] The present application provides an image fusion and segmentation method, which includes: obtaining an initial image, and segmenting the initial image into multiple initial image channels; performing feature fusion processing on the initial image channels, wherein the feature fusion processing includes calculating the cosine similarity of each initial image channel at each pixel position, and using the cosine similarity as a weight factor of the corresponding image channel; fusing each initial image channel based on the weight factor to obtain each fused image channel and a comprehensive image channel, wherein the comprehensive image channel is a result of the fusion of all initial image channels; using each fused image channel as an input value for the next feature fusion processing, repeating the above-mentioned feature fusion processing steps, and obtaining a comprehensive image channel corresponding to each feature fusion processing, so as to obtain a fused segmentation result of the initial image based on all comprehensive image channels.

[0006] In one embodiment of the present application, a fused segmentation result of the initial image is obtained based on all integrated image channels, including: according to the feature fusion order, selecting the integrated image channel after the last fusion as the current fused image channel, and performing image restoration processing on the current fused image, the image restoration processing including upsampling the current fused image channel to obtain the current upsampling result, fusing the current upsampling result with the previous integrated image channel to obtain the current fusion result; using the current fusion result as the input for the next image restoration, and repeating the image restoration processing until the fusion and upsampling processing are completed for all fused integrated image channels to obtain the fused segmentation result of the initial image.

[0007] In one embodiment of the present application, image restoration processing is performed on the current fused image, and the process also includes: inputting the current fused image channel into one or more preprocessing layers to enhance the image features of the current fused image channel, the preprocessing layer including a pooling layer, a convolution layer, a batch normalization layer or an activation function layer; performing dimensionality reduction processing on the current fused image channel to adjust the image dimension of the current fused image channel; extracting the image features of the current fused image channel to enhance the feature expression capability of the current fused image channel.

[0008] In one embodiment of the present application, the cosine similarity of each initial image channel at each pixel position is calculated, including: taking any initial image channel as a current image channel; calculating the cosine similarity between the current image channel and other initial image channels; determining the cosine similarity of the current image channel according to the cosine similarity between the current image channel and other initial image channels; and traversing all initial image channels to obtain the cosine similarity corresponding to each initial image channel.

[0009] In one embodiment of the present application, determining the cosine similarity of the current image channel according to the cosine similarity between the current image channel and other initial image channels includes: dividing the initial image into four initial image channels, including an original red channel, an original green channel, an original blue channel, and an original additional data channel; calculating the cosine similarity between the original red channel and the original green channel, recorded as a first red cosine similarity; calculating the cosine similarity between the original red channel and the original blue channel, recorded as a second red cosine similarity; calculating the cosine similarity between the original red channel and the original additional data channel, recorded as a third red cosine similarity; determining the average of the first red cosine similarity, the second red cosine similarity, and the third red cosine similarity as the cosine similarity of the original red channel; and traversing the remaining initial image channels to obtain the cosine similarities corresponding to each initial image channel.

[0010] In one embodiment of the present application, after obtaining the fused segmentation result of the initial image, it includes: inputting the segmentation result of the initial image into a target detection module, the target detection module is used to identify a specific target in the segmentation result and obtain the position information of the specific target, the specific target includes at least one of a pedestrian, an obstacle, and a lane line; sending the position information to a driving assistance system to trigger the driving assistance system to take corresponding driving assistance measures according to the received position information to ensure driving safety, the driving assistance measures include issuing a warning signal, starting an automatic emergency braking system, and adjusting at least one of the vehicle's driving direction.

[0011] In one embodiment of the present application, after obtaining the fused segmentation result of the initial image, it also includes: fusing the fused segmentation result with the initial image to generate a fused image containing a target area, wherein the target area includes a key object with annotation information; displaying the fused image on a display in the cockpit of the vehicle so that the driver can determine the relative position of the key object and the vehicle; combining the current position data of the vehicle and the relative position of the key object and the vehicle, predicting the collision risk between the vehicle and the key object, and generating a prediction result; based on the prediction result, triggering the driving assistance system to perform corresponding avoidance actions or issue an alarm to the driver.

[0012] The present application provides an image fusion and segmentation device, which includes: an image preprocessing module, which is used to obtain an initial image and segment the initial image into multiple initial image channels; a feature fusion module, which is used to perform feature fusion processing on the initial image channels, wherein the feature fusion processing includes calculating the cosine similarity of each initial image channel at each pixel position, and using the cosine similarity as a weight factor of the corresponding image channel; fusing each initial image channel based on the weight factor to obtain each fused image channel and a comprehensive image channel, wherein the comprehensive image channel is a result of the fusion of all initial image channels; and a loop processing module, which is used to use each fused image channel as an input value for the next feature fusion processing, repeat the above-mentioned feature fusion processing steps, obtain the comprehensive image channel corresponding to each feature fusion processing, and obtain the fusion segmentation result of the initial image based on all the comprehensive image channels.

[0013] The present application provides an electronic device, comprising a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute a computer program stored in the memory to implement the image fusion segmentation method as described above.

[0014] The present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is used to enable a computer to execute the image fusion and segmentation method as described above.

[0015] Beneficial effects of the present application: The image fusion segmentation method in the present application obtains an initial image and divides it into multiple initial image channels; then, the weight factor is determined by calculating the cosine similarity at each pixel position to perform feature fusion processing on each initial image channel; the fusion process based on the weight factor will produce a series of fused image channels and a comprehensive image channel, which is the result of the fusion of all initial image channels; then, the fused image channels are used as input again for the next round of feature fusion processing, and this process is repeated until all comprehensive image channels are obtained; finally, these comprehensive image channels are integrated to produce a fused segmentation result of the initial image. This method not only effectively improves the accuracy and efficiency of image segmentation, but also significantly enhances the generalization ability of the model, and can stably identify targets in complex and changing environments to achieve high-quality image segmentation effects.

[0016] In addition, the image segmentation proposed by the above method is particularly important for the vehicle's environmental information recognition. It can provide strong data support for the vehicle, ensuring that the surrounding objects and environmental features can be accurately identified under various conditions, thereby improving driving safety.

[0017] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0019] Figure 1 is a schematic diagram of an implementation environment of an image fusion segmentation method shown in an exemplary embodiment of the present application;

[0020] Figure 2 is a flowchart of an image fusion segmentation method shown in an exemplary embodiment of the present application;

[0021] Figure 3 is a schematic diagram of a feature fusion processing flow chart shown in an exemplary embodiment of the present application;

[0022] Figure 4 It is a schematic diagram of the complete steps of the image fusion segmentation method shown in an exemplary embodiment of the present application;

[0023] Figure 5is a block diagram of an image fusion and segmentation device shown in an exemplary embodiment of the present application;

[0024] Figure 6 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0025] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0026] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0027] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.

[0028] First of all, it should be noted that upsampling refers to the process of increasing the resolution of an image or feature map. In computer vision and image processing, upsampling is usually used to restore image details or increase image resolution in order to better capture detailed features in the image.

[0029] Downsampling refers to the process of reducing the resolution of an image or feature map. In computer vision and image processing, downsampling is mainly used to reduce the amount of data, speed up calculations, or reduce the demand for computing resources.

[0030] Cosine Similarity Fusion is a feature fusion method that evaluates the similarity between different features by calculating the cosine similarity between them. The fused similarity obtained can be used to compare the similarities between different image channels in image processing and assign weights accordingly to perform feature fusion.

[0031] Effective Context Extraction refers to extracting useful information from images or feature maps that helps understand the image content and contextual relationships. This usually involves using specific techniques or algorithms to identify and capture features such as patterns, textures, shapes, and their relationships in images, which helps improve the accuracy and robustness of image analysis.

[0032] Figure 1 It is a schematic diagram of an implementation environment of an image fusion segmentation method shown in an exemplary embodiment of the present application.

[0033] like Figure 1 As shown, the implementation environment of the image fusion segmentation method includes an image acquisition module 101 and a computer module 102. Among them, the image acquisition module 101 is responsible for collecting image data, which includes but is not limited to visible light images, infrared images, depth images and other sensor data. The image acquisition device can be various cameras installed on the vehicle (such as visible light cameras, thermal imaging cameras, depth cameras, etc.), or other types of sensors, such as laser radar (LiDAR), millimeter wave radar, etc. This application does not impose any restrictions on the specific implementation form of the image acquisition module. The computer module 102 is used to process the original image data collected by the image acquisition module 101, and perform image segmentation and fusion processing. The device determines the weight factor by calculating the cosine similarity at each pixel position, and performs feature fusion processing on the image channel based on these weight factors to generate high-quality segmentation results. The computer module 102 can also use a deep learning algorithm to further process the image data to improve the accuracy of segmentation. The computer module 102 can be a high-performance GPU computer, a GPU computing cluster, a dedicated edge computing device, or an embedded system integrated in a vehicle. This application also does not impose any restrictions on the specific type of the computer module.

[0034] Figure 2 It is a flowchart of an image fusion segmentation method shown in an exemplary embodiment of the present application.

[0035] like Figure 2 As shown, in an exemplary embodiment, the image fusion segmentation method includes at least steps S210 to S230, which are described in detail as follows:

[0036] Step S210, acquiring an initial image, and dividing the initial image into a plurality of initial image channels.

[0037] In one embodiment of the present application, it is assumed that there is an autonomous driving vehicle equipped with multiple sensors, on which a visible light camera is configured. Take the environment image collected by the vehicle as an initial image as an example. First, a color image is taken by an RGB camera, and then the collected image is segmented into three color channels of red (R), green (G), and blue (B) based on the RGB image, as well as an additional information channel for other additional information.

[0038] Step S220, performing feature fusion processing on the initial image channels, the feature fusion processing includes calculating the cosine similarity of each initial image channel at each pixel position, and using the cosine similarity as the weight factor of the corresponding image channel; fusing each initial image channel based on the weight factor to obtain each fused image channel and a comprehensive image channel, the comprehensive image channel is the result of the fusion of all initial image channels.

[0039] In one embodiment of the present application, the cosine similarity of each initial image channel at each pixel position is calculated, including: taking any initial image channel as a current image channel; calculating the cosine similarity between the current image channel and other initial image channels; determining the cosine similarity of the current image channel according to the cosine similarity between the current image channel and other initial image channels; and traversing all initial image channels to obtain the cosine similarity corresponding to each initial image channel.

[0040] In a specific embodiment of the present application, determining the cosine similarity of the current image channel according to the cosine similarity between the current image channel and other initial image channels includes: dividing the initial image into four initial image channels, including an original red channel, an original green channel, an original blue channel, and an original additional data channel; calculating the cosine similarity between the original red channel and the original green channel, recorded as a first red cosine similarity; calculating the cosine similarity between the original red channel and the original blue channel, recorded as a second red cosine similarity; calculating the cosine similarity between the original red channel and the original additional data channel, recorded as a third red cosine similarity; determining the average of the first red cosine similarity, the second red cosine similarity, and the third red cosine similarity as the cosine similarity of the original red channel; and traversing the remaining initial image channels to obtain the cosine similarities corresponding to each initial image channel.

[0041] In a specific embodiment of the present application, the initial image is divided into four channels, thereby obtaining feature images with different channels, as follows:

[0042] Set the feature images of any two channels to F A and F B ,in Input it into the adaptive pooling layer and reshape it to get and The dimension after shaping is C×N, where N=H×W. In this formula, represents all real numbers, C is the number of channels, H is the image height, and W is the image width.

[0043] The cosine similarity of the reshaped images is calculated. Assuming that the cosine similarity is expressed by , the formula is:

[0044]

[0045] Among them, S c represents the cosine similarity between any two channels, and are the reshaped feature images, and ||·|| is the vector norm.

[0046] It should be noted that S c The value range is between -1 and 1.

[0047] The cosine similarity is shaped and input into two convolutional layers to obtain the shaped cosine similarity, which is denoted as S c ′, normalized using the sigmoid function. In the two convolutional layers, a batch normalization layer BN and a ReLU layer are also included as activation functions. The formula is:

[0048] W=σ(Conv 1×1 (ReLU(BN(Conv 1×1 (S′ c ))))) Formula (2)

[0049] Among them, σ represents the sigmoid function, Conv 1×1 represents 1*1 convolution, ReLU represents ReLU activation function, and BN represents batch normalization layer.

[0050] The obtained weights are applied to different modules to obtain the cosine similarity fusion image, as shown in the formula,

[0051] The fusion output of any two channels is:

[0052] F′ A =F A +F B ×W Formula (3)

[0053] F′ B =F B +F A ×(1-W) Formula (4)

[0054] Where W is the cosine weight factor, F′ A and F′B They are the images after the A channel and B channel are fused.

[0055] Then fuse the original images of any two channels and finally get the average value, then we have:

[0056] F M =F B ×W+F A ×(1-W)

[0057]

[0058] Among them, F M is the fusion image between any two modalities, This is the final image after the fusion of the four modalities.

[0059] It should be noted that, in the above embodiment, two image channels are used as an example for description. In practical applications, the same calculation process can be used to process multiple image channels, which will not be described one by one here.

[0060] Step S230, using each fused image channel as the input value of the next feature fusion processing, repeating the above-mentioned feature fusion processing steps, obtaining the comprehensive image channel corresponding to each feature fusion processing, so as to obtain the fused segmentation result of the initial image based on all the comprehensive image channels.

[0061] In one embodiment of the present application, after obtaining the comprehensive image channel, data processing is performed on it through an Effective Context Extraction module to emphasize its image features. The specific steps are as follows:

[0062] a) Assuming the image dimension is C×H×W, the final output image is input to the adaptive average pooling layer;

[0063] b) Use 1×1 convolution to reduce the dimension of the image, and the output dimension is Then pass through the BN layer and the ReLU layer;

[0064] c) Apply parallel convolution to the output of step b), with the convolution layers being 1×4 convolution and 4×1 convolution, followed by a BN layer and a ReLU layer;

[0065] d) Use bilinear interpolation to upsample the features;

[0066] e) Use element-by-element addition to merge the branch results, and then use 3×3 convolution to output the final result.

[0067] It should be noted that the specific convolution scheme is only used as an exemplary description. In actual application, other related schemes can also be used to extract features from the comprehensive image channel. Its core purpose is to extract useful information or features related to a specific task from the comprehensive image channel. Therefore, this application does not impose any specific restrictions on the convolution method and its related data processing method.

[0068] In one embodiment of the present application, a fused segmentation result of an initial image is obtained based on all integrated image channels, including: selecting the integrated image channel after the last fusion as the current fused image channel according to the feature fusion order, and performing image restoration processing on the current fused image, the image restoration processing including upsampling the current fused image channel to obtain the current upsampling result, fusing the current upsampling result with the previous integrated image channel to obtain the current fusion result; using the current fusion result as the input for the next image restoration, and repeating the image restoration processing until the fusion and upsampling processing are completed for all fused integrated image channels to obtain the fused segmentation result of the initial image.

[0069] In one embodiment of the present application, it is assumed that three comprehensive image channels are obtained according to three feature fusion processes, and they are Feature_1, Feature_2, and Feature_3 in the order of feature fusion processing. The comprehensive image channel after the last fusion (i.e., Feature_1) is used as the current fused image channel, and Feature_1 is upsampled to obtain the current upsampled result. Then, this upsampled result is fused with the next comprehensive image channel Feature_2 to obtain the current fusion result, and the current fusion result is used as the input of the next image restoration process, and the current fusion result is upsampled to obtain a new upsampled result, and then this new upsampled result is fused with the next comprehensive image channel Feature_3 to obtain the final fusion result, and the final result is the fusion segmentation result of the initial image.

[0070] In one embodiment of the present application, image restoration processing is performed on the current fused image, and the process also includes: inputting the current fused image channel into one or more preprocessing layers to enhance the image features of the current fused image channel, the preprocessing layers including pooling layers, convolution layers, batch normalization layers or activation function layers; performing dimensionality reduction processing on the current fused image channel to adjust the image dimension of the current fused image channel; extracting the image features of the current fused image channel to enhance the feature expression capability of the current fused image channel.

[0071] In one embodiment of the present application, after obtaining the fused segmentation result of the initial image, it includes: inputting the segmentation result of the initial image into a target detection module, the target detection module is used to identify a specific target in the segmentation result and obtain the position information of the specific target, the specific target includes at least one of a pedestrian, an obstacle, and a lane line; sending the position information to a driving assistance system to trigger the driving assistance system to take corresponding driving assistance measures according to the received position information to ensure driving safety, the driving assistance measures include issuing a warning signal, activating an automatic emergency braking system, and adjusting at least one of the vehicle's driving direction.

[0072] In a specific embodiment of the present application, it is assumed that the feature fusion segmentation processing of the image has been completed, and the segmentation results including pedestrians, obstacles and lane lines have been obtained, and the driving assistance system is in a standby state, ready to receive the position information from the target detection module. After receiving the segmentation results, the target detection module begins to identify pedestrians, obstacles and lane lines in the image, and then performs subsequent processing based on the precise position information of each target calculated by the target detection module. Take the identification of a pedestrian, whose position information is: pedestrian center point coordinates: (150,120), pedestrian bounding box size: (80,150), lane center line coordinates: [(50,50), (50,480)] as an example. The target detection module sends the position information of each target to the driving assistance system, and the data format sent can be a list or a dictionary, which contains the position information of each target. Then, after receiving the location information, the driving assistance system takes appropriate measures based on the target type and distance, including: if a pedestrian is detected approaching in front, the system may issue a sound warning to alert the driver; if an obstacle is detected suddenly in front and the vehicle is moving too fast, the system may automatically activate the emergency braking system to avoid a collision; if the vehicle is detected to deviate from the lane line, the system may slightly adjust the steering wheel to keep it within the lane.

[0073] In another embodiment of the present application, after obtaining the fused segmentation result of the initial image, it also includes: fusing the fused segmentation result with the initial image to generate a fused image containing a target area, wherein the target area includes a key object with annotation information; displaying the fused image on a display in the cockpit of the vehicle to facilitate the driver to determine the relative position of the key object and the vehicle; combining the current position data of the vehicle and the relative position of the key object and the vehicle, predicting the collision risk between the vehicle and the key object, and generating a prediction result; based on the prediction result, triggering the driving assistance system to perform corresponding avoidance actions or issue an alarm to the driver.

[0074] It should be noted that the above embodiment is only an exemplary description of the application scenario of the fusion segmentation result of the initial image. In the actual application process, the fusion segmentation result based on the initial image can also be used for other more complex applications. Especially in the special operations environment, the application scope of the fusion segmentation result is wider, and can be applied to a series of application scenarios such as battlefield situation awareness, target tracking and identification, intelligent navigation and path planning, and covert action support. Therefore, this application does not impose any restrictions on its application scenarios and application methods.

[0075] Figure 4 It is a schematic diagram of the complete steps of the image fusion segmentation method shown in an exemplary embodiment of the present application.

[0076] like Figure 4 As shown, the obtained initial image is first divided into four channels, namely, an X grid information channel, a B blue channel, a G green channel, and an R red channel; then, the four channels obtained by the above division are downsampled to obtain a C / 2 image corresponding to each channel, and the obtained C / 2 image is input into a cosine similarity fusion module to perform fusion processing on multiple image channels to obtain a C / 4 image (i.e., the image channel after the first fusion) and a Y1 image channel (i.e., the comprehensive image channel after the first fusion) corresponding to each image channel; similarly, the obtained C / 4 image is input into a cosine similarity fusion module to perform fusion processing on multiple image channels to obtain a C / 8 image (i.e., the image channel after the second fusion) and a Y2 image channel (i.e., the comprehensive image channel after the second fusion) corresponding to each image channel; the obtained C / 8 image is input into a cosine similarity fusion module to perform fusion processing on multiple image channels to obtain a C / 16 image (i.e., the image channel after the third fusion) and a Y3 image channel (i.e., the comprehensive image channel after the third fusion) corresponding to each image channel. Then, the comprehensive image channel after the third fusion is used as the current image channel, and further convolved to obtain the C / 32 image, which is input into the efficient context extraction module to extract information to obtain the corresponding Y3' data. Then, the Y3' data is upsampled to obtain the corresponding A1 data, and the A1 data and the Y3 data are input into the cosine similarity fusion to obtain the A1' data; similarly, the A1' data is upsampled to obtain the corresponding A2 data, and the A2 data and the Y2 data are input into the cosine similarity fusion to obtain the A2' data; the A2' data is upsampled to obtain the corresponding A3 data, and the A3 data and the Y1 data are input into the cosine similarity fusion to obtain the A3' data, and finally the A3' data is upsampled to obtain the A4 data, and the A4 data obtained at the end is used as the fused segmentation image corresponding to the initial image.

[0077] In a specific embodiment of the present application, the steps of performing image segmentation on the acquired initial image are as follows:

[0078] a) Using RGB-X equipment to capture images, thereby obtaining the image to be segmented that needs to be input;

[0079] b) segmenting the input image according to the color channel and the additional information channel to obtain quad-modal data;

[0080] c) The quad-modal data, i.e. the original red channel F R , original green channel F G , original blue channel F B And the original extra data channel F X Downsampling is performed, where R is red, G is green, B is blue, and X is additional information:

[0081] d) Input the four channels after downsampling output into the cosine similarity fusion module to obtain the weight information between different modalities and achieve the effect of modal fusion. A total of five outputs are obtained, namely the fused red channel F′ R , the fused green channel F′ G , the fused blue channel F′ B , the additional data channel F after fusion X ′, and multimodal fusion data F M ;

[0082] e) Repeat c) twice and perform cosine similarity fusion, and finally output only multimodal fusion data;

[0083] f) downsampling the multimodal fusion data twice;

[0084] g) extracting features using a context information acquisition module;

[0085] h) performing an upsampling operation on the output of step g), and using a skip connection to input the first output of step f) into the cosine similarity fusion module;

[0086] i) upsampling the output of step h) and fusing it with the data of step e) using cosine similarity.

[0087] j) Upsample the output of step i) together with the multimodal fusion data F in step d) M They are input into the cosine similarity fusion module together, and finally upsampled to obtain the semantic segmentation result.

[0088] It should be noted that the above embodiment shows the whole process steps of fusion segmentation of the initial image. However, in the above embodiment, the step of feature fusion processing to obtain the comprehensive image channel is only repeated twice. However, in actual use, the number of cycles can be adaptively adjusted based on specific needs and computing power, and this application does not impose any specific restrictions on it.

[0089] The above embodiment shows the whole process steps of fusing and segmenting the initial image. However, in the above embodiment, the step of obtaining the comprehensive image channel by feature fusion processing is only repeated twice. In the actual application process, the number of cycles can be adaptively adjusted according to specific needs and computing power conditions. Specifically, if the available computing resources are relatively abundant, the number of cycles can be increased to obtain a more refined fusion result. If the computing resources are limited, it is necessary to weigh the relationship between the number of cycles and the processing time. In addition, different application scenarios may have different requirements for fusion results. For example, in medical image analysis that requires high precision, more cycles may be required to improve the accuracy of detail recognition; in real-time processing or resource-constrained scenarios, such as applications on mobile devices, it may be necessary to reduce the number of cycles to reduce latency. Therefore, this application does not impose any specific restrictions on the number of cycles.

[0090] Finally, it should be noted that a multi-branch cosine similarity-based image fusion segmentation method is proposed. This method first divides the RGB-X image captured in the driving scene into four channels: R (red), G (green), B (blue) and X (depth, heat, polarization). Then, these four channels are respectively input into the four-branch network for feature extraction. Considering that there may be noise or other features in the input data, in order to avoid the loss of these features, this method designs a cosine similarity fusion module, which corrects and fuses cross-modal features through cosine similarity. Specifically, this method uses adaptive average pooling to downsample the input four modal data, then reshapes the data, and then performs cosine similarity analysis. Each different cosine similarity will assign different weights to different modules, so as to obtain additional modal information and achieve the effect of multimodal fusion. In order to better capture multimodal information, this paper also adopts a context information acquisition module to model long-range dependencies. Finally, through three downsampling encoding and three upsampling decoding, the purpose of semantic segmentation is achieved. It effectively improves the accuracy and robustness of image segmentation, is suitable for a variety of driving scenarios, and has good real-time processing capabilities, providing strong technical support for applications such as autonomous driving.

[0091] Figure 5 is a block diagram of an image fusion and segmentation device shown in an exemplary embodiment of the present application. The device can be applied to Figure 1The device may also be applicable to other exemplary implementation environments and specifically configured in other devices, and this embodiment does not limit the implementation environment to which the device is applicable.

[0092] like Figure 5 As shown, the exemplary image fusion and segmentation device includes: an image preprocessing module 510, a feature fusion module 520, and a loop processing module 530.

[0093] Among them, the image preprocessing module 510 is used to obtain an initial image and divide the initial image into multiple initial image channels; the feature fusion module 520 is used to perform feature fusion processing on the initial image channels, and the feature fusion processing includes calculating the cosine similarity of each initial image channel at each pixel position, and using the cosine similarity as the weight factor of the corresponding image channel; fusing each initial image channel based on the weight factor to obtain each fused image channel and a comprehensive image channel, and the comprehensive image channel is the result of the fusion of all initial image channels; the loop processing module 530 is used to use each fused image channel as the input value of the next feature fusion processing, repeat the above-mentioned feature fusion processing steps, and obtain the comprehensive image channel corresponding to each feature fusion processing, so as to obtain the fused segmentation result of the initial image based on all the comprehensive image channels.

[0094] It should be noted that the image fusion segmentation device provided in the above embodiment and the image fusion segmentation method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the image fusion segmentation device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0095] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device implements the image fusion and segmentation methods provided in the above-mentioned embodiments.

[0096] Figure 6 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Figure 6 The computer system 600 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0097] like Figure 6As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage part 608 to the random access memory (RAM) 603, such as executing the method described in the above embodiment. In the RAM 603, various programs and data required for system operation are also stored. The CPU 601, ROM 602 and RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0098] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed so that a computer program read therefrom is installed into the storage section 608 as needed.

[0099] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 609, and / or installed from a removable medium 611. When the computer program is executed by a central processing unit (CPU) 601, various functions defined in the system of the present application are executed.

[0100] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0101] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0102] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0103] Another aspect of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor of a computer, the computer executes the image fusion segmentation method as described above. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0104] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the image fusion segmentation method provided in each of the above embodiments.

[0105] The above embodiments are merely illustrative of the principles and effects of the present application, and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. An image fusion segmentation method, characterized in that: The method comprises: Acquire an initial image, and divide the initial image into a plurality of initial image channels; Performing feature fusion processing on the initial image channels, the feature fusion processing comprising: calculating the cosine similarity of each initial image channel at each pixel position, and using the cosine similarity as a weight factor of the corresponding image channel; fusing each initial image channel based on the weight factor to obtain each fused image channel and a comprehensive image channel, the comprehensive image channel being the result of fusion of all initial image channels; Each fused image channel is used as the input value of the next feature fusion processing, and the above-mentioned feature fusion processing steps are repeated to obtain the comprehensive image channel corresponding to each feature fusion processing, so as to obtain the fused segmentation result of the initial image based on all the comprehensive image channels.

2. The image fusion segmentation method according to claim 1, characterized in that: The fusion segmentation result of the initial image is obtained based on all comprehensive image channels, including: According to the feature fusion order, the comprehensive image channel after the last fusion is selected as the current fused image channel, and the image restoration processing is performed on the current fused image, wherein the image restoration processing includes upsampling the current fused image channel to obtain a current upsampling result, and fusing the current upsampling result with the previous comprehensive image channel to obtain a current fusion result; The current fusion result is used as the input for the next image restoration, and the image restoration process is repeated until the fusion and upsampling processes are completed on all fused comprehensive image channels to obtain the fused segmentation result of the initial image.

3. The image fusion segmentation method according to claim 2, characterized in that: Performing image restoration processing on the current fused image also includes: Inputting the current fused image channel to one or more preprocessing layers to enhance image features of the current fused image channel, wherein the preprocessing layers include pooling layers, convolutional layers, batch normalization layers, or activation function layers; Performing dimensionality reduction processing on the current fused image channel to adjust the image dimension of the current fused image channel; The image features of the current fused image channel are extracted to enhance the feature expression capability of the current fused image channel.

4. The image fusion segmentation method according to claim 1, characterized in that: Calculate the cosine similarity of each initial image channel at each pixel position, including: Take any initial image channel as the current image channel; Calculating the cosine similarity between the current image channel and other initial image channels; Determining the cosine similarity of the current image channel according to the cosine similarity between the current image channel and other initial image channels; All initial image channels are traversed to obtain the cosine similarity corresponding to each initial image channel.

5. The image fusion segmentation method according to claim 4, characterized in that: Determining the cosine similarity of the current image channel according to the cosine similarity between the current image channel and other initial image channels includes: The initial image is divided into four initial image channels, including an original red channel, an original green channel, an original blue channel, and an original additional data channel; Calculate the cosine similarity between the original red channel and the original green channel, recorded as the first red cosine similarity; calculate the cosine similarity between the original red channel and the original blue channel, recorded as the second red cosine similarity; calculate the cosine similarity between the original red channel and the original extra data channel, recorded as the third red cosine similarity; Determine the average of the first red cosine similarity, the second red cosine similarity, and the third red cosine similarity as the cosine similarity of the original red channel; The remaining initial image channels are traversed to obtain the cosine similarity corresponding to each initial image channel.

6. The image fusion segmentation method according to any one of claims 1 to 5, characterized in that: After obtaining the fusion segmentation result of the initial image, the following steps are performed: Inputting the segmentation result of the initial image into a target detection module, the target detection module is used to identify a specific target in the segmentation result and obtain the position information of the specific target, the specific target includes at least one of a pedestrian, an obstacle, and a lane line; The location information is sent to a driving assistance system to trigger the driving assistance system to take corresponding driving assistance measures according to the received location information to ensure driving safety. The driving assistance measures include at least one of issuing a warning signal, activating an automatic emergency braking system, and adjusting the vehicle's driving direction.

7. The image fusion segmentation method according to any one of claims 1 to 5, characterized in that: After obtaining the fusion segmentation result of the initial image, the method further includes: Fusing the fused segmentation result with the initial image to generate a fused image containing a target area, wherein the target area includes a key object with annotation information; Displaying the fused image on a display in a cockpit of a vehicle so that a driver can determine the relative position of the key object to the vehicle; Based on the current position data of the vehicle and the relative position of the key object and the vehicle, predict the collision risk between the vehicle and the key object, and generate a prediction result; Based on the prediction result, the driving assistance system is triggered to perform corresponding evasive actions or issue an alarm to the driver.

8. An image fusion and segmentation device, characterized in that: The device comprises: An image preprocessing module, used for acquiring an initial image and dividing the initial image into a plurality of initial image channels; A feature fusion module is used to perform feature fusion processing on the initial image channels, wherein the feature fusion processing includes calculating the cosine similarity of each initial image channel at each pixel position and using the cosine similarity as a weight factor of the corresponding image channel; fusing each initial image channel based on the weight factor to obtain each fused image channel and a comprehensive image channel, wherein the comprehensive image channel is the result of fusion of all initial image channels; The loop processing module is used to use each fused image channel as the input value of the next feature fusion processing, repeat the above-mentioned feature fusion processing steps, obtain the comprehensive image channel corresponding to each feature fusion processing, and obtain the fused segmentation result of the initial image based on all the comprehensive image channels.

9. An electronic device, characterized in that: It comprises a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute the computer program stored in the memory to implement the image fusion segmentation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is used to enable a computer to execute the image fusion and segmentation method as described in any one of claims 1 to 7.

Images