Multimodal image segmentation method, device, electronic device and computer readable medium

Through multimodal feature fusion and enhancement processing, the problem of low accuracy and completeness of existing text-guided image segmentation models in multimodal image segmentation is solved, and higher image segmentation accuracy and mask integrity are achieved. Especially in complex scenarios, edge information and noise processing.

CN118823359BActive Publication Date: 2024-11-29BEIJING ZHONGQI HUIYUN TECH CO LTD +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411312378.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-20
Publication Date
2024-11-29
Estimated Expiration
2044-09-20

AI Technical Summary

Technical Problem

The existing text-guided image segmentation model lacks targeting in multimodal image segmentation, and only modal fusion of high-level features leads to low image segmentation accuracy and low integrity of segmentation mask product images, and in complex scenarios It is prone to image edge information loss and noise interference, affecting the segmentation accuracy.

Method used

By obtaining the target product image and product image description text information, input it to the feature extraction network and the text description feature extraction network respectively, performing multimodal feature fusion, multi-scale feature alignment fusion and feature enhancement, generating cross-modal enhancement feature vectors, and Linearization processing and vector multiplication are performed to obtain multimodal mask segmentation feature vectors for image segmentation.

Benefits of technology

It improves the accuracy of image segmentation and the integrity of segmentation mask product images, gives full play to the guiding role of text on image segmentation, enhances the processing ability of noise and information loss, and improves the quality of segmentation results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118823359B_ABST
    Figure CN118823359B_ABST
Patent Text Reader

Abstract

The embodiments of the present disclosure disclose a multimodal image segmentation method, device, electronic device and computer-readable medium. A specific implementation of the method includes: inputting the target product image into a product image feature extraction network to obtain a product image feature vector set; inputting the product image description text information into a text description feature extraction network to obtain a product text feature vector; performing multimodal feature fusion and multiscale feature alignment fusion processing on the product image feature vector set and the product text feature vector to obtain a multiscale alignment fusion feature vector; performing feature enhancement processing on the multiscale alignment fusion feature vector and the product text feature vector to obtain a cross-modal enhanced feature vector; performing image segmentation processing on the target product image to obtain a segmentation mask product image set. This implementation gives full play to the guiding role of text through multiple multimodal fusions, improves the accuracy of image segmentation and the integrity of the segmentation mask product image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a multimodal image segmentation method, device, electronic device, and computer-readable medium. Background Art

[0002] At present, with the rapid development of deep learning technology in the field of artificial intelligence, text-guided image segmentation technology achieves accurate segmentation of target objects on images by integrating natural language descriptions and image information, which is more in line with the way humans process information and has become a key issue of concern. For the segmentation of multimodal images, the usual method is to input the target product image and the product image description text information into the existing text-guided image segmentation model (using the attention mechanism to fuse the two modalities of image and language) to obtain a segmentation mask product image set.

[0003] However, it is found in practice that when the above method is used to segment multimodal images, the following technical problems often occur: Since most of the existing text-guided image segmentation models are based on simple images (such as landscapes, people, animals and plants, and medical images), they lack specificity for product images, and the existing text-guided image segmentation models perform modal fusion through the attention mechanism, only performing modal fusion on high-level features, and the fusion is dominated by visual features, lacking the capture of the correlation between text and image, resulting in low image segmentation accuracy and low integrity of the segmentation mask product image, which increases the number of image segmentation times.

[0004] In the process of adopting technical solutions to solve the above-mentioned technical problem 1, the following technical problem 2 is often accompanied: in the process of multi-scale feature alignment and fusion of text and image vision, the attention coefficient is only generated by a single feature, which makes the accuracy and quality of the feature vector generated by the multi-scale feature alignment and fusion of text and image vision low, resulting in low accuracy of text-guided image segmentation. For the above-mentioned technical problem 2, the conventional solution is generally: only convolution processing is performed on the product text feature vector to generate a text-image fusion feature vector with an attention coefficient generated only by text features. However, the above-mentioned conventional solution still has the following problem: due to the fusion feature vector of the attention coefficient generated only by a single text feature, the quality of the feature vector obtained by multi-scale feature alignment and fusion is low, which further causes the accuracy of image segmentation to be low and the integrity of the segmented mask product image to be low.

[0005] In the process of adopting technical solutions to solve the above-mentioned technical problem one, the following technical problem three is often accompanied: for target product images in complex scenes, there may be problems such as image edge information loss and noise interference, resulting in low accuracy of text and image visual fusion, and further resulting in low accuracy of image segmentation. For the above-mentioned technical problem three, the conventional solution is generally: through a decoder based on the attention mechanism, feature enhancement is performed on the multi-scale aligned fusion feature vector and the product text feature vector. However, the above-mentioned conventional solution still has the following problems: because the decoder based on the attention mechanism can only fuse the extracted text features and image features with high-level information and extract correlation relationships, it cannot solve the problem of insufficient boundary information extraction and removal of noise data, resulting in low quality of the enhanced feature vector and low accuracy of image segmentation.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0007] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0008] Some embodiments of the present disclosure propose a multimodal image segmentation method, apparatus, electronic device, and computer-readable medium to solve one or more of the technical problems mentioned in the above background technology section.

[0009] In a first aspect, some embodiments of the present disclosure provide a multimodal image segmentation method, comprising: obtaining a target product image and product image description text information, wherein the target product image and the product image description text information have a semantic association relationship; inputting the target product image into a product image feature extraction network to obtain a product image feature vector set; inputting the product image description text information into a text description feature extraction network to obtain a product text feature vector; performing multimodal feature fusion processing on the product image feature vector set and the product text feature vector to obtain a multimodal fusion feature vector; The vector is subjected to multi-scale feature alignment and fusion processing to obtain a multi-scale aligned fused feature vector; the multi-scale aligned fused feature vector and the product text feature vector are subjected to feature enhancement processing to obtain a cross-modal enhanced feature vector; the product text feature vector is subjected to linearization processing to obtain a linearized product text feature vector; the cross-modal enhanced feature vector and the linearized product text feature vector are subjected to vector multiplication to obtain a multi-modal mask segmentation feature vector; according to the multi-modal mask segmentation feature vector, the target product image is subjected to image segmentation processing for the text information describing the product image to obtain a segmentation mask product image set.

[0010] In a second aspect, some embodiments of the present disclosure provide a multimodal image segmentation device, including: an acquisition unit, configured to acquire a target product image and product image description text information, wherein the target product image and the product image description text information have a semantic association relationship; a first input unit, configured to input the target product image into a product image feature extraction network to obtain a product image feature vector set; a second input unit, configured to input the product image description text information into a text description feature extraction network to obtain a product text feature vector; a multimodal feature fusion unit, configured to perform multimodal feature fusion processing on the product image feature vector set and the product text feature vector to obtain a multimodal fusion feature vector; and a multiscale feature alignment and fusion unit, configured to perform multimodal feature fusion processing on the product image feature vector set and the product text feature vector to obtain a multimodal fusion feature vector. The multimodal fusion feature vector is subjected to multiscale feature alignment and fusion processing to obtain a multiscale aligned fusion feature vector; a feature enhancement unit is configured to perform feature enhancement processing on the multiscale aligned fusion feature vector and the product text feature vector to obtain a cross-modal enhanced feature vector; a linearization processing unit is configured to perform linearization processing on the product text feature vector to obtain a linearized product text feature vector; a vector multiplication unit is configured to perform vector multiplication on the cross-modal enhanced feature vector and the linearized product text feature vector to obtain a multimodal mask segmentation feature vector; an image segmentation unit is configured to perform image segmentation processing on the target product image according to the multimodal mask segmentation feature vector for the product image description text information to obtain a segmentation mask product image set.

[0011] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0012] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method described in any implementation manner in the first aspect is implemented.

[0013] The above-mentioned embodiments of the present disclosure have the following beneficial effects: the multimodal image segmentation method of some embodiments of the present disclosure gives full play to the guiding role of text in image segmentation by multiple fusions of images and texts, and improves the accuracy of image segmentation and the integrity of the segmentation mask product image. Specifically, the reason for the low image segmentation accuracy is that the existing text-guided image segmentation models are mostly based on simple images (such as landscapes, characters, animals and plants, and medical images), lack of pertinence to product images, and the existing text-guided image segmentation models perform modal fusion through the attention mechanism, only perform modal fusion on high-level features, and take visual features as the leading fusion, lack of capturing the association between text and image, resulting in low image segmentation accuracy and low integrity of the segmentation mask product image, and increasing the number of image segmentation times. Based on this, the multimodal image segmentation method of some embodiments of the present disclosure can first obtain the target product image and the product image description text information, wherein the target product image and the product image description text information have a semantic association relationship. Here, the target product image and the image description text information are used for subsequent text and image feature extraction. Secondly, the target product image is input into the product image feature extraction network to obtain a product image feature vector set. Here, the key information in the target product image can be extracted to improve the detail information and high resolution of the product image feature vector. Again, the above product image description text information is input into the text description feature extraction network to obtain the product text feature vector. Here, the key semantic information in the product image description text information can be extracted to improve the quality of the product text feature vector and enrich the semantic information. Then, the above product image feature vector set and the above product text feature vector are subjected to multimodal feature fusion processing to obtain a multimodal fusion feature vector. Here, preliminary cross-modal interactive fusion is performed to extract richer semantic information while maintaining the high-resolution detail features of the image. Subsequently, the above multimodal fusion feature vector is subjected to multiscale feature alignment fusion processing to obtain a multiscale alignment fusion feature vector. Here, text image features of different levels and scales can be effectively integrated, and the detail information at high resolution and the semantic information at low resolution can be fully integrated to capture the association between text and image. Afterwards, the above multiscale alignment fusion feature vector and the above product text feature vector are subjected to feature enhancement processing to obtain a cross-modal enhanced feature vector. Here, noise can be removed and information loss problems can be solved, and key visual clues can be further extracted from the fused features. Then, the product text feature vector is linearized to obtain a linearized product text feature vector. Here, the product text feature vector is transformed in feature dimension so as to conform to the dimension of the cross-modal enhancement feature vector for vector multiplication and improve the guiding role of the text in image segmentation. Then, the cross-modal enhancement feature vector and the linearized product text feature vector are vector multiplied to obtain a multimodal mask segmentation feature vector.Here, the guiding role of text on image segmentation can be fully reflected. Finally, according to the multimodal mask segmentation feature vector, the target product image is subjected to image segmentation processing for the text information describing the product image, and a segmentation mask product image set is obtained. Here, the accuracy of image segmentation and the image integrity of the segmentation mask product image set can be improved. It can be concluded that the multimodal image segmentation method fully utilizes the guiding role of text on image segmentation through multiple fusions of images and texts, thereby improving the accuracy of image segmentation and the integrity of the segmentation mask product images. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0015] Figure 1 is a flow chart of some embodiments of the multimodal image segmentation method according to the present disclosure;

[0016] Figure 2 is a schematic structural diagram of some embodiments of a multimodal image segmentation device according to the present disclosure;

[0017] Figure 3 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0020] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0021] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0022] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0023] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0024] Figure 1 The process 100 of some embodiments of the multimodal image segmentation method according to the present disclosure is shown. The multimodal image segmentation method comprises the following steps:

[0025] Step 101, obtaining a target product image and product image description text information.

[0026] In some embodiments, the execution subject (e.g., electronic device) of the multimodal image segmentation method can obtain the target product image and the product image description text information through a wired connection or a wireless connection, wherein the target product image and the product image description text information have a semantic association relationship. The target product image can be an image of a processed part or an intermediate part related to industrial manufacturing that displays a product. For example, the target product image can be an image of automobile parts such as automobile lights and chassis. The product image description text information can be text information that describes the type, quantity, and form of the product in the target product image in the form of text. The semantic association relationship can be that the image description text information is information describing the target product image.

[0027] Step 102: input the target product image into a product image feature extraction network to obtain a product image feature vector set.

[0028] In some embodiments, the execution entity may input the target product image into a product image feature extraction network to obtain a product image feature vector set. The product image feature vector in the product image feature vector set may represent the visual information of the target product image. For example, the visual information may include but is not limited to at least one of the following: color, texture, shape. The product image feature extraction network may be a deep neural network that extracts the visual information of the input target product image. The product image feature extraction network may be a model that converts the global maximum pooling layer included in the ResNet (Residual Neural Network)-D model into an anti-aliased rect-2 blur pool. D may represent an improved version identifier of the ResNet model.

[0029] In some optional implementations of some embodiments, the product image feature extraction network includes: a convolutional network, a global maximum pooling layer, a first-stage convolutional residual network, a second-stage convolutional residual network, a third-stage convolutional residual network, and a fourth-stage convolutional residual network. Among them, the convolutional network can be a convolutional layer with a convolution kernel of 7*7, a number of convolution kernels of 64, a step size of 2, and an edge padding of 3. The first-stage convolutional residual network can include convolutional layers of 3 residual bottleneck attention convolutional networks. The residual bottleneck attention convolutional network includes: 3 convolutional layers with a convolution kernel of 1*1, 1 convolutional layer with a convolution kernel of 3*3 and a step size of 2, and 1 attention pooling layer. The attention pooling layer can be a multi-head QKV attention layer in the form of a single-layer Transformer. The second-stage convolutional residual network can be a convolutional layer including 4 residual bottleneck attention convolutional networks. The third-stage convolutional residual network can be a convolutional layer including 6 residual bottleneck attention convolutional networks. The fourth-stage convolutional residual network may be a convolutional layer including three residual bottleneck attention convolutional networks.

[0030] Optionally, the step of inputting the target product image into a product image feature extraction network to obtain a product image feature vector set may include the following steps:

[0031] In the first step, the target product image is convolved through the convolutional network to obtain a first image feature vector.

[0032] The second step is to perform fuzzy filtering on the first image feature vector to obtain the filtered first image feature vector. The fuzzy filtering can be a filtering process performed using a Gaussian blur algorithm.

[0033] In the third step, the global maximum pooling layer is used to perform global maximum pooling processing on the filtered first image feature vector to obtain a second image feature vector.

[0034] In the fourth step, the second image feature vector is input into the first-stage convolution residual network to obtain the third image feature vector. The first-stage convolution residual network may include: a first residual bottleneck attention convolution network, a second residual bottleneck attention convolution network and a third residual bottleneck attention convolution network. In practice, the execution subject may first input the second image feature vector into the 1*1 convolution layer included in the first residual bottleneck attention convolution network to obtain the seventh image feature vector. Secondly, the seventh image feature vector is input into the 3*3 convolution layer to obtain the eighth image feature vector. Again, the eighth image feature vector is input into the 1*1 convolution layer to obtain the ninth image feature vector. Subsequently, the second image feature vector is input into the attention pooling layer to obtain the tenth image feature vector. Next, the tenth image feature vector is input into the 1*1 convolution layer to obtain the eleventh image feature vector. Then, the ninth image feature vector and the eleventh image feature vector are feature added to obtain the twelfth image feature vector. Afterwards, the twelfth image feature vector is input into the second residual bottleneck attention convolutional network to obtain the thirteenth image feature vector. Finally, the thirteenth image feature vector is input into the third residual bottleneck attention convolutional network to obtain the third image feature vector.

[0035] The fifth step is to input the third image feature vector into the second stage convolutional residual network to obtain a fourth image feature vector.

[0036] In the sixth step, the fourth image feature vector is input into the third stage convolutional residual network to obtain the fifth image feature vector.

[0037] In the seventh step, the fifth image feature vector is input into the fourth-stage convolutional residual network to obtain a sixth image feature vector.

[0038] In the eighth step, the fourth image feature vector, the fifth image feature vector and the sixth image feature vector are determined as a product image feature vector set.

[0039] Step 103: input the product image description text information into a text description feature extraction network to obtain a product text feature vector.

[0040] In some embodiments, the execution entity may input the product image description text information into a text description feature extraction network to obtain a product text feature vector. The product text feature vector may represent the semantic information of the product image description text information. The semantic information may include but is not limited to at least one of the following: product target object category, location and attributes. The text description feature extraction network may be a deep neural network that extracts text semantic information of the product image description text information. For example, the text description feature extraction network may be a text encoder in a CLIP model.

[0041] In some optional implementations of some embodiments, the text description feature extraction network includes: multiple bidirectional attention encoding networks. The multiple bidirectional attention encoding networks can be a network of 12 Transformer models.

[0042] Optionally, the step of inputting the product image description text information into a text description feature extraction network to obtain a product text feature vector may include the following steps:

[0043] The first step is to perform character segmentation processing on the product image description text information to obtain a product text character set. The product text characters in the product text character set can be single Chinese characters. In practice, the execution subject can use the WordPiece algorithm to perform character segmentation processing on the product image description text information to obtain a product text character set.

[0044] The second step is to perform character encoding processing on each product text character in the product text character set to obtain a product text character feature vector set. The product text character feature vector in the product text character feature vector set may be a vectorized representation of a character.

[0045] The third step is to perform the following text feature extraction steps based on the product text character feature vector set:

[0046] Sub-step 1, input the product text character feature vector set and the preset text position encoding feature vector set into the multi-head attention mechanism layer included in the bidirectional attention encoding network to obtain the product text character weight feature vector set, wherein the above-mentioned bidirectional attention encoding network also includes: a multi-head attention mechanism layer (Multi-Head Attention), a residual connection layer and a layer normalization layer (Add & Norm), and a feedforward neural network layer (Feed Forward). Among them, the preset text position encoding feature vector in the above-mentioned preset text position encoding feature vector set can represent the position information of the product text characters in the above-mentioned product image description text information. The above-mentioned multi-head attention mechanism layer can be a network layer that determines the relationship between each text character in the product image description text information and the remaining text characters in the sentence. The above-mentioned residual connection layer and layer normalization layer can be network layers that solve the gradient disappearance, speed up the model training speed and the model convergence speed. The above-mentioned feedforward network includes two network layers, the activation function of the first layer of the feedforward network is ReLU (Rectified Linear Unit, linear rectification function), and the activation function of the second layer of the feedforward network is a nonlinear activation function.

[0047] Sub-step 2: Input the product text character weight feature vector set into the multi-head attention mechanism layer to obtain the product text character attention feature vector set.

[0048] Sub-step 3, input the product text character attention feature vector set into the residual connection layer and the layer normalization layer to obtain the product text character normalized feature vector set.

[0049] Sub-step 4, inputting the normalized feature vector set of product text characters into the feed-forward neural network layer to obtain a nonlinear feature vector set of product text.

[0050] Sub-step 5, in response to determining that the bidirectional attention encoding network is a terminating bidirectional attention encoding network, a plurality of product text nonlinear feature vector sets output by a preset number of bidirectional attention encoding networks adjacent to the terminating bidirectional attention encoding network are determined as product text feature vectors, wherein the terminating bidirectional attention encoding network is a bidirectional attention encoding network at the terminating position of the plurality of bidirectional attention encoding networks. The preset number may be a pre-set value. For example, the preset number may be 3.

[0051] In the fourth step, in response to determining that the above-mentioned bidirectional attention encoding network is not a terminating bidirectional attention encoding network, the bidirectional attention encoding network located after the bidirectional attention encoding network is determined as a bidirectional attention encoding network to perform the above-mentioned text feature extraction step again.

[0052] Step 104 , performing multimodal feature fusion processing on the product image feature vector set and the product text feature vector to obtain a multimodal fusion feature vector.

[0053] In some embodiments, the execution subject may perform multimodal feature fusion processing on the product image feature vector set and the product text feature vector to obtain a multimodal fusion feature vector, wherein the multimodal fusion feature vector may represent information that combines text semantic information and image visual information.

[0054] As an example, the execution subject may utilize an attention mechanism to perform cross-attention fusion on the product image feature vector set and the product text feature vector to obtain a multimodal fusion feature vector.

[0055] In some optional implementations of some embodiments, the product image feature vector set includes: a first product image feature vector, a second product image feature vector, and a third product image feature vector. The first product image feature vector, the second product image feature vector, and the third product image feature vector may be feature vectors output by the second stage convolutional residual network, the third stage convolutional residual network, and the fourth stage convolutional residual network included in the product image feature extraction network.

[0056] Optionally, the multimodal feature fusion processing is performed on the product image feature vector set and the product text feature vector to obtain the multimodal fusion feature vector, which may include the following steps:

[0057] In the first step, channel convolution processing is performed on the first product image feature vector to obtain the first product image channel feature vector. The channel convolution processing may be a convolution processing of inputting the first product image feature vector into a convolution layer with a convolution kernel of 1*1. The 1*1 convolution layer may be a convolution layer that changes the number of channels of the first product image feature vector.

[0058] The second step is to perform average pooling downsampling processing on the first product image channel feature vector to obtain the first product image downsampled feature vector. The average pooling downsampling processing may be a downsampling processing of inputting the first product image channel feature vector into an average pooling layer. The average pooling layer may be a network layer that doubles the resolution of the first product image channel feature vector.

[0059] The third step is to perform channel convolution processing on the second product image feature vector to obtain the second product image channel feature vector.

[0060] The fourth step is to perform channel convolution processing on the third product image feature vector to obtain the third product image channel feature vector.

[0061] Step 5: Perform bilinear interpolation upsampling on the third product image channel feature vector to obtain a third product image upsampling feature vector, wherein the number of channels of the first product image downsampling feature vector, the second product image channel feature vector and the third product image upsampling feature vector are the same.

[0062] Step 6: Perform channel feature concatenation on the downsampled feature vector of the first product image, the channel feature vector of the second product image, the upsampled feature vector of the third product image, and the preset visual position coding feature vector to obtain an image channel concatenated feature vector. The preset visual position coding feature vector may be pre-set and characterizes the position information of the product included in the target product image.

[0063] The seventh step is to perform channel convolution processing on the above product text feature vector to obtain the product channel text feature vector.

[0064] In the eighth step, Kronecker product processing is performed on the above-mentioned image channel splicing feature vector and the above-mentioned product channel text feature vector to obtain a multimodal fusion feature vector.

[0065] Step 105 , performing multi-scale feature alignment and fusion processing on the multi-modal fusion feature vector to obtain a multi-scale alignment and fusion feature vector.

[0066] In some embodiments, the execution subject may perform multi-scale feature alignment and fusion processing on the multi-modal fusion feature vector to obtain a multi-scale aligned fusion feature vector, wherein the multi-scale aligned fusion feature vector may be a feature vector obtained by fusing text features and image features of different scales.

[0067] In some optional implementations of some embodiments, the product text feature vector includes: a first product text feature vector, a second product text feature vector, and a third product text feature vector. The first product text feature vector, the second product text feature vector, and the third product text feature vector may be feature vectors output by the last three bidirectional attention encoding networks included in the text description feature extraction network.

[0068] Optionally, performing multi-scale feature alignment and fusion processing on the multimodal fusion feature vector to obtain a multi-scale aligned fusion feature vector may include the following steps:

[0069] In the first step, downsampling convolution is performed on the multimodal fusion feature vector to obtain a first multimodal fusion feature vector. The downsampling convolution may be a dimensionality reduction feature alignment process that uses a 1*1 convolution layer to halve the spatial resolution of the multimodal fusion feature vector but keeps the feature dimension unchanged.

[0070] The second step is to perform linear processing on the first product text feature vector to obtain the first product linear text feature vector. The linear processing may be a linear processing of inputting the first product text feature vector into a linear layer.

[0071] The third step is to perform multi-scale stage fusion processing on the first multimodal fusion feature vector and the first product linear text feature vector to obtain a first text image fusion feature vector. The multi-scale stage fusion can be a feature fusion performed through a simple gating mechanism.

[0072] The fourth step is to perform downsampling convolution processing on the first text image fusion feature vector to obtain a second text image fusion feature vector, wherein the spatial resolution of the second text image fusion feature vector is half of the first product linear text feature vector.

[0073] The fifth step is to perform linear processing on the second product text feature vector to obtain the second product linear text feature vector.

[0074] The sixth step is to perform multi-scale stage fusion processing on the second product linear text feature vector and the second text image fusion feature vector to obtain a third text image fusion feature vector.

[0075] In the seventh step, down-sampling convolution is performed on the third text image fusion feature vector to obtain a fourth text image fusion feature vector, wherein the spatial resolution of the fourth text image fusion feature vector may be half of that of the second text image fusion feature vector.

[0076] In the eighth step, linear processing is performed on the third product text feature vector to obtain the third product linear text feature vector.

[0077] In the ninth step, a multi-scale stage fusion process is performed on the third product linear text feature vector and the fourth text image fusion feature vector to obtain a fifth text image fusion feature vector.

[0078] In the tenth step, down-sampling convolution is performed on the fifth text image fusion feature vector to obtain a sixth text image fusion feature vector.

[0079] In the eleventh step, down-sampling is performed on the sixth text image fusion feature vector to obtain a seventh text image fusion feature vector.

[0080] In the twelfth step, multi-scale deconvolution up-sampling and splicing processing is performed on the seventh text image fusion feature vector to obtain a multi-scale aligned fusion feature vector.

[0081] In some optional implementations of some embodiments, performing multi-scale deconvolution up-sampling and splicing processing on the seventh text image fusion feature vector to obtain a multi-scale aligned fusion feature vector may include the following steps:

[0082] In the first step, the seventh text image fusion feature vector is upsampled to obtain a first upsampled fusion feature vector. The upsampling may be performed by using a 1*1 deconvolution layer to double the spatial resolution of the multimodal fusion feature vector, but the feature dimension remains unchanged.

[0083] In the second step, channel feature concatenation is performed on the sixth text image fusion feature vector and the first up-sampled fusion feature vector to obtain a second up-sampled fusion feature vector.

[0084] The third step is to perform deconvolution upsampling processing on the second up-sampled fused feature vector to obtain a third up-sampled fused feature vector.

[0085] The fourth step is to perform channel feature concatenation on the third upsampled fusion feature vector and the fourth text image fusion feature vector to obtain a fourth upsampled fusion feature vector. The third upsampled fusion feature vector and the fourth text image fusion feature vector have the same spatial resolution. The channel feature concatenation can be performed by copying the third upsampled fusion feature vector to the upsampled network layer through a skip connection structure so as to perform channel fusion with the fourth text image fusion feature vector.

[0086] In the fifth step, a deconvolution upsampling process is performed on the fourth up-sampled fused feature vector to obtain a fifth up-sampled fused feature vector.

[0087] In the sixth step, channel feature concatenation is performed on the fifth up-sampled fusion feature vector and the second text image fusion feature vector to obtain a sixth up-sampled fusion feature vector.

[0088] In the seventh step, a deconvolution upsampling process is performed on the sixth upsampled fusion feature vector to obtain a multi-scale aligned fusion feature vector.

[0089] In some optional implementations of some embodiments, the multi-scale stage fusion processing of the first multimodal fusion feature vector and the first product linear text feature vector to obtain the first text image fusion feature vector may include the following steps:

[0090] In the first step, channel convolution processing is performed on the first product linear text feature vector to obtain a product channel text feature vector. The channel convolution processing may be a processing of calculating the average value of the first product linear text feature vector along the channel direction through a 1*1 convolution layer.

[0091] In the second step, channel convolution is performed on the first multimodal fusion feature vector to obtain a multimodal fusion channel feature vector.

[0092] The third step is to perform feature flattening processing on the multimodal fusion channel feature vector to obtain the flattened multimodal fusion feature vector. Among them, the feature flattening processing can be a dimensionality reduction processing on the multimodal fusion channel feature vector, and only retain the number of channels. For example, the multimodal fusion channel feature vector is C*H*W, and it is C*HW after feature flattening processing. C represents the number of channels. H represents the length of the multimodal fusion channel feature. W represents the width of the multimodal fusion channel feature. The feature size of the product channel text feature vector and the flattened multimodal fusion feature vector is the same.

[0093] The fourth step is to perform channel feature splicing on the above-mentioned product text channel feature vector and the above-mentioned flattened multimodal fusion feature vector to obtain a spliced ​​image text feature vector.

[0094] The fifth step is to perform linear processing on the above-mentioned spliced ​​image text feature vector to obtain the image text linear feature vector.

[0095] Step 6: Perform feature mapping on the image text linear feature vector to obtain the image text weight feature vector. The feature mapping process may be inputting the image text linear feature vector into a Sigmoid activation function.

[0096] In the seventh step, linear processing is performed on the flattened multimodal fusion feature vector to obtain a multimodal fusion linear feature vector.

[0097] In the eighth step, feature mapping is performed on the multimodal fusion linear feature vector to obtain a multimodal fusion mapping feature vector.

[0098] In the ninth step, the image-text weight feature vector and the multimodal fusion mapping feature vector are multiplied element by element to obtain the image-text cross-modal feature vector.

[0099] In the tenth step, feature reconstruction is performed on the above-mentioned image-text cross-modal feature vector to obtain a first text-image fusion feature vector. The above-mentioned feature reconstruction can be the inverse process of the above-mentioned feature flattening process.

[0100] The above technical solution and its related contents, combined with step "step 109", as an inventive point of an embodiment of the present disclosure, solve the technical problem 2 mentioned in the background technology "due to the fusion feature vector of the attention coefficient generated only by a single text feature, the quality of the feature vector obtained by multi-scale feature alignment fusion is low, which further causes the image segmentation accuracy to be low and the integrity of the segmented mask product image to be low". The factors that lead to the low quality of the feature vector obtained by multimodal fusion again, further causing the image segmentation accuracy to be low and the integrity of the segmented mask product image to be low are often as follows: due to the fusion feature vector of the attention coefficient generated only by a single text feature. If the above factors are solved, the effect of improving the accuracy of vehicle-mounted data can be achieved. In order to achieve this effect, the present disclosure first performs channel convolution processing on the first product linear text feature vector, and performs channel convolution processing and flattening processing on the first multimodal fusion feature vector, which can reduce the dimensions of the text features and the multimodal fusion features, so that the scales of the text features and the multimodal fusion features are the same, and facilitate the subsequent feature splicing of the text features and the multimodal features. Then, the channel features of the pooled product text feature vector and the flattened multimodal fusion feature vector are spliced, which can further fuse the text features and image features, fully capture the correlation between text and image, and improve the semantic and visual richness of the fused feature vector. Finally, the spliced ​​image text feature vector and the multimodal fusion mapping feature vector are linearly mapped and reconstructed, and the text-guided image segmentation is performed based on the reconstructed feature vector. Through lightweight linear attention, the text features and the image features of the multimodal features after the initial fusion are used to generate the attention coefficient, which can further improve the richness of the first text-image fusion feature vector, improve the accuracy of image segmentation and the integrity of the segmented mask product image.

[0101] Step 106: Perform feature enhancement processing on the multi-scale aligned fusion feature vector and the product text feature vector to obtain a cross-modal enhanced feature vector.

[0102] In some embodiments, the execution subject may perform feature enhancement processing on the multi-scale aligned fusion feature vector and the product text feature vector to obtain a cross-modal enhanced feature vector, wherein the cross-modal enhanced feature vector may represent an image prediction mask.

[0103] In some optional implementations of some embodiments, the step of performing feature enhancement processing on the multi-scale aligned fusion feature vector and the product text feature vector to obtain a cross-modal enhanced feature vector may include the following steps:

[0104] In the first step, a first downsampling process is performed on the multi-scale aligned fusion feature vector to obtain a first multi-scale aligned fusion feature vector.

[0105] The second step is to perform a first feature normalization process on the first multi-scale aligned fusion feature vector to obtain a second multi-scale aligned fusion feature vector. In practice, the execution subject may input the first multi-scale aligned fusion feature vector into a 1*1 convolutional layer connected to a BatchNorm layer to obtain a second multi-scale aligned fusion feature vector.

[0106] The third step is to perform nonlinear activation processing on the second multi-scale aligned fusion feature vector to obtain a third multi-scale aligned fusion feature vector. The nonlinear activation processing may be a process of inputting the second multi-scale aligned fusion feature vector into a ReLu activation function.

[0107] In the fourth step, a second downsampling process is performed on the third multi-scale aligned fusion feature vector to obtain a fourth multi-scale aligned fusion feature vector.

[0108] In the fifth step, a second feature normalization process is performed on the fourth multi-scale aligned fusion feature vector to obtain a fifth multi-scale aligned fusion feature vector.

[0109] In the sixth step, feature convolution is performed on the fifth multi-scale aligned fusion feature vector to obtain a sixth multi-scale aligned fusion feature vector. The feature convolution may be performed by a 1*1 convolution layer.

[0110] In the seventh step, feature flattening is performed on the sixth multi-scale aligned fusion feature vector to obtain a seventh multi-scale aligned fusion feature vector.

[0111] In the eighth step, the product text feature vector is subjected to a linear transformation to obtain a text weight feature vector, wherein the text weight feature vector can represent the weight value of the product image description text information in guiding image segmentation.

[0112] In the ninth step, matrix multiplication is performed on the seventh multi-scale aligned fusion feature vector and the text weight feature vector to obtain a cross-modal mask feature vector.

[0113] In the tenth step, the feature dimension of the above cross-modal mask feature vector is reconstructed to obtain a reconstructed cross-modal mask feature vector.

[0114] In the eleventh step, nonlinear activation processing is performed on the above reconstructed cross-modal mask feature vector to obtain a first reconstructed cross-modal mask feature vector.

[0115] In the twelfth step, feature convolution is performed on the first reconstructed cross-modal mask feature vector to obtain a second reconstructed cross-modal mask feature vector as a cross-modal enhanced feature vector.

[0116] The above technical solution and its related contents, combined with step "step 109", as an inventive point of an embodiment of the present disclosure, solves the technical problem three mentioned in the background technology: "Since the decoder based on the attention mechanism can only fuse the high-level information and extract the correlation relationship of the extracted text features and image features, it cannot solve the problem of insufficient boundary information extraction and removal of noise data, resulting in low quality of the enhanced feature vector and low accuracy of image segmentation". The factors that lead to low quality of the enhanced feature vector and low accuracy of image segmentation are often as follows: Since the decoder based on the attention mechanism can only fuse the high-level information and extract the correlation relationship of the extracted text features and image features, it cannot solve the problem of insufficient boundary information extraction and removal of noise data. If the above factors are solved, the effect of improving the accuracy of vehicle-mounted data can be achieved. In order to achieve this effect, the present disclosure first performs multiple downsampling, normalization, convolution and flattening on the multi-scale aligned fusion feature vector, which can reduce the dimension of the multi-scale aligned fusion feature vector, extract edge and texture features in the multi-scale aligned fusion feature vector, improve the quality of the multi-scale aligned fusion feature vector, and the flattening process facilitates subsequent further fusion with text features. Then, the product text feature vector is linearly transformed to obtain the text weight feature vector, which can filter out the noise data in the image that is irrelevant to the text information of the product image description, enhance the features related to the target product image and the text information of the product image description, and reduce the feature dimension. Finally, the text weight feature vector and the seventh multi-scale alignment fusion feature vector are multiplied, re-entered and convolved, and the target product image is subjected to text-guided image segmentation based on the obtained cross-modal enhanced feature vector. The feature enhancement distillation decoder is used to further optimize and extract the features after cross-text and image visual fusion, thereby improving the quality of the cross-modal enhanced feature vector and the accuracy of text-guided image segmentation.

[0117] Step 107: linearize the product text feature vector to obtain a linearized product text feature vector.

[0118] In some embodiments, the execution subject may perform linear processing on the product text feature vector to obtain a linearized product text feature vector. The linearized product text feature vector may be a feature vector obtained by mapping the dimension of the product text feature vector to the space corresponding to the cross-modal enhancement feature vector. The linear processing may be inputting the product text feature vector into a linear layer for linear processing.

[0119] Step 108, performing vector multiplication on the cross-modal enhancement feature vector and the linearized product text feature vector to obtain a multimodal mask segmentation feature vector.

[0120] In some embodiments, the execution subject may perform vector multiplication on the cross-modal enhancement feature vector and the linearized product text feature vector to obtain a multimodal mask segmentation feature vector, wherein the multimodal mask segmentation feature vector may represent the segmentation result of each image pixel in the target product image.

[0121] Step 109 , performing image segmentation processing on the target product image according to the multimodal mask segmentation feature vector, targeting the product image description text information, to obtain a segmentation mask product image set.

[0122] In some embodiments, the execution subject may perform image segmentation processing on the target product image according to the multimodal mask segmentation feature vector, and obtain a segmentation mask product image set. The segmentation mask product image in the segmentation mask product image set may be a sub-image obtained by segmenting the product included in the target product image according to the guidance of the product image description text information.

[0123] As an example, the execution entity may input the multimodal mask segmentation feature vector into a fully connected layer to obtain a segmentation mask product image set for the product image description text information.

[0124] The above-mentioned embodiments of the present disclosure have the following beneficial effects: the multimodal image segmentation method of some embodiments of the present disclosure gives full play to the guiding role of text in image segmentation by multiple fusions of images and texts, and improves the accuracy of image segmentation and the integrity of the segmentation mask product image. Specifically, the reason for the low image segmentation accuracy is that the existing text-guided image segmentation models are mostly based on simple images (such as landscapes, characters, animals and plants, and medical images), lack of pertinence to product images, and the existing text-guided image segmentation models perform modal fusion through the attention mechanism, only perform modal fusion on high-level features, and take visual features as the leading fusion, lack of capturing the association between text and image, resulting in low image segmentation accuracy and low integrity of the segmentation mask product image, and increasing the number of image segmentation times. Based on this, the multimodal image segmentation method of some embodiments of the present disclosure can first obtain the target product image and the product image description text information, wherein the target product image and the product image description text information have a semantic association relationship. Here, the target product image and the image description text information are used for subsequent text and image feature extraction. Secondly, the target product image is input into the product image feature extraction network to obtain a product image feature vector set. Here, the key information in the target product image can be extracted to improve the detail information and high resolution of the product image feature vector. Again, the above product image description text information is input into the text description feature extraction network to obtain the product text feature vector. Here, the key semantic information in the product image description text information can be extracted to improve the quality of the product text feature vector and enrich the semantic information. Then, the above product image feature vector set and the above product text feature vector are subjected to multimodal feature fusion processing to obtain a multimodal fusion feature vector. Here, preliminary cross-modal interactive fusion is performed to extract richer semantic information while maintaining the high-resolution detail features of the image. Subsequently, the above multimodal fusion feature vector is subjected to multiscale feature alignment fusion processing to obtain a multiscale alignment fusion feature vector. Here, text image features of different levels and scales can be effectively integrated, and the detail information at high resolution and the semantic information at low resolution can be fully integrated to capture the association between text and image. Afterwards, the above multiscale alignment fusion feature vector and the above product text feature vector are subjected to feature enhancement processing to obtain a cross-modal enhanced feature vector. Here, noise can be removed and information loss problems can be solved, and key visual clues can be further extracted from the fused features. Then, the product text feature vector is linearized to obtain a linearized product text feature vector. Here, the product text feature vector is transformed in feature dimension so as to conform to the dimension of the cross-modal enhancement feature vector for vector multiplication and improve the guiding role of the text in image segmentation. Then, the cross-modal enhancement feature vector and the linearized product text feature vector are vector multiplied to obtain a multimodal mask segmentation feature vector.Here, the guiding role of text on image segmentation can be fully reflected. Finally, according to the multimodal mask segmentation feature vector, the target product image is subjected to image segmentation processing for the text information describing the product image, and a segmentation mask product image set is obtained. Here, the accuracy of image segmentation and the image integrity of the segmentation mask product image set can be improved. It can be concluded that the multimodal image segmentation method fully utilizes the guiding role of text on image segmentation through multiple fusions of images and texts, thereby improving the accuracy of image segmentation and the integrity of the segmentation mask product images.

[0125] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a multimodal image segmentation device. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the multimodal image segmentation device can be specifically applied to various electronic devices.

[0126] like Figure 2As shown, a multimodal image segmentation device 200 includes: an acquisition unit 201, a first input unit 202, a second input unit 203, a multimodal feature fusion unit 204, a multiscale feature alignment fusion unit 205, a feature enhancement unit 206, a linearization processing unit 207, a vector multiplication unit 208 and an image segmentation unit 209. The acquisition unit 201 is configured to: acquire a target product image and product image description text information, wherein the target product image and the product image description text information have a semantic association relationship. The first input unit 202 is configured to input the target product image into a product image feature extraction network to obtain a product image feature vector set. The second input unit 203 is configured to: input the product image description text information into a text description feature extraction network to obtain a product text feature vector. The multimodal feature fusion unit 204 is configured to: perform multimodal feature fusion processing on the product image feature vector set and the product text feature vector to obtain a multimodal fusion feature vector. The multi-scale feature alignment and fusion unit 205 is configured to: perform multi-scale feature alignment and fusion processing on the above-mentioned multi-modal fusion feature vector to obtain a multi-scale alignment and fusion feature vector. The feature enhancement unit 206 is configured to: perform feature enhancement processing on the above-mentioned multi-scale alignment and fusion feature vector and the above-mentioned product text feature vector to obtain a cross-modal enhanced feature vector. The linearization processing unit 207 is configured to: perform linearization processing on the above-mentioned product text feature vector to obtain a linearized product text feature vector. The vector multiplication unit 208 is configured to: perform vector multiplication on the above-mentioned cross-modal enhanced feature vector and the above-mentioned linearized product text feature vector to obtain a multi-modal mask segmentation feature vector. The image segmentation unit 209 is configured to: perform image segmentation processing on the above-mentioned target product image for the above-mentioned product image description text information according to the above-mentioned multi-modal mask segmentation feature vector to obtain a segmentation mask product image set.

[0127] It can be understood that the units described in the multimodal image segmentation device 200 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the multimodal image segmentation device 200 and the units included therein, and will not be described in detail here.

[0128] Reference below Figure 3 , which shows a structural schematic diagram of an electronic device (eg, an electronic device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0129] like Figure 3As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 to a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0130] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0131] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0132] It should be noted that the computer-readable medium mentioned above in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0133] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0134] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a target product image and product image description text information, wherein the target product image and the product image description text information have a semantic association relationship; inputs the target product image into a product image feature extraction network to obtain a product image feature vector set; inputs the product image description text information into a text description feature extraction network to obtain a product text feature vector; performs multimodal feature fusion processing on the product image feature vector set and the product text feature vector to obtain a multimodal fusion feature vector; The multimodal fusion feature vector is subjected to multiscale feature alignment and fusion processing to obtain a multiscale alignment and fusion feature vector; the multiscale alignment and fusion feature vector and the product text feature vector are subjected to feature enhancement processing to obtain a cross-modal enhancement feature vector; the product text feature vector is subjected to linearization processing to obtain a linearized product text feature vector; the cross-modal enhancement feature vector and the linearized product text feature vector are subjected to vector multiplication to obtain a multimodal mask segmentation feature vector; according to the multimodal mask segmentation feature vector, the target product image is subjected to image segmentation processing for the text information describing the product image to obtain a segmentation mask product image set.

[0135] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0136] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0137] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor including an acquisition unit, a first input unit, a second input unit, a multimodal feature fusion unit, a multi-scale feature alignment fusion unit, a feature enhancement unit, a linearization processing unit, a vector multiplication unit, and an image segmentation unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the acquisition unit may also be described as a "unit for acquiring target product images and product image description text information".

[0138] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0139] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A multimodal image segmentation method, comprising: Acquire a target product image and product image description text information, wherein the target product image and the product image description text information have a semantic association relationship; Inputting the target product image into a product image feature extraction network to obtain a product image feature vector set; Inputting the product image description text information into a text description feature extraction network to obtain a product text feature vector; Performing multimodal feature fusion processing on the product image feature vector set and the product text feature vector to obtain a multimodal fusion feature vector; Performing multi-scale feature alignment and fusion processing on the multimodal fusion feature vector to obtain a multi-scale aligned fusion feature vector; Performing feature enhancement processing on the multi-scale aligned fusion feature vector and the product text feature vector to obtain a cross-modal enhanced feature vector, wherein the performing feature enhancement processing on the multi-scale aligned fusion feature vector and the product text feature vector to obtain a cross-modal enhanced feature vector includes: Performing a first downsampling process on the multi-scale aligned fused feature vector to obtain a first multi-scale aligned fused feature vector; Performing a first feature normalization process on the first multi-scale aligned fusion feature vector to obtain a second multi-scale aligned fusion feature vector; Performing nonlinear activation processing on the second multi-scale aligned fusion feature vector to obtain a third multi-scale aligned fusion feature vector; Performing a second downsampling process on the third multi-scale aligned fused feature vector to obtain a fourth multi-scale aligned fused feature vector; Performing a second feature normalization process on the fourth multi-scale aligned fusion feature vector to obtain a fifth multi-scale aligned fusion feature vector; Performing feature convolution processing on the fifth multi-scale aligned fusion feature vector to obtain a sixth multi-scale aligned fusion feature vector; Performing feature flattening processing on the sixth multi-scale aligned fusion feature vector to obtain a seventh multi-scale aligned fusion feature vector; Performing linear transformation processing on the product text feature vector to obtain a text weight feature vector; Performing matrix multiplication processing on the seventh multi-scale aligned fusion feature vector and the text weight feature vector to obtain a cross-modal mask feature vector; Performing feature dimension reconstruction processing on the cross-modal mask feature vector to obtain a reconstructed cross-modal mask feature vector; Performing nonlinear activation processing on the reconstructed cross-modal mask feature vector to obtain a first reconstructed cross-modal mask feature vector; Performing feature convolution processing on the first reconstructed cross-modal mask feature vector to obtain a second reconstructed cross-modal mask feature vector as a cross-modal enhanced feature vector; Performing linearization processing on the product text feature vector to obtain a linearized product text feature vector; Performing vector multiplication on the cross-modal enhancement feature vector and the linearized product text feature vector to obtain a multimodal mask segmentation feature vector; According to the multimodal mask segmentation feature vector, the target product image is subjected to image segmentation processing for the product image description text information to obtain a segmentation mask product image set.

2. The method according to claim 1, wherein: The product image feature vector set includes: a first product image feature vector, a second product image feature vector and a third product image feature vector; and The performing multimodal feature fusion processing on the product image feature vector set and the product text feature vector to obtain a multimodal fusion feature vector includes: Performing channel convolution processing on the first product image feature vector to obtain a first product image channel feature vector; Performing average pooling downsampling processing on the first product image channel feature vector to obtain a first product image downsampled feature vector; Performing channel convolution processing on the second product image feature vector to obtain a second product image channel feature vector; Performing channel convolution processing on the third product image feature vector to obtain a third product image channel feature vector; Performing bilinear interpolation upsampling processing on the third product image channel feature vector to obtain a third product image upsampled feature vector; Performing channel feature splicing on the first product image down-sampling feature vector, the second product image channel feature vector, the third product image up-sampling feature vector, and a preset visual position coding feature vector to obtain an image channel splicing feature vector; Performing channel convolution processing on the product text feature vector to obtain a product channel text feature vector; The image channel splicing feature vector and the product channel text feature vector are processed by Kronecker product to obtain a multimodal fusion feature vector.

3. The method according to claim 1, wherein: The product text feature vector includes: a first product text feature vector, a second product text feature vector and a third product text feature vector; and The performing multi-scale feature alignment and fusion processing on the multi-modal fusion feature vector to obtain a multi-scale aligned fusion feature vector includes: Performing downsampling convolution processing on the multimodal fusion feature vector to obtain a first multimodal fusion feature vector; Performing linear processing on the first product text feature vector to obtain a first product linear text feature vector; Performing multi-scale stage fusion processing on the first multimodal fusion feature vector and the first product linear text feature vector to obtain a first text image fusion feature vector; Performing downsampling convolution processing on the first text image fusion feature vector to obtain a second text image fusion feature vector; Performing linear processing on the second product text feature vector to obtain a second product linear text feature vector; Performing multi-scale stage fusion processing on the second product linear text feature vector and the second text image fusion feature vector to obtain a third text image fusion feature vector; Performing downsampling convolution processing on the third text image fusion feature vector to obtain a fourth text image fusion feature vector; Performing linear processing on the third product text feature vector to obtain a third product linear text feature vector; Performing multi-scale stage fusion processing on the third product linear text feature vector and the fourth text image fusion feature vector to obtain a fifth text image fusion feature vector; Performing downsampling convolution processing on the fifth text image fusion feature vector to obtain a sixth text image fusion feature vector; Downsampling the sixth text image fusion feature vector to obtain a seventh text image fusion feature vector; The seventh text image fusion feature vector is subjected to multi-scale deconvolution up-sampling and splicing processing to obtain a multi-scale aligned fusion feature vector.

4. The method according to claim 3, wherein: The step of performing multi-scale deconvolution up-sampling and splicing processing on the seventh text image fusion feature vector to obtain a multi-scale aligned fusion feature vector includes: Performing upsampling processing on the seventh text image fusion feature vector to obtain a first upsampling fusion feature vector; Perform channel feature concatenation on the sixth text image fusion feature vector and the first up-sampled fusion feature vector to obtain a second up-sampled fusion feature vector; Performing a deconvolution upsampling process on the second up-sampled fused feature vector to obtain a third up-sampled fused feature vector; Performing channel feature concatenation on the third up-sampled fused feature vector and the fourth text image fused feature vector to obtain a fourth up-sampled fused feature vector; Performing a deconvolution upsampling process on the fourth up-sampled fused feature vector to obtain a fifth up-sampled fused feature vector; Perform channel feature concatenation on the fifth up-sampled fused feature vector and the second text image fused feature vector to obtain a sixth up-sampled fused feature vector; The sixth up-sampled fused feature vector is subjected to a deconvolution up-sampling process to obtain a multi-scale aligned fused feature vector.

5. The method according to claim 1, wherein: The product image feature extraction network includes: a convolutional network, a global maximum pooling layer, a first-stage convolutional residual network, a second-stage convolutional residual network, a third-stage convolutional residual network, and a fourth-stage convolutional residual network; and The step of inputting the target product image into a product image feature extraction network to obtain a product image feature vector set includes: Performing convolution processing on the target product image through the convolution network to obtain a first image feature vector; Performing fuzzy filtering on the first image feature vector to obtain a filtered first image feature vector; Performing global maximum pooling processing on the filtered first image feature vector through the global maximum pooling layer to obtain a second image feature vector; Inputting the second image feature vector into the first-stage convolutional residual network to obtain a third image feature vector; Inputting the third image feature vector into the second-stage convolutional residual network to obtain a fourth image feature vector; Inputting the fourth image feature vector into the third-stage convolutional residual network to obtain a fifth image feature vector; Inputting the fifth image feature vector into the fourth-stage convolutional residual network to obtain a sixth image feature vector; The fourth image feature vector, the fifth image feature vector and the sixth image feature vector are determined as a product image feature vector set.

6. The method according to claim 1, wherein: The text description feature extraction network includes: a plurality of bidirectional attention encoding networks; and The step of inputting the product image description text information into a text description feature extraction network to obtain a product text feature vector includes: Performing character segmentation processing on the product image description text information to obtain a product text character set; Performing character encoding processing on each product text character in the product text character set to obtain a product text character feature vector set; Based on the product text character feature vector set, perform the following text feature extraction steps: Inputting the product text character feature vector set and the preset text position encoding feature vector set into the multi-head attention mechanism layer included in the bidirectional attention encoding network to obtain the product text character weight feature vector set, wherein the bidirectional attention encoding network further includes: a multi-head attention mechanism layer, a residual connection and normalization layer, and a feedforward neural network layer; Input the product text character weight feature vector set into the multi-head attention mechanism layer to obtain the product text character attention feature vector set; Input the product text character attention feature vector set into the residual connection and normalization layer to obtain the product text character normalized feature vector set; Input the normalized feature vector set of product text characters into the feedforward neural network layer to obtain a nonlinear feature vector set of product text; In response to determining that the bidirectional attention encoding network is a terminating bidirectional attention encoding network, a plurality of product text nonlinear feature vector sets output by a preset number of bidirectional attention encoding networks adjacent to the terminating bidirectional attention encoding network are determined as product text feature vectors, wherein the terminating bidirectional attention encoding network is a bidirectional attention encoding network at a terminating position of the plurality of bidirectional attention encoding networks; In response to determining that the bidirectional attention encoding network is not a terminating bidirectional attention encoding network, the bidirectional attention encoding network located after the bidirectional attention encoding network is determined to be a bidirectional attention encoding network to perform the text feature extraction step again.

7. A multimodal image segmentation device, comprising: An acquisition unit is configured to acquire a target product image and product image description text information, wherein the target product image and the product image description text information have a semantic association relationship; A first input unit is configured to input the target product image into a product image feature extraction network to obtain a product image feature vector set; A second input unit is configured to input the product image description text information into a text description feature extraction network to obtain a product text feature vector; a multimodal feature fusion unit configured to perform multimodal feature fusion processing on the product image feature vector set and the product text feature vector to obtain a multimodal fusion feature vector; A multi-scale feature alignment and fusion unit is configured to perform a multi-scale feature alignment and fusion process on the multi-modal fusion feature vector to obtain a multi-scale alignment and fusion feature vector; The feature enhancement unit is configured to perform feature enhancement processing on the multiscale aligned fusion feature vector and the product text feature vector to obtain a cross-modal enhanced feature vector, wherein the feature enhancement processing on the multiscale aligned fusion feature vector and the product text feature vector to obtain a cross-modal enhanced feature vector includes: performing a first downsampling processing on the multiscale aligned fusion feature vector to obtain a first multiscale aligned fusion feature vector; performing a first feature normalization processing on the first multiscale aligned fusion feature vector to obtain a second multiscale aligned fusion feature vector; performing a nonlinear activation processing on the second multiscale aligned fusion feature vector to obtain a third multiscale aligned fusion feature vector; performing a second downsampling processing on the third multiscale aligned fusion feature vector to obtain a fourth multiscale aligned fusion feature vector; performing a second feature normalization processing on the fourth multiscale aligned fusion feature vector to obtain a five multi-scale aligned fusion feature vectors; perform feature convolution processing on the fifth multi-scale aligned fusion feature vector to obtain a sixth multi-scale aligned fusion feature vector; perform feature flattening processing on the sixth multi-scale aligned fusion feature vector to obtain a seventh multi-scale aligned fusion feature vector; perform linear transformation processing on the product text feature vector to obtain a text weight feature vector; perform matrix multiplication processing on the seventh multi-scale aligned fusion feature vector and the text weight feature vector to obtain a cross-modal mask feature vector; perform feature dimension reconstruction processing on the cross-modal mask feature vector to obtain a reconstructed cross-modal mask feature vector; perform nonlinear activation processing on the reconstructed cross-modal mask feature vector to obtain a first reconstructed cross-modal mask feature vector; perform feature convolution processing on the first reconstructed cross-modal mask feature vector to obtain a second reconstructed cross-modal mask feature vector as a cross-modal enhanced feature vector; A linearization processing unit, configured to perform linearization processing on the product text feature vector to obtain a linearized product text feature vector; A vector multiplication unit, configured to perform vector multiplication on the cross-modal enhancement feature vector and the linearized product text feature vector to obtain a multimodal mask segmentation feature vector; The image segmentation unit is configured to perform image segmentation processing on the target product image according to the multimodal mask segmentation feature vector for the product image description text information to obtain a segmentation mask product image set.

8. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Binocular image self-restoration network model training method

    CN114022381A

  • Anaphora image segmentation method and device based on basic model, and storage medium

    CN117877029A