Image division method, device, equipment, and storage medium

Abstract

The present disclosure provides an image segmentation method, apparatus, device, and storage medium, which includes: fusing visual features corresponding to an original image and text features corresponding to a description language for specifying a target object to be segmented in the original image to obtain multimodal features; determining visual regions of the target object based on the image corresponding to the multimodal features, and representing the image corresponding to the visual regions as a response heat map; and determining a segmentation result of the target object based on the image corresponding to the multimodal features and the response heat map.

Description

[Technical field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims priority to a Chinese patent application bearing application number 202011197790.9, filed with the China Patent Office on October 30, 2020, the entire contents of which are incorporated herein by reference.

[0002] [Technical field] The present disclosure relates to the technical field of image processing, for example, to an image segmentation method, device, apparatus, and storage medium. [Background technology]

[0003] Language-directed image segmentation is a very important technique in cross-modal learning, also called directed image segmentation, in which the goal is to segment linguistically specified objects in an image. Language-directed image segmentation is more challenging because it needs to overcome the problem of the semantic gap between the image and the linguistic description. Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides an image segmentation method, device, apparatus, and storage medium that can effectively segment a specified object in an image according to instructions in a description language. [Means for solving the problem]

[0005] The present disclosure relates to fusing visual features corresponding to an original image and text features corresponding to a description language for specifying a target object to be segmented in the original image to obtain multimodal features; determining visual regions of the target object based on images corresponding to the multi-modal features and notating the images corresponding to the visual regions as a response heat map; determining a segmentation result of the target object based on an image corresponding to the multi-modal features and the response heat map. A method for segmenting an image is provided.

[0006] The present disclosure relates to a fusion module configured to fuse visual features corresponding to an original image and text features corresponding to a description language for specifying a target object to be segmented in the original image to obtain multi-modal features; a visual region determination module configured to determine visual regions of the target object based on images corresponding to the multi-modal features and to mark the images corresponding to the visual regions as a response heat map; and a segmentation result determination module configured to determine a segmentation result of the target object based on an image corresponding to the multi-modal features and the response heat map. An image segmentation device is also provided.

[0007] The present disclosure relates to one or more processors; a memory configured to store one or more programs; An electronic device comprising: When the one or more programs are executed by the one or more processors, the image segmentation method is realized. An electronic device is also provided.

[0008] The present disclosure further provides a computer-readable storage medium having a computer program stored thereon, the computer-readable storage medium implementing the image segmentation method when the program is executed by a processor. [Brief description of the drawings]

[0009] [Figure 1] 1 is a flowchart of an image division method according to a first embodiment of the present disclosure. [Diagram 2] 11 is a flowchart of an image division method according to a second embodiment of the present disclosure. [Diagram 3] FIG. 11 is a structural schematic diagram of an image segmentation model according to a second embodiment of the present disclosure. [Figure 4] 11 is a flowchart illustrating an implementation of an image segmentation method according to a second embodiment of the present disclosure. [Diagram 5] FIG. 11 is a schematic diagram of an original image according to a second embodiment of the present disclosure. [Figure 6] FIG. 11 is a schematic diagram of a division result according to the second embodiment of the present disclosure. [Figure 7] FIG. 1 is a schematic diagram of a segmentation result obtained by a conventional method. [Figure 8] FIG. 11 is a comparison diagram of results of segmenting the same image using an image segmentation method according to an embodiment of the present disclosure and a related technique according to a second embodiment of the present disclosure. [Figure 9] FIG. 11 is a structural diagram of an image segmentation device according to a third embodiment of the present disclosure. [Figure 10] FIG. 11 is a structural diagram of an electronic device according to a fourth embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Although several embodiments of the present disclosure are shown in the drawings, the present disclosure can be realized in various forms, and should not be construed as being limited to the embodiments described herein, but rather, these embodiments are provided for a clearer and more complete understanding of the present disclosure.

[0011] Steps described in method embodiments of the present disclosure may be performed in different orders and / or in parallel. Method embodiments may also include additional steps and / or steps shown omitted from execution. The scope of the present disclosure is not limited in this respect.

[0012] As used herein, the term "comprising" and variations thereof are open inclusive, i.e., "including, but not limited to." The term "based on" means "based at least in part on." The term "one embodiment" refers to "at least one embodiment." The term "another embodiment" refers to "at least one other embodiment." The term "some embodiments" refers to "at least some embodiments." Relevant definitions of other terms are provided below in the description.

[0013] It should be noted that the concepts of "first", "second", etc. referred to in this disclosure are merely intended to distinguish different devices, modules, units or operations, and are not intended to limit the order or interdependence of functions performed by these devices, modules, units or operations.

[0014] It should be noted that the modifications "a" and "a plurality" referred to in this disclosure are exemplary and not limiting, and should be understood as "one or more" unless the context clearly indicates otherwise.

[0015] The names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of these messages or information. EXAMPLES

[0016] FIG. 1 is a flowchart of an image segmentation method according to a first embodiment of the present disclosure, which can be applied to segmenting an object in an image with a language instruction, and can be applied to fields such as image editing during interaction or man-machine interaction, and can also be applied to fields such as language-driven image object detection or language-driven image understanding. The method can be executed by an image segmentation device, which can be realized in the form of software and / or hardware, and which can be configured as an electronic device, which can be a terminal having an image data processing function, for example, a mobile terminal such as a mobile phone, a tablet computer, or a notebook computer, or a fixed terminal such as a desktop computer or a server. As shown in FIG. 1, the method includes the following steps.

[0017] In S110, multi-modal features are obtained by fusing visual features corresponding to an original image and text features corresponding to a description language for specifying target objects to be segmented in the original image.

[0018] The original image may be an image including at least one object, and may be acquired by a method such as a camera or a scanner, or may be selected from an existing image library. The visual feature may be an image feature that the original image corresponds to at a set resolution, and the visual feature is actually an image, and the embodiment does not limit the size of the resolution. In one example, the image feature at the set resolution of the original image can be extracted through a visual feature extraction network to obtain a corresponding visual feature image. The visual feature extraction network may be a Darknet network structure or other network structure that can be used to extract visual features. Darknet is an open source deep learning framework, which has a simple structure, no dependencies, and is more flexible when used to extract visual features.

[0019] The description language may be a text corresponding to a descriptive language description, and is used to specify the target object to be divided in the original image. The description language may include image information, position information of the target object, and appearance information of the target object, etc. The image information is used to determine the information of the image that needs to be divided, and may include information that uniquely identifies the image, such as the name or number of the image. The position information is used to initially determine the position of the target object in the original image. The appearance information is used to determine the final target object. For example, the description language may be a user holding a badminton racket in his hand and wearing red clothes in image A. The text feature may be a feature that reflects the meaning of the description language, and can generally be expressed in the form of a vector. Preferably, the text feature of the description language can be extracted through a text feature extraction network, and the present embodiment is not limited on the structure of the text feature extraction network, and for example, a gated recurrent unit (GRU) network can be adopted, where GRU is a kind of recurrent neural network.

[0020] A multimodal feature is a fusion feature obtained by fusing features of multiple modalities, and this embodiment fuses visual features and text features to obtain multimodal features, realizes cross-modal feature display, and eliminates the semantic gap between images and description languages. Preferably, a cross product is performed between the visual feature and the position data corresponding to the text feature to obtain a multimodal feature, which is a fusion feature of the visual feature and the text feature. A multimodal feature is also actually an image, and a multimodal feature may be called a multimodal feature image or an image corresponding to a multimodal feature, i.e., an image including a multimodal feature.

[0021] At S120, visual regions of the target object are determined based on images corresponding to the multi-modal features, and the images corresponding to the visual regions are noted as a response heat map.

[0022] The visual area is the area where the target object is located, for example, if the target object is a pizza, the visual area is the area where the pizza is located. In this embodiment, when dividing the target object, the visual area of ​​the target object is first determined, and the target object is divided based on the visual area, so as to effectively reduce the complexity of image division. Preferably, the image corresponding to the multimodal feature is filtered to remove the interference of non-target objects with the target object, and the visual area of ​​the target object is obtained. In this embodiment, the filtered image, i.e., the image corresponding to the visual area, is recorded as a response heat map, which represents the position information of the target object, and has one response value corresponding to each area, and the larger the response value, the higher the possibility that the target object exists in the area. Preferably, the area with a response value larger than a set threshold can be the visual area of ​​the target object and highlighted, and this embodiment does not limit the size of the set threshold.

[0023] At S130, a segmentation result of the target object is determined based on the image corresponding to the multi-modal features and the response heat map.

[0024] Preferably, the image segmentation model can determine the segmentation result by combining the image and the response heat map corresponding to the multimodal features. The image segmentation model is used to determine the segmentation result of the target object, and the structure of the model can be set as necessary, for example, it may include a convolution layer and an upsampling layer, the convolution layer is used to perform a convolution operation on the input image, the upsampling layer is used to upsample the convolution result to obtain the segmentation result, and the size of the image corresponding to the segmentation result is the same as the size of the real segmentation result in the original image. The image segmentation model of this embodiment takes the image and the response heat map corresponding to the multimodal features as input, and before use, the image and the response heat map corresponding to the multimodal features are input into the image segmentation model, the image segmentation model is trained, and the parameters of the convolution layer and the upsampling layer can be adjusted. Preferably, the loss value of the segmentation result output from the image segmentation model for the real segmentation result corresponding to the original image can be determined, and if the loss value is smaller than a set threshold, the training is terminated, and the model corresponding to the case where the loss value is smaller than the set threshold is the image segmentation model for segmenting the target object of this embodiment.

[0025] The first embodiment of the present disclosure provides an image segmentation method, which obtains multi-modal features by fusing visual features corresponding to an original image with text features corresponding to a description language for specifying a target object to be segmented in the original image, determines a visual region of the target object according to the image corresponding to the multi-modal features, records the image corresponding to the visual region as a response heat map, and determines a segmentation result of the target object according to the image corresponding to the multi-modal features and the response heat map. The method decomposes the image segmentation process, firstly determines a visual region of the target object according to the image corresponding to the multi-modal features obtained by fusion, obtains a response heat map, and then determines a segmentation result according to the image corresponding to the multi-modal features and the response heat map, effectively bridging the semantic gap between the image and the description language, and segments the target object specified in the description language. EXAMPLES

[0026] FIG. 2 is a flowchart of an image segmentation method according to a second embodiment of the present disclosure, which is described based on the above embodiments. Referring to FIG. 2, the method may include the following steps:

[0027] In S210, visual features corresponding to the original image and text features corresponding to the description language are fused to obtain multi-modal features.

[0028] The number of visual features may be one or more, and may include visual features extracted from the original image at at least two resolutions respectively to improve the accuracy of the segmentation result. If the number of visual features is too small, it will affect the accuracy of the segmentation result, and if it is too large, it will increase the amount of calculation. In this embodiment, three visual features are taken as an example, which can improve the accuracy of the segmentation result and reduce the amount of calculation. The visual features in this embodiment are extracted from the original image (R H×W×3 ) first visual feature extracted at the first resolution

number

number

number

[0029] In one example, multi-modal features can be obtained in the following manner.

[0030] sorting the at least two visual features in order of resolution size, obtaining a sorted result; mapping the text feature to a feature space where a first visual feature corresponding to a first resolution having the smallest value in the sorted result is located using a mapping matrix; stitching the first visual feature and the mapped text feature to obtain a first stitching feature; up-sampling the first stitching feature; stitching the up-sampled first stitching feature with a second visual feature corresponding to a second resolution in the sorted result that is larger than the first resolution and is the smallest resolution other than the first resolution; obtaining a second stitching feature; stitching the up-sampled stitching feature with the visual feature having the largest resolution in the sorted result; and cyclically performing the up-sampling and stitching operations until a multi-modal feature is obtained.

[0031] Considering that the lengths of the text features and the visual features are different, in order to ensure the fusion effect, the text features and the visual features may be first mapped to the same feature space. For example, the visual features may be mapped to the space where the text features are located, the text features may be mapped to the space where the visual features are located, or the text features and the visual features may be mapped to other feature spaces. The embodiment takes the mapping of the text features to the feature space where the visual features are located as an example, which can simplify the mapping process and reduce the amount of calculation.

[0032] When there are multiple visual features, the corresponding resolutions are different, and the corresponding image sizes are different. In order to ensure the effectiveness of stitching, the embodiment sorts the multiple visual features in ascending order of resolution to obtain a sorting result. When stitching features, the process is performed in ascending order of resolution of the visual features in the sorting result, that is, the visual feature with the smallest resolution is stitched with the text feature first, and then the stitching result is upsampled and stitched with the visual feature corresponding to the next resolution in the sorting result, until the stitching of the visual feature with the largest resolution is completed.

[0033] For example, the visual features include a first visual feature, a second visual feature, and a third visual feature, and the first visual feature, the second visual feature, and the third visual feature correspond to a first resolution, a second resolution, and a third resolution, respectively, where the first resolution<the second resolution<the third resolution. The position value corresponding to the first visual feature and the mapped text feature can be calculated by a cross product method, and the stitching between the first visual feature and the mapped text feature can be realized to obtain a first stitching feature. As mentioned above, the resolution of the first visual feature is < the resolution of the second visual feature < the resolution of the third visual feature, that is, the resolution of the first stitching feature is smaller than that of the second visual feature. In order to realize the stitching of features with the same resolution, this embodiment upsamples the first stitching feature, and makes the resolution of the stitching feature obtained by upsampling the same as that of the second visual feature, and then performs a stitching operation similar to the first stitching feature, stitching the second visual feature and the first stitching feature after upsampling to obtain a second stitching feature, and then upsamples the second stitching feature, stitching the second stitching feature and the third visual feature after upsampling to obtain a third stitching feature, that is, obtaining a multi-modal feature. This embodiment fuses text features and visual features to realize cross-modal representation of features, and can improve the accuracy of the segmentation result when the target object is segmented later.

[0034] At S220, correlation filtering is performed on the image corresponding to the multi-modal features to obtain the visual region of the target object.

[0035] In one example, a convolution kernel is generated based on the text features, and a convolution operation is performed on the image corresponding to the multi-modal features based on the convolution kernel, so as to realize correlation filtering for the multi-modal feature image, and a response value corresponding to each region can be obtained, and the larger the response value of the region, the greater the probability that the target object exists. According to the response value, the corresponding visual region can be obtained.

[0036] At S230, the images corresponding to the visual regions are recorded as a response heatmap.

[0037] In S240, the image corresponding to the multi-modal features and the response heat map are input into a target image segmentation model, and the output result from the target image segmentation model is obtained as the segmentation result of the target object.

[0038] In this embodiment, an initial image segmentation model is designed based on the visual region to obtain a more accurate segmentation result. For example, refer to FIG. 3, which is a structural schematic diagram of an initial image segmentation model according to the second embodiment of the present disclosure, in which the image segmentation model includes an input layer, a parallel first convolution layer, a stitching layer, a second convolution layer, an upsampling layer, and an output layer, and the number of the first convolution layers can be set according to circumstances. FIG. 3 takes five first convolution layers as an example, which can better capture the content of the image at different scales. Each first convolution layer corresponds to one sampling rate, that is, the first convolution operation of five different sampling rates is respectively performed on the input image to obtain five convolution results. The stitching layer is used to stitch the five convolution results. The second convolution layer is used to perform the convolution operation again on the stitched result. The upsampling layer is used to ensure that the resolution of the segmentation result output from the image segmentation model matches the resolution of the real segmentation result of the original image.

[0039] Before using the image segmentation model, the parameters of the first convolution layer, stitching layer, second convolution layer and upsampling layer in the initial image segmentation model can be trained to obtain a target image segmentation model. The training process is as follows: The method includes obtaining a sample image and a sample description language, extracting sample visual features of the sample image and sample text features of the sample description language, fusing the sample visual features and the sample text features to obtain sample multi-modal features, determining a sample visual region of the sample target object based on an image corresponding to the sample multi-modal features, notating the image corresponding to the sample visual region as a sample response heat map, training an initial image segmentation model based on the image corresponding to the sample multi-modal features and the sample response heat map, and obtaining the target image segmentation model.

[0040] In this embodiment, the number of sample images and sample description languages ​​is not limited, and in order to improve the accuracy of the image segmentation model, multiple groups of sample images and multiple groups of sample description languages ​​are selected, and then the sample visual features of the sample images and the sample text features of the sample description languages ​​are extracted and stitched to obtain sample multi-modal features, and correlation filtering is performed on the multi-modal features to obtain sample response heat maps, so that an initial image segmentation model can be trained according to the images and sample response heat maps corresponding to the sample multi-modal features, and a target image segmentation model can be obtained. The feature extraction, stitching and filtering processes can refer to the above embodiments, and the description is omitted here.

[0041] According to the structure shown in Figure 3, input the image corresponding to the sample multi-modal feature and the sample response heat map into the initial image segmentation model, obtain a plurality of first convolution results of the image corresponding to the sample multi-modal feature and the sample response heat map, stitch the plurality of first convolution results, obtain the stitching result, perform a second convolution operation on the stitching result, obtain a second convolution result, upsample the second convolution result, obtain the sample segmentation result, determine the loss value of the sample segmentation result for the real segmentation result of the sample image, if the loss value is smaller than the set threshold, stop training the initial image segmentation model, and take the image segmentation model whose loss value is smaller than the set threshold as the target image segmentation model, if the loss value is equal to or larger than the set threshold, continue to train the initial image segmentation model until the loss value is smaller than the set threshold. Preferably, the loss value of the sample segmentation result for the real segmentation result of the sample image can be determined by the following loss function:

number

[0042] For example, refer to FIG. 4, which is a flowchart of the implementation of the image segmentation method according to the second embodiment of the present disclosure. First, obtain the original image and the description language, and then extract the visual features of the original image at different levels and the text features corresponding to the description language. FIG. 4 takes three levels as an example, corresponding to three resolutions respectively. Then, extract the first visual feature F v1 The first stitching feature F is obtained by stitching the text features after mapping. m1Then, the first stitching feature F m1 After upsampling, the second visual feature F v2 and stitching the second stitching feature F m2 and obtain the second stitching feature F m2 After upsampling, the third visual feature F v3 and stitching the third stitching feature F m3 Then, the multimodal features F m3 Correlation filtering is performed on the ,response heatmap, and the response heatmap and the multimodal feature F m3 By inputting the image corresponding to the target object into the target image segmentation model, the segmentation result of the target object can be obtained, which is simple and effective.

[0043] Illustratively, refer to FIG. 5, which is a schematic diagram of an original image according to the second embodiment of the present disclosure. Assuming that the language text is "Pizza Nearest", that is, the pizza with the closest distance is divided, the division result shown in FIG. 6 can be obtained according to the image division method according to the above embodiment. The division result obtained by the conventional method is as shown in FIG. 7. Illustratively, refer to FIG. 8, which is a comparison diagram of the results of dividing the same image using the image division method according to the second embodiment of the present disclosure and the related technology according to the second embodiment of the present disclosure. Here, the first column is three kinds of original images, the second column is an object divided using the method according to the embodiment of the present disclosure, and the third and fourth columns are objects divided using the related technology. As can be seen from FIGS. 6 and 8, the object divided using the image division method according to the embodiment of the present disclosure is closer to the real result, and the accuracy of the image division result is improved.

[0044] Example 2 of the present disclosure provides an image segmentation method, which decomposes the image segmentation process based on the above example, by first determining the visual area of ​​the target object initially, then constructing an initial image segmentation model, simplifying the complexity of the initial image segmentation model, training the initial image segmentation model using a multi-modal feature image and a response heat map, obtaining a target image segmentation model, and further using the target image segmentation model to obtain the segmentation result, thereby effectively bridging the semantic gap between the image and the description language, and improving the accuracy of the segmentation result to a certain extent. EXAMPLES

[0045] FIG. 9 is a structural diagram of an image segmentation device according to a third embodiment of the present disclosure, which can execute the image segmentation method described in the above embodiments, and which can be integrated into an electronic device. See FIG. 9, which shows the structure of the image segmentation device according to the third embodiment of the present disclosure. The system may include a fusion module 31 configured to obtain multimodal features by fusing visual features corresponding to an original image and text features corresponding to a description language for specifying a target object to be segmented in the original image, a visual area determination module 32 configured to determine a visual area of ​​the target object based on an image corresponding to the multimodal features and to record the image corresponding to the visual area as a response heat map, and a segmentation result determination module 33 configured to determine a segmentation result of the target object based on the image corresponding to the multimodal features and the response heat map.

[0046] An embodiment of the present disclosure provides an image segmentation device, which obtains multi-modal features by fusing visual features corresponding to an original image with text features corresponding to a description language for specifying a target object to be segmented in the original image, determines a visual region of the target object according to the image corresponding to the multi-modal features, records the image corresponding to the visual region as a response heat map, and determines a segmentation result of the target object according to the image corresponding to the multi-modal features and the response heat map. The device decomposes the image segmentation process, firstly determines a visual region of the target object according to the image corresponding to the multi-modal features obtained by fusion, obtains a response heat map, and then determines a segmentation result according to the image corresponding to the multi-modal features and the response heat map, effectively bridging the semantic gap between the image and the description language, and segments the target object specified in the description language.

[0047] Based on the above embodiment, the visual area determination module 32: A filtering unit configured to perform correlation filtering on an image corresponding to the multi-modal features to obtain a visual region of the target object.

[0048] Based on the above embodiment, the filtering unit comprises: The method is configured to determine a convolution kernel based on the text features, and perform a convolution operation on an image corresponding to the multi-modal features based on the convolution kernel to obtain a visual region of the target object.

[0049] According to the above embodiment, the visual features include visual features extracted from the original image at at least two resolutions respectively.

[0050] Based on the above embodiment, the fusion module 31 includes: The method is configured to: sort the at least two visual features in order of resolution size, obtain a sorted result, map the text feature to a feature space in which a first visual feature corresponding to a first resolution having the smallest value in the sorted result is located using a mapping matrix, stitch the first visual feature and the mapped text feature to obtain a first stitching feature, upsample the first stitching feature, stitch the upsampled first stitching feature with a second visual feature corresponding to a second resolution in the sorted result, obtain a second stitching feature, stitch the upsampled stitching feature with the visual feature having the largest resolution in the sorted result, and cyclically perform the upsampling and stitching operations until a multi-modal feature is obtained, wherein the second resolution is greater than the first resolution and is the smallest resolution other than the first resolution.

[0051] Based on the above embodiment, the division result determination module 33: The method is configured to input the image corresponding to the multi-modal features and the response heat map into a target image segmentation model, and obtain an output result from the target image segmentation model as a segmentation result of the target object.

[0052] Based on the above embodiment, the training process of the target image segmentation model includes: The method includes obtaining a sample image and a sample description language for specifying a sample target object in the sample image to be segmented, extracting sample visual features of the sample image and sample text features of the sample description language, fusing the sample visual features and the sample text features to obtain sample multi-modal features, determining a sample visual region of the sample target object based on an image corresponding to the sample multi-modal features, notating the image corresponding to the sample visual region as a sample response heat map, training an initial image segmentation model based on the image corresponding to the sample multi-modal features and the sample response heat map, and obtaining the target image segmentation model.

[0053] Based on the above embodiment, training an initial image segmentation model based on the images corresponding to the sample multi-modal features and the sample response heatmap to obtain the target image segmentation model includes: The method includes inputting the image corresponding to the sample multimodal features and the sample response heatmap into the initial image segmentation model, obtaining a plurality of first convolution results of the image corresponding to the sample multimodal features and the sample response heatmap, where the plurality of first convolution results are obtained by performing a first convolution operation on the image corresponding to the sample multimodal features and the sample response heatmap at different sampling rates; stitching the plurality of first convolution results to obtain a stitching result; performing a second convolution operation on the stitching result to obtain a second convolution result; upsampling the second convolution result to obtain a sample segmentation result; determining a loss value of the sample segmentation result relative to a realistic segmentation result of the sample image; if the loss value is smaller than a set threshold, stopping training of the initial image segmentation model, and taking an image segmentation model whose loss value is smaller than a set threshold as the target image segmentation model; if the loss value is equal to or greater than the set threshold, continuing to train the initial image segmentation model until the loss value is smaller than the set threshold.

[0054] The image segmentation device according to the embodiments of the present disclosure belongs to the same idea as the image segmentation method according to the above embodiments, and for technical details not described in detail in this embodiment, reference can be made to the above embodiments, and this embodiment has the same effect as executing the image segmentation method. EXAMPLES

[0055] Hereinafter, referring to FIG. 10, a structural schematic diagram of an electronic device 600 suitable for implementing the embodiment of the present disclosure is shown. The electronic device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (PDA), tablet computers (PAD), portable media players (PMP), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital televisions, desktop computers, and the like. The electronic device shown in FIG. 10 is merely an example and does not impose any limitations on the functions and scope of use of the embodiment of the present disclosure.

[0056] As shown in Fig. 10, the electronic device 600 may include a processing unit (e.g., a central processor, a graphic processor, etc.) 601 capable of performing various suitable operations and processes based on programs stored in a Read-Only Memory (ROM) 602 or loaded from a storage device 608 into a Random Access Memory (RAM) 603. The RAM 603 further stores various programs and data necessary for the operation of the electronic device 600. The processing unit 601, the ROM 602 and the RAM 603 are connected to each other via a bus 604. An Input / Output (I / O) interface 605 is also connected to the bus 604.

[0057] Typically, input devices 606 including, for example, a touch panel, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc., output devices 607 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc., storage devices 608 including, for example, a magnetic tape, hard disk, etc., and communication devices 609 can be connected to the I / O interface 605. The communication devices 609 enable the electronic device 600 to communicate wirelessly or wired with other devices to exchange data. Although FIG. 10 illustrates the electronic device 600 with various devices, it should be understood that it is not necessary to implement or include all the illustrated devices. Instead, more or fewer devices may be implemented or included.

[0058] According to an embodiment of the present disclosure, the process described with reference to the flowchart above may be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product including a computer program carried on a non-transitory computer-readable medium, the computer program including program code for performing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network by the communication device 609, or may be installed from the storage device 608, or may be installed from the ROM 602. When the computer program is executed by the processing device 601, it performs the above-mentioned functions defined in the method of the embodiment of the present disclosure. EXAMPLES

[0059] The computer readable medium described in this disclosure may be a computer readable signal medium, a computer readable storage medium, or any combination of the above. The computer readable storage medium may be, for example, but not limited to, an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. Further specific examples of the computer readable storage medium may include, but are not limited to, an electrical connection with one or more leads, a portable computer disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this disclosure, the computer readable storage medium may be any tangible medium that contains or stores a program that can be used in or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, in which computer-readable program code is carried. Such a propagated data signal may take various forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which may transmit, propagate, or transmit a program for use in or in conjunction with an instruction execution system, apparatus, or device. The program code contained in the computer-readable medium may be transmitted by any suitable medium, including, but not limited to, an electrical wire, an optical cable, a radio frequency (RF), or the like, or any suitable combination of the above.

[0060] In some embodiments, the clients and servers may communicate using any now known or later developed network protocol, such as HyperText Transfer Protocol (HTTP), and may interconnect with any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a Local Area Network (LAN), a Wide Area Network (WAN), a network off a network (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), and any now known or later developed network.

[0061] The computer-readable medium may be included in the electronic device, or may exist separately and not be attached to the electronic device.

[0062] The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device obtains multi-modal features by fusing visual features corresponding to an original image and text features corresponding to a description language for specifying a target object to be segmented in the original image, determines visual areas of the target object based on the image corresponding to the multi-modal features, records the image corresponding to the visual areas as a response heat map, and determines a segmentation result of the target object based on the image corresponding to the multi-modal features and the response heat map.

[0063] Computer program code for carrying out the operations of the present disclosure can be written in one or more programming languages ​​or combinations thereof, which may include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and also include conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a single, separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a LAN or WAN, or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0064] The flowcharts and block diagrams in the drawings illustrate possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, program, or part of code, which includes one or more executable instructions for implementing a certain logical function. It should be noted that in some alternative implementations, the functions described in the blocks may occur in a different order than the order described in the drawings. For example, two blocks shown as connected may actually be executed substantially in parallel or in reverse order depending on the functions involved. It should be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts may be implemented in a system based on dedicated hardware that executes a certain function or operation, or may be implemented in a combination of dedicated hardware and computer instructions.

[0065] The modules or units according to the embodiments of the present disclosure may be implemented in a software manner or a hardware manner, where the names of the modules or units may not be limited to the modules or units themselves, for example, a stitching module may be described as a "module for stitching visual features corresponding to an original image and text features corresponding to a language text to obtain multimodal features."

[0066] The functionality of the present invention may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Parts (ASSPs), System on Chip (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0067] In the specification of the present disclosure, a machine-readable medium may be a tangible medium that can contain or store a program used in or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. Examples of machine-readable storage media include an electrical connection by one or more wires, a portable computer disk, a hard disk, a RAM, a ROM, an EPROM or flash memory, an optical fiber, a CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0068] According to one or more embodiments of the present disclosure, the present disclosure provides a method for producing a method for manufacturing a semiconductor device comprising: The method includes: fusing visual features corresponding to an original image with text features corresponding to a description language for specifying a target object to be segmented in the original image to obtain multi-modal features; determining visual regions of the target object based on an image corresponding to the multi-modal features, and notating the image corresponding to the visual regions as a response heat map; and determining a segmentation result of the target object based on the image corresponding to the multi-modal features and the response heat map. A method for segmenting an image is provided.

[0069] According to one or more embodiments of the present disclosure, in the image segmentation method according to the present disclosure, determining a visual region of the target object based on an image corresponding to the multi-modal features includes: Correlation filtering is performed on the image corresponding to the multi-modal features to obtain a visual region of the target object.

[0070] According to one or more embodiments of the present disclosure, in the image segmentation method of the present disclosure, performing correlation filtering on the image corresponding to the multi-modal features to obtain the visual area of ​​the target object includes: The method includes determining a convolution kernel based on the text features; and performing a convolution operation on an image corresponding to the multi-modal features based on the convolution kernel to obtain a visual region of the target object.

[0071] According to one or more embodiments of the present disclosure, in the image segmentation method of the present disclosure, the visual features include visual features extracted from the original image at at least two resolutions, respectively.

[0072] According to one or more embodiments of the present disclosure, in the image segmentation method of the present disclosure, fusing visual features corresponding to the original image and text features corresponding to a description language to obtain multi-modal features includes: The method includes: sorting the at least two visual features in order of resolution size to obtain a sorted result; mapping the text feature to a feature space where a first visual feature corresponding to a first resolution having the smallest value in the sorted result is located using a mapping matrix; stitching the first visual feature and the mapped text feature to obtain a first stitching feature; upsampling the first stitching feature, stitching the upsampling first stitching feature with a second visual feature corresponding to a second resolution in the sorted result to obtain a second stitching feature; and cyclically performing the upsampling and stitching operations until a multi-modal feature is obtained by stitching the upsampling stitching feature with a visual feature having the largest resolution in the sorted result, wherein the second resolution is a smallest resolution other than the first resolution and larger than the first resolution.

[0073] According to one or more embodiments of the present disclosure, in the image segmentation method of the present disclosure, determining a segmentation result of the target object based on the image corresponding to the multi-modal features and the response heat map includes: The method includes inputting the image corresponding to the multi-modal features and the response heat map into a target image segmentation model, and obtaining an output result from the target image segmentation model as a segmentation result of the target object.

[0074] According to one or more embodiments of the present disclosure, in the image segmentation method of the present disclosure, the training process of the target image segmentation model includes: The method includes obtaining a sample image and a sample description language for specifying a sample target object in the sample image to be segmented, extracting sample visual features of the sample image and sample text features of the sample description language, fusing the sample visual features and the sample text features to obtain sample multi-modal features, determining a sample visual region of the sample target object based on an image corresponding to the sample multi-modal features, notating the image corresponding to the sample visual region as a sample response heat map, training an initial image segmentation model based on the image corresponding to the sample multi-modal features and the sample response heat map, and obtaining the target image segmentation model.

[0075] According to one or more embodiments of the present disclosure, in the image segmentation method of the present disclosure, training an initial image segmentation model based on images corresponding to the sample multi-modal features and the sample response heatmap to obtain the target image segmentation model includes: The method includes inputting the image corresponding to the sample multimodal features and the sample response heatmap into the initial image segmentation model, obtaining a plurality of first convolution results of the image corresponding to the sample multimodal features and the sample response heatmap, where the plurality of first convolution results are obtained by performing a first convolution operation on the image corresponding to the sample multimodal features and the sample response heatmap at different sampling rates; stitching the plurality of first convolution results to obtain a stitching result; performing a second convolution operation on the stitching result to obtain a second convolution result; upsampling the second convolution result to obtain a sample segmentation result; determining a loss value of the sample segmentation result relative to a realistic segmentation result of the sample image; if the loss value is smaller than a set threshold, stopping training of the initial image segmentation model, and taking an image segmentation model whose loss value is smaller than a set threshold as the target image segmentation model; if the loss value is equal to or greater than the set threshold, continuing to train the initial image segmentation model until the loss value is smaller than the set threshold.

[0076] According to one or more embodiments of the present disclosure, the present disclosure provides a method for producing a method for manufacturing a semiconductor device comprising: The method includes: a fusion module configured to obtain multi-modal features by fusing visual features corresponding to an original image and text features corresponding to a description language for specifying a target object to be segmented in the original image; a visual area determination module configured to determine visual areas of the target object based on an image corresponding to the multi-modal features and to note the image corresponding to the visual areas as a response heat map; and a segmentation result determination module configured to determine a segmentation result of the target object based on the image corresponding to the multi-modal features and the response heat map. An image segmentation device is provided.

[0077] According to one or more embodiments of the present disclosure, the present disclosure provides a method for producing a method for manufacturing a semiconductor device comprising: An electronic device comprising one or more processors and a memory configured to store one or more programs, the one or more programs being configured to, when executed by the one or more processors, realize an image segmentation method according to any one of the embodiments of the present disclosure. Provide electronic equipment.

[0078] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, the computer-readable storage medium implementing an image segmentation method according to any of the embodiments of the present disclosure when executed by a processor.

[0079] Also, although operations have been described in a particular order, it should not be understood that these operations must be performed in the particular order shown, or in a forward order. In certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although the above discussion includes a number of implementation details, these should not be understood as limiting the scope of the disclosure. Some features described in the description of a single embodiment may be implemented in combination in a single embodiment. Conversely, various features described in the description of a single embodiment may be implemented in multiple embodiments alone or in any suitable subcombination.

Claims

1. fusing visual features corresponding to an original image and text features corresponding to a description language for specifying a target object to be segmented in the original image to obtain multimodal features; determining visual regions of the target object based on images corresponding to the multi-modal features and notating the images corresponding to the visual regions as a response heat map; determining a segmentation result of the target object based on an image corresponding to the multi-modal features and the response heat map; the visual features include at least three visual features extracted from the original image at at least three resolutions, respectively; Acquiring multimodal features by fusing visual features corresponding to the original image and text features corresponding to a description language includes: sorting the at least three visual features in order of resolution size, obtaining a sorted result, and mapping the text feature to a feature space in which a first visual feature corresponding to a first resolution having the smallest value in the sorted result is located by a mapping matrix; stitching the first visual features and the mapped text features to obtain a first stitching feature; upsampling the first stitching feature; stitching the upsampling first stitching feature with a second visual feature corresponding to a second resolution in the sorting result to obtain a second stitching feature; and cyclically performing the upsampling and stitching operations until stitching the upsampling stitching feature with the visual feature having the highest resolution in the sorting result to obtain the multi-modal feature, wherein the second resolution is a smallest resolution greater than the first resolution and other than the first resolution. Image segmentation methods.

2. Determining a visual region of the target object based on an image corresponding to the multimodal features includes: performing correlation filtering on the image corresponding to the multi-modal features to obtain a visual region of the target object; The method of claim 1.

3. performing correlation filtering on an image corresponding to the multi-modal features to obtain a visual region of the target object; determining a convolution kernel based on the text features; performing a convolution operation on an image corresponding to the multi-modal features based on the convolution kernel to obtain a visual area of ​​the target object; The method of claim 2.

4. determining a segmentation result of the target object based on an image corresponding to the multi-modal features and the response heat map; inputting the image corresponding to the multi-modal features and the response heat map into a target image segmentation model, and obtaining an output result from the target image segmentation model as a segmentation result of the target object. The method according to any one of claims 1 to 3.

5. The training process of the target image segmentation model includes: Obtaining a sample image and a sample description language for specifying a sample target object in the sample image to be segmented, and extracting sample visual features of the sample image and sample text features of the sample description language; Fusing the sample visual features and the sample text features to obtain sample multi-modal features; determining a sample visual region of the sample target object based on an image corresponding to the sample multi-modal features, and labeling the image corresponding to the sample visual region as a sample response heat map; training an initial image segmentation model based on images corresponding to the sample multi-modal features and the sample response heatmap to obtain the target image segmentation model; The method according to claim 4.

6. training an initial image segmentation model based on images corresponding to the sample multi-modal features and the sample response heatmap to obtain the target image segmentation model, inputting the image corresponding to the sample multimodal features and the sample response heatmap into the initial image segmentation model, and obtaining a plurality of first convolution results of the image corresponding to the sample multimodal features and the sample response heatmap, the plurality of first convolution results being obtained by performing a first convolution operation on the image corresponding to the sample multimodal features and the sample response heatmap at different sampling rates; stitching the first convolution results to obtain a stitching result; performing a second convolution operation on the stitching result to obtain a second convolution result; Upsampling the second convolution result to obtain a sample division result; determining a loss value of the sample segmentation result relative to a real segmentation result of the sample image; If the loss value is smaller than a set threshold, stopping training of the initial image segmentation model, and setting an image segmentation model having a loss value smaller than the set threshold as the target image segmentation model; if the loss value is equal to or greater than the set threshold, continuing to train the initial image segmentation model until the loss value becomes smaller than the set threshold. The method according to claim 5.

7. a fusion module configured to fuse visual features corresponding to an original image and text features corresponding to a description language for specifying a target object to be segmented in the original image to obtain multi-modal features; a visual region determination module configured to determine visual regions of the target object based on images corresponding to the multi-modal features and to mark the images corresponding to the visual regions as a response heat map; a segmentation result determination module configured to determine a segmentation result of the target object based on an image corresponding to the multi-modal features and the response heat map; the visual features include at least three visual features extracted from the original image at at least three resolutions, respectively; The fusion module comprises: sorting the at least three visual features in order of resolution size, obtaining a sorted result, and mapping the text feature to a feature space in which a first visual feature corresponding to a first resolution having the smallest value in the sorted result is located by a mapping matrix; stitching the first visual features and the mapped text features to obtain a first stitching feature; upsampling the first stitching feature; stitching the upsampling first stitching feature with a second visual feature corresponding to a second resolution in the sorting result to obtain a second stitching feature; and cyclically performing the upsampling and stitching operations until stitching the upsampling stitching feature with the visual feature having the highest resolution in the sorting result to obtain the multimodal feature, thereby realizing fusing the visual feature corresponding to the original image and the text feature corresponding to the description language to obtain the multimodal feature, wherein the second resolution is greater than the first resolution and is the smallest resolution other than the first resolution. Image segmentation device.

8. At least one processor; a memory configured to store at least one program; An electronic device comprising: When the at least one program is executed by the at least one processor, the image segmentation method according to any one of claims 1 to 3 is realized. electronic equipment.

9. A computer-readable storage medium having a computer program stored therein, the computer-readable storage medium realizing the image segmentation method according to any one of claims 1 to 3 when the program is executed by a processor. A computer-readable storage medium.

Images