Upsampling method and device, electronic equipment, storage medium and computer program product

Abstract

The embodiment of the invention provides an up-sampling method, an up-sampling device, electronic equipment, a chip, a storage medium and a computer program product. The method comprises the following steps: extracting visual features of a target image and text features of a to-be-segmented target; obtaining a segmentation feature and a deconvolution offset of the target image according to the visual feature and the text feature; and on the basis of the deconvolution offset, performing up-sampling processing on the segmentation features through a deformation deconvolution mode to obtain a segmentation result of the target image.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and specifically to an upsampling method, an upsampling device, an electronic device, a chip, a storage medium, and a computer program product. Background Art

[0002] In the image segmentation task, there is an upsampling process. Upsampling is the most important step before the model outputs the segmentation result. This step determines the final image segmentation quality. At present, the upsampling process usually uses traditional upsampling methods, such as linear interpolation, depooling and deconvolution. Using these methods for upsampling will cause the segmentation results obtained by the upsampling process to have serious distortion problems. Summary of the invention

[0003] Embodiments of the present application provide an upsampling method, an upsampling device, an electronic device, a chip, a storage medium, and a computer program product.

[0004] The upsampling method provided in the embodiment of the present application includes:

[0005] Extract the visual features of the target image and the text features of the target to be segmented;

[0006] Obtaining segmentation features and deconvolution offsets of the target image according to the visual features and the text features;

[0007] Based on the deconvolution offset, the segmentation feature is upsampled by deformed deconvolution to obtain a segmentation result of the target image.

[0008] The upsampling device provided in the embodiment of the present application includes:

[0009] Extraction unit: used to extract the visual features of the target image and the text features of the target to be segmented;

[0010] Processing unit: used for obtaining the segmentation features and deconvolution offset of the target image according to the visual features and the text features;

[0011] The processing unit is used to upsample the segmentation feature based on the deconvolution offset by deforming deconvolution to obtain a segmentation result of the target image.

[0012] The electronic device provided in the embodiment of the present application includes: a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute any upsampling method provided in the embodiment of the present application.

[0013] The chip provided in the embodiment of the present application includes: a processor, which is used to call and run a computer program from a memory, so that a device equipped with the chip executes any upsampling method provided in the embodiment of the present application.

[0014] The storage medium provided in the embodiments of the present application is used to store a computer program, and the computer program enables a computer to execute any upsampling method provided in the embodiments of the present application.

[0015] The computer program product provided by the embodiments of the present application includes a computer program, and when the computer program is executed by a processor, it implements any upsampling method provided by the embodiments of the present application.

[0016] Through the upsampling method provided in the embodiment of the present application, the segmentation features are upsampled by a deformed deconvolution method, and the deconvolution offset is calculated based on the text features of the target to be segmented by the visual features, so that during the upsampling process, information of other modalities besides the visual features can be used to correct the segmentation results, thereby improving the accuracy of the segmentation results and avoiding distortion problems in the image segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 Schematic diagram of the implementation process of the upsampling method provided in the embodiment of the present application Figure 1 ;

[0019] Figure 2 A schematic diagram of upsampling processing by deformable deconvolution provided in an embodiment of the present application;

[0020] Figure 3 Schematic diagram of the implementation process of the upsampling method provided in the embodiment of the present application Figure 2 ;

[0021] Figure 4 A schematic structural diagram of an up-sampling device 400 provided in an embodiment of the present application;

[0022] Figure 5 A schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0023] Figure 6 A schematic structural diagram of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] It should be noted that in the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the embodiments of the present application, the character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0026] In the description of the embodiments of the present application, the term "corresponding" may indicate a direct or indirect correspondence between two items, or an association relationship between the two items, or a relationship between indication and being indicated, configuration and being configured, and the like.

[0027] To facilitate understanding of the technical solutions of the embodiments of the present application, the relevant technologies of the embodiments of the present application are described below. The following related technologies can be arbitrarily combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0028] In the process of upsampling, the related technical solutions lack design optimization for the task itself, and only use traditional upsampling methods, such as linear interpolation, depooling and deconvolution, to output the final upsampling results. Among these upsampling methods, linear interpolation and depooling methods will cause serious distortion problems in the upsampled images; the deconvolution method introduces learnable parameters, which alleviates the distortion problem of the upsampled images, but the process still lacks additional attention to key local structures, affecting the final model performance. In the task of reference image segmentation, upsampling is the most important step before outputting the segmentation result. This step determines the final image segmentation quality. Therefore, how to better design the upsampling module is an urgent problem to be solved. At the same time, because reference image segmentation of characters is one of the basic tasks of computer vision, it has important application value in autonomous driving, image generation, conference office, behavior analysis, security monitoring, etc., and upsampling technology is also particularly important. The reference image segmentation task is to obtain the pixel-level segmentation result of the image containing the given category to be segmented. The current technology has the above-mentioned upsampling method problems when segmenting. It only uses common upsampling methods. These upsampling methods only restore the image to its original size based on the input visual features. They do not have the ability to emphasize and correct the segmented target, and cannot adaptively correct the upsampling results according to the specific semantics of the target to be segmented, resulting in low segmentation accuracy and blurring of the key local structure of the target. A new upsampling method that can solve the problem of image segmentation result distortion is urgently needed.

[0029] Figure 1 Schematic diagram of the implementation process of the upsampling method provided in the embodiment of the present application Figure 1 ,like Figure 1 As shown, the embodiment of the present application provides an upsampling method, the method comprising the following steps:

[0030] Step 101: extracting visual features of the target image and text features of the target to be segmented.

[0031] The upsampling method provided in the embodiment of the present application can be applied to image segmentation tasks.

[0032] For example, the target image is in, represents a real number, W represents the width of the target image, H represents the height of the target image, and C represents the number of channels in the color space where the target image is located; the visual features of the target image x can be extracted through the image encoder to obtain the visual features in, Represents a real number, W1 represents the width of the visual feature, H1 represents the height of the visual feature, and C1 represents the number of channels of the visual feature. In actual application, the visual features of the target image can also be extracted by other methods, such as non-convolutional neural network models, self-supervised learning, ensemble learning, etc., which is not limited to the embodiments of the present application.

[0033] For example, the text features of the target to be segmented can be extracted from the description text of the target to be segmented, and the encoder extracts the text features from the description text w of the target to be segmented. in, represents a real number, L represents the total length of the text, C t The feature dimension representing the text feature can also be extracted from the text features of the target to be segmented in practical applications by other means, such as shallow neural networks, dictionary-based methods, etc., which is not limited in the embodiments of the present application.

[0034] Step 102: Obtain segmentation features and deconvolution offsets of the target image based on the visual features and the text features.

[0035] In an embodiment of the present application, a deconvolution offset is calculated based on the visual features and the text features. For example, a cross-attention feature of the visual features and the text features can be calculated through a cross-attention mechanism so that the visual features pay attention to the information of the text features, and then the cross-attention feature is input into a convolution layer to obtain a deconvolution offset. For example, the convolution kernel size of the convolution layer can be set to 3*3.

[0036] Based on this, in an optional implementation manner of the present application, obtaining the deconvolution offset according to the visual features and the text features includes:

[0037] According to the visual feature and the text feature, a cross-attention feature of the visual feature and the text feature is obtained so that the visual feature pays attention to information of the text feature;

[0038] Based on the cross-attention features, the deconvolution offset is calculated.

[0039] In an embodiment of the present application, the visual features and the text features can be respectively input into the self-attention layer to obtain the self-attention features of the visual features and the self-attention features of the text features, and then the self-attention features of the visual features and the self-attention features of the text features can be input into the cross-attention layer to obtain the cross-attention features of the visual features and the text features, thereby further improving the accuracy of the deconvolution offset results.

[0040] In an embodiment of the present application, the upsampling process includes one or more upsampling processes. Each time an upsampling process is performed, the size of the segmentation feature increases until it is consistent with the size of the target image. Therefore, one or more deconvolution offsets can be calculated to provide a deconvolution offset for each upsampling process in the one or more upsampling processes.

[0041] Exemplarily, the size of the visual feature can be adjusted through one or more linear layers to obtain one or more visual features of different sizes; wherein the size of the visual feature includes the spatial dimension and the number of channels. The size of the text feature can be adjusted through one or more linear layers to obtain one or more text features of different sizes; the size of the text feature includes: text length and feature dimension. The one or more visual features of different sizes correspond to one or more text features of different sizes one by one; wherein the number of channels of the corresponding visual features and the feature dimension of the text features are the same, ensuring that the corresponding visual features and text features can smoothly obtain cross-attention features through the cross-attention mechanism.

[0042] The corresponding visual features and text features are a pair of visual features and text features, and one or more visual features of different sizes and one or more text features of different sizes can form one or more pairs of visual features and text features, wherein each pair of visual features and text features can obtain a deconvolution offset, and through the cross-attention mechanism, one or more deconvolution offsets can be obtained according to the one or more pairs of visual features and text features, and the one or more deconvolution offsets correspond one-to-one to the one or more text features of different sizes.

[0043] Based on this, in an optional implementation manner of the present application, the size of the visual feature includes: the number of channels, and the size of the text feature includes: the feature dimension; according to the visual feature and the text feature, the cross attention feature of the visual feature and the text feature is obtained, including:

[0044] The visual features and the text features are resized to obtain one or more visual features of different sizes and one or more text features of different sizes; the one or more visual features of different sizes correspond to the one or more text features of different sizes; wherein the number of channels of the corresponding visual features and the feature dimension of the text features are the same;

[0045] Obtaining one or more cross-attention features according to one or more pairs of one-to-one corresponding visual features and text features, wherein the one or more cross-attention features correspond one-to-one to the one or more visual features of different sizes;

[0046] The deconvolution offset is obtained based on the cross attention feature, including:

[0047] Based on the one or more cross-attention features, one or more deconvolution offsets are obtained, and the one or more deconvolution offsets correspond one-to-one to the one or more visual features of different sizes.

[0048] Here, the size of the one or more visual features of different sizes can be designed according to the upsampling processing flow. For example, the segmentation feature needs to be upsampled four times. The size of the segmentation feature input for the first upsampling is 14*14*1024, where 14*14 is the spatial dimension of the segmentation feature and 1024 is the number of channels of the segmentation feature. After the first upsampling, the size of the segmentation feature becomes 28*28*1024, and the size of the segmentation feature after the second upsampling is 56*56*256. The size of the segmentation feature after the third upsampling is 112*112*128, and the size of the segmentation feature after the fourth upsampling is 224*224*1, which is the same as the spatial dimension of the target image. If each upsampling process needs to be corrected by a deconvolution offset, then four deconvolution offsets need to be calculated. The four deconvolution offsets correspond to four visual features of different sizes, and the sizes of the four visual features of different sizes are 14*14*1024, 28*28*256, 56*56*128, and 112*112*64, respectively. Furthermore, the four visual features of different sizes correspond to four text features of different sizes, and the feature dimensions of the four text features of different sizes are 1024, 256, 128, and 64, respectively. The spatial dimensions and channel numbers of the four deconvolution offsets obtained by the cross-attention mechanism for the four pairs of visual features and text features are 14*14*1024, 28*28*256, 56*56*128, and 112*112*1, respectively. Among them, the number of channels of the fourth deconvolution offset is set to 1 because the segmentation features of the same spatial dimensions as the target image are output after the fourth upsampling process, that is, the final segmentation result.

[0049] In an embodiment of the present application, the extracted visual feature can be a visual feature with a relatively small size and a large number of channels, for example, H1=H / 16, W1=H / 16, and then the spatial dimension of the visual feature is enlarged through one or more linear layers. For example, the spatial dimension of the visual feature can be enlarged to 2H1*2W1, 4H1*4W1, 8H1*8W1. The number of channels can be designed according to actual needs. The number of channels can decrease as the size increases, or it can remain unchanged. The embodiment of the present application does not limit the design of the number of channels.

[0050] In the embodiment of the present application, the visual feature and text feature with the smallest spatial dimension among the one or more visual features of different sizes can be used to learn the segmentation feature. For example, if the spatial dimension of the extracted visual feature is the smallest, the learned segmentation feature in, represents a real number, f r The size is H1×W1×C. In practical applications, small-sized visual features can be used to focus on higher-level semantic features and improve the accuracy of learned segmentation features.

[0051] Based on this, in an optional implementation manner of the present application, obtaining the segmentation features of the target image according to the visual features and the text features includes:

[0052] The visual feature with the smallest size among the one or more visual features of different sizes and the text feature are feature-fused to obtain a segmentation feature of the target image; wherein the size of the segmentation feature is the same as the size of the visual feature with the smallest spatial dimension among the one or more visual features of different sizes.

[0053] In an embodiment of the present application, visual features and text features can be input into a decoder to obtain segmentation features, and the decoder can be composed of N layers of standard transform decoding layers in series. Each transform decoding layer includes: a self-attention layer, a cross-attention layer, and a feedforward neural network. After N layers of transform decoding layers, the visual features and text features are fused to obtain segmentation features, wherein the text features of other numbered decoding layers except the first transform decoding layer are input as text features of the previous transform decoding layer after the self-attention layer fuses the self-attention features. Preferably, the value of N can be 4, 5 or 6.

[0054] Step 103: Based on the deconvolution offset, up-sample the segmentation feature by means of deformation deconvolution to obtain a segmentation result of the target image.

[0055] In an embodiment of the present application, the deconvolution offset includes one or more deconvolution offsets, and based on the deconvolution offsets, the segmentation feature is upsampled by deformed deconvolution to obtain the segmentation result of the target image, including:

[0056] Based on the one or more deconvolution offsets, the segmentation feature is upsampled by deformed deconvolution to obtain a segmentation result of the target image.

[0057] In an embodiment of the present application, the upsampling process includes one or more upsampling processes until a segmentation feature with the same size as the target image is obtained, wherein the deconvolution offset used in each upsampling process is selected from the one or more deconvolution offsets, and the spatial dimension of the selected deconvolution offset is the same as the spatial dimension of the segmentation feature input in this upsampling process.

[0058] Based on this, in an optional implementation manner of the present application, the size of the visual feature further includes: a spatial dimension; the size of the deconvolution offset includes: a spatial dimension; based on the one or more deconvolution offsets, the segmentation feature is upsampled by a deformed deconvolution method to obtain a segmentation result of the target image, including:

[0059] The segmentation feature is upsampled once or multiple times by means of deformed deconvolution until a segmentation feature with the same size as the target image is obtained, and the segmentation feature with the same size as the target image is the segmentation result; wherein,

[0060] The spatial dimension of the deconvolution offset used in each upsampling process in the one or more upsampling processes is the same as the spatial dimension of the segmentation feature input in the upsampling process.

[0061] refer to Figure 2 , Figure 2 This is a schematic diagram of upsampling processing by deformable deconvolution provided in an embodiment of the present application. Figure 2 As shown in the figure, the size of the deformed deconvolution kernel is 2*2. The sampling points for shadowing become 4 sampling points after upsampling, namely sampling point 1, sampling point 2, sampling point 3 and sampling point 4. According to the deconvolution offset of each of the 4 sampling points, the positions of each of the 4 sampling points with the deconvolution offset are sampling point 1', sampling point 2', sampling point 3' and sampling point 4'. Taking sampling point 1 as an example, sampling point 1' has the highest overlap with the sampling point above sampling point 1. Finally, the position of sampling point 1 is adjusted to the sampling point above sampling point 1.

[0062] In an embodiment of the present application, the size of the deconvolution offset also includes the number of sampling points and the horizontal and vertical coordinates; the number of sampling points is the number of pixels in the deconvolution area corresponding to each pixel position. For example, if the size of the deformed convolution kernel is 2*2, the number of sampling points is 4; the horizontal and vertical coordinates represent the position of the pixel corresponding to the deconvolution offset.

[0063] Based on this, in an optional implementation of the present application, the size of the deconvolution offset also includes: the number of sampling points; the number of sampling points is related to the deformed convolution kernel in the upsampling process.

[0064] In the embodiment of the present application, the linear layer, decoder, convolutional layer and deconvolutional layer are all learnable and can be obtained through a loss function, such as l focal = Focal_loss(P,Y), by calculating the difference between the predicted result P and the manually labeled result Y as the loss, update the parameters of the linear layer, decoder, convolution layer and deconvolution layer.

[0065] refer to Figure 3 , Figure 3 Schematic diagram of the implementation process of the upsampling method provided in the embodiment of the present application Figure 2 ,like Figure 3 As shown, the embodiment of the present application provides an upsampling method, the method comprising the following steps:

[0066] Step 301: Obtain the image to be segmented and the text description.

[0067] Exemplarily, the size of the segmented image is 224*224*3, and the text description of the object to be segmented is: a swimming athlete.

[0068] Step 302: Extract visual features and text features.

[0069] The segmented image is input into the visual encoder to obtain 14*14*1024 visual features.

[0070] Input the text description into the text encoder to obtain 17*512 text features.

[0071] The visual features and text features are sent to four layers of serial linear layers, the size of the visual features is adjusted to 14*14*1024, 28*28*256, 56*56*128, 112*112*64, the size of the text features is adjusted to 17*1024, 17*256, 17*128, 112*64, the visual features of size 14*14*1024 and the text features of size 17*1024 are a pair, the visual features of size 28*28*256 and the text features of size 17*256 are a pair, the visual features of size 56*56*128 and the text features of size 17*128 are a pair, and the visual features of size 112*112*64 and the text features of size 17*64 are a pair. The four pairs of visual features and text features are input into the self-attention layer and the cross-attention layer to obtain the visual feature and text feature cross-attention feature v t = {v t16 ,v t8 ,v t4 ,v t2}, where v t16 is the cross-attention feature of the visual feature with a size of 14*14*1024 and the text feature with a size of 17*1024, vt8 is the cross-attention feature of the visual feature with a size of 28*28*256 and the text feature with a size of 17*256, v t4 is the cross-attention feature of the visual feature with a size of 56*56*128 and the text feature with a size of 17*128, v t2 It is a cross-attention feature of visual features with a size of 112*112*64 and text features with a size of 17*64.

[0072] Step 303: Calculate segmentation features.

[0073] The visual features of size 14*14*1024 and the text features of size 17*512 are sent to the decoder. The decoder is composed of 4 layers of standard transform decoding layers in series and outputs segmentation features with a size of 14*14*1024.

[0074] Step 304: Calculate the deconvolution offset.

[0075] According to the learned feature v containing text information t Learn the deconvolution offset of deformable deconvolution, for v t16 , the offset o can be calculated by convolution with a channel number of 1024*4*2 and a convolution kernel size of 3*3 on the visual feature. 16 , where 4 in the channel is the number of sampling points, the corresponding deconvolution kernel size is 2*2, and 2 in the channel represents the horizontal and vertical coordinates of the sampling points. 16 The size of is 14*14*1024*4*2, and similarly through v t8 、v t4 、v t2 The corresponding deconvolution offsets o8, o4, and o2 are obtained respectively, where the size of o8 is 28*28*256*4*2, the size of o4 is 56*56*128*4*2, and the size of o2 is 112*112*1*4*2.

[0076] Step 305: Use the deconvolution offset to upsample and correct the segmentation features.

[0077] By o 16 , o8, o4 and o2, the segmentation features obtained in step 303 are upsampled by deformed deconvolution, and the deconvolution offset of the first upsampling is o 16, the segmentation feature size after upsampling is 28*28*1024, the deconvolution offset of the second upsampling is o8, the segmentation feature size after upsampling is 56*56*256, the deconvolution offset of the third upsampling is o4, the segmentation feature size after upsampling is 112*112*128, the deconvolution offset of the fourth upsampling is o2, the segmentation feature size after upsampling is 224*224*1, and the segmentation result is consistent with the size of the image to be segmented.

[0078] It should be noted that in the embodiments of the present application, the size of the visual features, the number of channels, the number of network layers, the number of upsampling processes, the size of the convolution kernel, etc. can be designed according to actual needs, and the embodiments of the present application do not limit this.

[0079] The upsampling method provided in the embodiment of the present application focuses on the upsampling module at the end of the model, associates the semantics of the target to be segmented with the upsampling process, and integrates the text into the upsampling process through the deconvolution offset of the deformed deconvolution, so that the upsampling process combines the features of multiple modalities, and corrects the segmentation results according to the features of multiple modalities, thereby generating a more accurate segmented image.

[0080] The present application embodiment also provides an up-sampling device 400, referring to Figure 4 , the up-sampling device 400 in this embodiment comprises:

[0081] Extraction unit: used to extract the visual features of the target image and the text features of the target to be segmented;

[0082] Processing unit: used for obtaining the segmentation features and deconvolution offset of the target image according to the visual features and the text features;

[0083] The processing unit is used to upsample the segmentation feature based on the deconvolution offset by deforming deconvolution to obtain a segmentation result of the target image.

[0084] In an embodiment of the present application, the processing unit is used to obtain a cross-attention feature of the visual feature and the text feature based on the visual feature and the text feature, so that the visual feature pays attention to the information of the text feature; and calculate the deconvolution offset based on the cross-attention feature.

[0085] In an embodiment of the present application, the size of the visual feature includes: the number of channels, and the size of the text feature includes: the feature dimension; the processing unit is used to resize the visual feature and the text feature to obtain one or more visual features of different sizes and one or more text features of different sizes; the one or more visual features of different sizes and the one or more text features of different sizes correspond one to one; wherein the number of channels of the corresponding visual features and the feature dimension of the text features are the same; based on one or more pairs of one-to-one corresponding visual features and text features, one or more cross-attention features are obtained, and the one or more cross-attention features correspond one to one with the one or more visual features of different sizes; based on the one or more cross-attention features, one or more deconvolution offsets are obtained, and the one or more deconvolution offsets correspond one to one with the one or more visual features of different sizes.

[0086] In an embodiment of the present application, the processing unit is used to perform feature fusion on the visual feature with the smallest size among the one or more visual features of different sizes and the text feature to obtain a segmentation feature of the target image; wherein the size of the segmentation feature is the same as the size of the visual feature with the smallest spatial dimension among the one or more visual features of different sizes.

[0087] In an embodiment of the present application, the processing unit is used to upsample the segmentation feature by deformable deconvolution based on the one or more deconvolution offsets to obtain a segmentation result of the target image.

[0088] In an embodiment of the present application, the size of the visual feature also includes: a spatial dimension; the size of the deconvolution offset includes: a spatial dimension; the processing unit is used to upsample the segmentation feature once or multiple times by means of deformed deconvolution until a segmentation feature with the same size as the target image is obtained, and the segmentation feature with the same size as the target image is the segmentation result; wherein the spatial dimension of the deconvolution offset used in each upsampling process in the one or more upsampling processes is the same as the spatial dimension of the segmentation feature input in the upsampling process.

[0089] In an embodiment of the present application, the size of the deconvolution offset also includes: the number of sampling points; the number of sampling points is related to the deformed convolution kernel in the upsampling process.

[0090] Those skilled in the art should understand that Figure 4 The implementation functions of each unit in the up-sampling device 400 shown can be understood by referring to the relevant description of the aforementioned method. Figure 4The functions of each unit in the up-sampling device 400 shown may be implemented by a program running on a processor, or may be implemented by a specific logic circuit.

[0091] Figure 5 It is a schematic structural diagram of an electronic device 500 provided in an embodiment of the present application. Figure 5 The electronic device 500 shown includes a processor 510, which can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0092] Alternatively, if Figure 5 As shown, the electronic device 500 may further include a memory 520. The processor 510 may call and run a computer program from the memory 520 to implement the method in the embodiment of the present application.

[0093] The memory 520 may be a separate device independent of the processor 510 , or may be integrated into the processor 510 .

[0094] Alternatively, if Figure 5 As shown, the electronic device 500 may further include a transceiver 530, and the processor 510 may control the transceiver 530 to communicate with other devices, specifically, may send information or data to other devices, or receive information or data sent by other devices.

[0095] The transceiver 530 may include a transmitter and a receiver. The transceiver 530 may further include an antenna, and the number of the antennas may be one or more.

[0096] The electronic device 500 may specifically be the upsampling device 400 of the embodiment of the present application, and the electronic device 500 may implement the corresponding processes implemented by the upsampling device 400 in each method of the embodiment of the present application, which will not be described in detail here for the sake of brevity.

[0097] Exemplarily, an embodiment of the present application further provides a computer program product, including a computer program, which can be executed by the processor 510 of the communication device 500 to complete the steps of any of the aforementioned methods.

[0098] Figure 6 It is a schematic structural diagram of the chip of an embodiment of the present application. Figure 6 The chip 600 shown includes a processor 610, which can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0099] Alternatively, if Figure 6As shown, the chip 600 may further include a memory 620. The processor 610 may call and run a computer program from the memory 620 to implement the method in the embodiment of the present application.

[0100] The memory 620 may be a separate device independent of the processor 610 , or may be integrated into the processor 610 .

[0101] Optionally, the chip 600 may further include an input interface 630. The processor 610 may control the input interface 630 to communicate with other devices or chips, and specifically, may obtain information or data sent by other devices or chips.

[0102] Optionally, the chip 600 may further include an output interface 640. The processor 610 may control the output interface 640 to communicate with other devices or chips, and specifically, may output information or data to other devices or chips.

[0103] The chip can be applied to the electronic device 500 in the embodiment of the present application, and the chip can implement the corresponding processes implemented by the electronic device 500 in the various methods of the embodiment of the present application, which will not be described in detail here for the sake of brevity.

[0104] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0105] It should be understood that the processor of the embodiment of the present application may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and performed. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, and other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0106] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0107] It should be understood that the above-mentioned memory is exemplary but not restrictive. For example, the memory in the embodiments of the present application may also be static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synch link DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DR RAM), etc. That is to say, the memory in the embodiments of the present application is intended to include but not limited to these and any other suitable types of memory.

[0108] The embodiment of the present application further provides a storage medium for storing a computer program. The storage medium can be applied to the electronic device 500 in the embodiment of the present application, and the computer program enables the computer to execute the corresponding processes implemented by the electronic device 500 in each method of the embodiment of the present application, which will not be described in detail for the sake of brevity.

[0109] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0110] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0111] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0112] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0113] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0114] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or an electronic device 500, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0115] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An upsampling method, characterized in that: include: Extract the visual features of the target image and the text features of the target to be segmented; Obtaining segmentation features and deconvolution offsets of the target image according to the visual features and the text features; Based on the deconvolution offset, the segmentation feature is upsampled by deformed deconvolution to obtain a segmentation result of the target image.

2. The method according to claim 1, characterized in that: The step of obtaining a deconvolution offset according to the visual feature and the text feature includes: According to the visual feature and the text feature, a cross-attention feature of the visual feature and the text feature is obtained so that the visual feature pays attention to information of the text feature; Based on the cross-attention features, the deconvolution offset is calculated.

3. The method according to claim 2, characterized in that The size of the visual feature includes: the number of channels, and the size of the text feature includes: the feature dimension; according to the visual feature and the text feature, a cross attention feature of the visual feature and the text feature is obtained, including: The visual features and the text features are resized to obtain one or more visual features of different sizes and one or more text features of different sizes; the one or more visual features of different sizes correspond to the one or more text features of different sizes; wherein the number of channels of the corresponding visual features and the feature dimension of the text features are the same; Obtaining one or more cross-attention features according to one or more pairs of one-to-one corresponding visual features and text features, wherein the one or more cross-attention features correspond one-to-one to the one or more visual features of different sizes; The deconvolution offset is obtained based on the cross attention feature, including: Based on the one or more cross-attention features, one or more deconvolution offsets are obtained, and the one or more deconvolution offsets correspond one-to-one to the one or more visual features of different sizes.

4. The method according to claim 3, characterized in that The step of obtaining the segmentation features of the target image according to the visual features and the text features includes: The visual feature with the smallest spatial dimension among the one or more visual features of different sizes and the text feature are feature-fused to obtain a segmentation feature of the target image; wherein the size of the segmentation feature is the same as the size of the visual feature with the smallest spatial dimension among the one or more visual features of different sizes.

5. The method according to claim 3 or 4, characterized in that: The upsampling process is performed on the segmentation feature by deforming deconvolution based on the deconvolution offset to obtain the segmentation result of the target image, including: Based on the one or more deconvolution offsets, the segmentation feature is upsampled by deformed deconvolution to obtain a segmentation result of the target image.

6. The method according to claim 5, characterized in that The size of the visual feature also includes: a spatial dimension; the size of the deconvolution offset includes: a spatial dimension; the segmentation result of the target image is obtained by upsampling the segmentation feature by deforming deconvolution based on the one or more deconvolution offsets, including: The segmentation feature is upsampled once or multiple times by means of deformed deconvolution until a segmentation feature with the same size as the target image is obtained, and the segmentation feature with the same size as the target image is the segmentation result; wherein, The spatial dimension of the deconvolution offset used in each upsampling process in the one or more upsampling processes is the same as the spatial dimension of the segmentation feature input in the upsampling process.

7. The method according to claim 6, characterized in that The size of the deconvolution offset also includes: the number of sampling points; the number of sampling points is related to the deformed convolution kernel in the upsampling process.

8. An upsampling device, characterized in that: include: Extraction unit: used to extract the visual features of the target image and the text features of the target to be segmented; Processing unit: used for obtaining the segmentation features and deconvolution offset of the target image according to the visual features and the text features; The processing unit is used to upsample the segmentation feature based on the deconvolution offset by deforming deconvolution to obtain a segmentation result of the target image.

9. An electronic device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the upsampling method according to any one of claims 1 to 7.

10. A chip, characterized in that: include: A processor, configured to call and run a computer program from a memory, so that a device equipped with the chip executes the upsampling method according to any one of claims 1 to 7.

11. A storage medium, characterized in that: Used to store a computer program, wherein the computer program enables a computer to execute the upsampling method according to any one of claims 1 to 7.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the upsampling method according to any one of claims 1 to 7.

Images