Image processing method based on semantic information and electronic device

Abstract

The application provides an image processing method based on semantic information and an electronic device, and relates to the technical field of image processing. In the application scheme, the semantic category of each pixel in the image to be processed is taken as prior information, different semantic information is subjected to different two-dimensional convolution kernels, and then spliced into a U-Net neural network to optimize the image processing capability. Since the application scheme acquires feature maps based on different semantic information according to the original image, and fine-tunes the feature mapping matrix obtained after the original image passes through the U-Net neural network based on the feature maps based on different semantic information, a more accurate image segmentation result can be obtained, the image quality of the corresponding semantic region is improved, and the image definition of the corresponding semantic region is higher. After optimization by the application scheme, the specific semantic part in the image is obviously optimized, and the texture is clearer.

Description

[0001] This application claims priority to Chinese patent application filed on November 30, 2021, with application number 202111442371.1 and title "Image Processing Method and Electronic Device Based on Semantic Information", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing technology, and in particular to an image processing method and electronic device based on semantic information. Background Technology

[0003] Image semantic segmentation is an important research area in computer vision. Its goal is to segment an image into regions with different semantic information and label each region with a corresponding semantic label.

[0004] Currently, using convolutional neural networks (CNNs) to process image semantic segmentation tasks is a common approach in the industry. In this approach, the CNN first encodes (downsamples) the input image, then decodes (upsamples) and fuses the data to obtain the final image segmentation result. However, the image segmentation results obtained using this approach lack clarity in image texture details, leading to poor image processing performance. Summary of the Invention

[0005] This application provides an image processing method and electronic device based on semantic information, which can optimize specific semantic parts in an image and make the image texture clearer.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] Firstly, this application provides an image processing method based on semantic information, the method comprising:

[0008] Based on the semantic label map of the image to be processed, first feature data and second feature data are obtained; wherein, the semantic label map is a matrix that uses different labels to represent different semantic information; the first feature data and the second feature data respectively contain semantic information with different feature intensities;

[0009] Based on the first feature data, the second feature data, and the first feature map, feature maps based on different semantic information are obtained; wherein, the first feature map is a feature mapping matrix obtained by inputting the image to be processed into a two-dimensional convolutional network;

[0010] The feature maps based on different semantic information and the second feature map of the image to be processed are fused to obtain the target feature map; wherein, the second feature map is the feature mapping matrix obtained by inputting the image to be processed into the U-Net neural network;

[0011] The target feature map is subjected to a two-dimensional convolution operation to obtain the target image.

[0012] The semantic information-based image processing method provided in this application uses the semantic category of each pixel in the image to be processed as prior information. Different semantic information is processed through different two-dimensional convolutional kernels and then concatenated into a U-Net neural network to optimize image processing capabilities. Because this application's solution fine-tunes the feature mapping matrix obtained after the U-Net neural network based on feature maps of different semantic information, it can obtain more accurate segmentation results, improve the image quality of the corresponding semantic regions, and make the image clarity of the corresponding semantic regions higher. After optimization by this application's solution, specific semantic parts in the image are significantly optimized, and the texture is clearer.

[0013] In some possible implementations, obtaining a feature map based on different semantic information based on the first feature data, the second feature data, and the first feature map may include:

[0014] The dot product between the second feature data and the first feature map of the image to be processed is added to the first feature data to obtain the feature map based on different semantic information.

[0015] In some possible implementations, the acquisition of first and second feature data based on the semantic label map of the image to be processed may specifically include:

[0016] The semantic label map of the image to be processed is convolved using the first weight matrix to obtain the first feature data;

[0017] The semantic label map of the image to be processed is convolved using a second weight matrix to obtain the second feature data.

[0018] The proposed solution uses the semantic category of each pixel in the image to be processed as prior information. Different semantic information is processed through different two-dimensional convolution kernels and then concatenated into the U-Net neural network, which can optimize image processing capabilities.

[0019] In some possible implementations, the feature strength of the preset semantic information in the second feature data is greater than the feature strength of the preset semantic information in the first feature data.

[0020] In some possible implementations, the above-mentioned feature map based on different semantic information and the second feature map of the image to be processed are fused to obtain the target feature map, which may specifically include:

[0021] The target feature map is obtained by performing a pixel-by-pixel summation operation between the feature map based on different semantic information and the second feature map of the image to be processed.

[0022] Since the proposed solution is based on feature maps with different semantic information, and fine-tunes the feature mapping matrix obtained after the U-Net neural network, it can obtain more accurate segmentation results, thereby improving the image quality of the corresponding semantic region and making the image of the corresponding semantic region clearer.

[0023] In some possible implementations, before obtaining the first feature data and the second feature data based on the semantic label map of the image to be processed, the image processing method based on semantic information provided in this application embodiment may further include:

[0024] The image to be processed is input into the semantic segmentation model;

[0025] The semantic segmentation model is used to perform semantic segmentation on the image to be processed to obtain the semantic label map.

[0026] In some possible implementations, the image processing method based on semantic information provided in this application embodiment may further include:

[0027] Multiple frames of images are continuously captured using the camera of an electronic device;

[0028] The multiple frames of images are registered to obtain the image to be processed.

[0029] In some possible implementations, after obtaining the target feature map, the image processing method based on semantic information provided in this application embodiment may further include:

[0030] The target image is displayed on the screen of an electronic device.

[0031] After being optimized by the solution proposed in this application, the images captured by electronic devices show significant improvements in specific semantic components and clearer textures.

[0032] In some possible implementations, before obtaining the target feature map based on the feature maps based on different semantic information and the second feature map of the image to be processed, the image processing method based on semantic information provided in this application embodiment may further include:

[0033] Perform a two-dimensional convolution operation on the image to be processed to obtain an intermediate feature map;

[0034] The intermediate feature map is input into the U-Net neural network, and the second feature map is output.

[0035] In some possible implementations, the intermediate feature map is input into the U-Net neural network, and the output is the second feature map. Specifically, this may include:

[0036] The intermediate feature map is input into the U-Net neural network;

[0037] The intermediate feature map is subjected to N downsampling processes and N upsampling processes sequentially through the U-Net neural network. After each upsampling process, the feature map obtained is fused with the feature map obtained from the corresponding level of downsampling process, and then the next upsampling process is performed based on the feature map obtained from the feature fusion.

[0038] After the Nth upsampling process is completed, the second feature map is obtained.

[0039] Secondly, this application provides an image processing apparatus based on semantic information, which includes units for performing the method described in the first aspect above. This apparatus can correspond to performing the method described in the first aspect above, and the relevant descriptions of the units in this apparatus are provided in the description of the first aspect above; for the sake of brevity, they will not be repeated here.

[0040] The method described in the first aspect above can be implemented in hardware or by executing corresponding software within hardware. The hardware or software includes one or more modules or units corresponding to the above functions. For example, an image feature extraction unit or module, an image feature fusion unit or module, and an image output unit or module, etc.

[0041] The image feature extraction unit can be used to obtain first feature data and second feature data of the image to be processed based on the semantic label map of the image to be processed; wherein the semantic label map is a matrix that uses different labels to represent different semantic information; the first feature data and the second feature data each contain different semantic information;

[0042] The image feature fusion unit can be used to obtain feature maps based on different semantic information based on the first feature data, the second feature data, and the first feature map; wherein, the first feature map is a feature mapping matrix obtained by inputting the image to be processed into a two-dimensional convolutional network;

[0043] The image feature fusion unit can also be used to fuse the feature map based on different semantic information and the second feature map of the image to be processed to obtain a target feature map; wherein the second feature map is a feature mapping matrix obtained by inputting the image to be processed into the U-Net neural network;

[0044] The image output unit can be used to perform two-dimensional convolution operations on the target feature map to obtain the target image.

[0045] Optionally, in some possible implementations, the image feature fusion unit described above can specifically be used for:

[0046] The dot product between the second feature data and the first feature map of the image to be processed is added to the first feature data to obtain the feature map based on different semantic information.

[0047] Optionally, in some possible implementations, the image feature extraction unit described above can specifically be used for:

[0048] The semantic label map of the image to be processed is convolved using the first weight matrix to obtain the first feature data;

[0049] The semantic label map of the image to be processed is convolved using a second weight matrix to obtain the second feature data.

[0050] Optionally, in some possible implementations, the feature strength of the preset semantic information in the second feature data is greater than the feature strength of the preset semantic information in the first feature data.

[0051] Optionally, in some possible implementations, the image feature fusion unit described above can be specifically used for:

[0052] The target feature map is obtained by performing a pixel-by-pixel summation operation between the feature map based on different semantic information and the second feature map of the image to be processed.

[0053] Optionally, in some possible implementations, the image processing apparatus based on semantic information provided in the embodiments of this application may further include an image semantic segmentation unit;

[0054] The image semantic segmentation module can be used to: input the image to be processed into the semantic segmentation model; and perform semantic segmentation on the image to be processed through the semantic segmentation model to obtain the semantic label map.

[0055] Furthermore, the image feature extraction unit can obtain the first feature data and the second feature data of the image to be processed based on the semantic label map of the image to be processed.

[0056] Optionally, in some possible implementations, the semantic information-based image processing apparatus provided in this application embodiment may further include an image acquisition unit. The image acquisition unit may be used to: continuously acquire multiple frames of images using a camera of an electronic device; and perform image registration on the multiple frames of images to obtain the image to be processed.

[0057] Optionally, in some possible implementations, the semantic information-based image processing method provided in this application embodiment may further include an image display unit. The image display unit can be used to display the target image on a display screen of an electronic device after the image output unit 830 obtains the target feature map.

[0058] Optionally, in some possible implementations, the image feature extraction unit may also be used to: perform a two-dimensional convolution operation on the image to be processed to obtain an intermediate feature map; and input the intermediate feature map into the U-Net neural network to output the second feature map.

[0059] Furthermore, the image feature fusion unit can obtain the target feature map based on the feature maps with different semantic information and the second feature map of the image to be processed.

[0060] Optionally, in some possible implementations, the image feature extraction unit may specifically be used to: input the intermediate feature map into the U-Net neural network; and perform N downsampling processes and N upsampling processes on the intermediate feature map sequentially through the U-Net neural network; wherein, after feature fusion between the feature map obtained from each upsampling process and the feature map obtained from the corresponding level of downsampling process, the next upsampling process is performed based on the feature map obtained from feature fusion. After performing the Nth upsampling process, the second feature map is obtained.

[0061] Thirdly, this application provides an electronic device including a processor coupled to a memory for storing computer programs or instructions. The processor executes the computer programs or instructions stored in the memory, causing the method in the first aspect to be performed. For example, the processor executes the computer programs or instructions stored in the memory, causing the device to perform the method in the first aspect.

[0062] Fourthly, this application provides a computer-readable storage medium having a computer program (also referred to as instructions or code) stored thereon for implementing the method of the first aspect. For example, when the computer program is executed by a computer, it enables the computer to perform the method of the first aspect.

[0063] Fifthly, this application provides a chip including a processor. The processor is used to read and execute a computer program stored in a memory to perform the methods in the first aspect and any possible implementation thereof. Optionally, the chip further includes a memory connected to the processor via a circuit or wire.

[0064] Sixthly, this application provides a chip system including a processor. The processor is used to read and execute a computer program stored in a memory to perform the methods in the first aspect and any possible implementation thereof. Optionally, the chip system further includes a memory connected to the processor via a circuit or wire.

[0065] In a seventh aspect, this application provides a computer program product comprising a computer program (also referred to as instructions or code), which, when executed by a computer, causes the computer to implement the method in the first aspect.

[0066] It is understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0067] Figure 1 This is a schematic diagram illustrating the image processing effect of image semantic segmentation in an embodiment of this application;

[0068] Figure 2 A flowchart illustrating an image processing method based on semantic information provided in an embodiment of this application;

[0069] Figure 3 A schematic diagram of the semantic tag map used in the semantic information-based image processing method provided in the embodiments of this application;

[0070] Figure 4 A schematic block diagram illustrating the acquisition of semantic information-based feature maps in the semantic information-based image processing method provided in the embodiments of this application;

[0071] Figure 5 A schematic block diagram of the U-Net neural network used in the semantic information-based image processing method provided in the embodiments of this application;

[0072] Figure 6 A schematic block diagram illustrating the overall semantic information-based image processing method provided in the embodiments of this application;

[0073] Figure 7 A flowchart illustrating another semantic information-based image processing method provided in this application embodiment;

[0074] Figure 8A schematic diagram illustrating the image processing effects of semantic segmentation and semantic labeling in the semantic information-based image processing method provided in this application embodiment;

[0075] Figure 9 A flowchart illustrating another image processing method based on semantic information provided in this application embodiment;

[0076] Figure 10 A flowchart illustrating yet another image processing method based on semantic information provided in this application embodiment;

[0077] Figure 11 A schematic diagram showing the image effect comparison of the semantic information-based image processing method provided in the embodiments of this application;

[0078] Figure 12 This is a schematic diagram of the structure of an image processing device based on semantic information provided in an embodiment of this application;

[0079] Figure 13 A schematic diagram of another semantic information-based image processing device provided in this application embodiment;

[0080] Figure 14 A schematic diagram of the structure of another semantic information-based image processing device provided in this application embodiment;

[0081] Figure 15 A schematic diagram of the structure of another semantic information-based image processing device provided in an embodiment of this application;

[0082] Figure 16 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0083] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0084] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0085] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first feature map" and "second feature map," etc., are used to distinguish different feature maps, not to describe a specific order of feature maps.

[0086] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0087] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0088] To facilitate understanding of the embodiments of this application, some terms used in the embodiments of this application are explained below, so that those skilled in the art can understand them.

[0089] 1) Image segmentation

[0090] Image segmentation is a crucial task in computer vision, aiming to classify each pixel in an image—a pixel-level classification task. Currently, there are three main types of image segmentation tasks: ordinary segmentation, semantic segmentation, and instance segmentation.

[0091] Ordinary segmentation: Separating pixel regions belonging to different objects, i.e., object detection.

[0092] Semantic segmentation: Based on ordinary segmentation, it classifies the semantics of each region (i.e., what kind of object is in this region), that is, it identifies the category of all objects in the image.

[0093] Instance segmentation: Building upon semantic segmentation, each object is assigned a unique identifier. This means not only performing pixel-level classification but also distinguishing different instances based on specific categories.

[0094] Specifically, this application's solution mainly involves semantic segmentation, which will be explained below. Since an image is composed of many pixels, semantic segmentation can be understood as grouping or segmenting pixels according to their different semantic meanings expressed in the image. Through image semantic segmentation, the content in the image can be automatically segmented and identified. In semantic segmentation, the number of categories is first determined, and then an output channel is created for each category. A single channel represents the region where a specific category exists. The result of semantic segmentation can be represented as (H, W, C) or H×W×C.

[0095] For example, such as Figure 1 As shown in (a) and (b), after semantic segmentation, the original image is divided into four semantic categories: cat, tree, cloud and background, and four channels can be output accordingly. The result of semantic segmentation can be represented as (H, W, 4) or H×W×4, where each channel classifies the pixel into a single category as 1 or 0.

[0096] This application provides an image processing method based on semantic information, which uses the semantic category of each pixel in the image to be processed (also known as the original image or raw image) as prior information to optimize image processing capabilities and improve the image quality of the corresponding semantic region.

[0097] 2) Deep Neural Networks

[0098] The image semantic segmentation task described above can be accomplished using deep neural networks. Deep neural networks can include convolutional neural networks and deconvolutional neural networks.

[0099] Convolutional neural networks (CNNs) are related to image feature extraction, transforming input images into multidimensional feature matrices. Deconvolutional neural networks (DNNs) function as region segmentation generators, utilizing the image features extracted by CNNs to perform semantic segmentation of objects.

[0100] The output of the entire deep neural network is a probability matrix graph, which is the same size as the input image. The value of each element at each position in the matrix graph represents the classification probability of the corresponding pixel in the image, that is, the probability that the object to which the pixel belongs is a certain category of object.

[0101] Understandably, a deep neural network can be viewed as a network structure that encodes and decodes. Through convolutional networks, images are convolved into small matrices; through deconvolutional networks, these small matrices are restored to a large image.

[0102] 3) Feature map

[0103] Convolutional neural networks (CNNs) include, but are not limited to, one or more convolutional layers. Each convolutional layer can include multiple filters (or convolutional kernels), and each filter is essentially an array, where the numbers are called convolution weights or parameters. The function of the convolutional layers in a CNN is to perform convolution operations on the input. For example, the filters in the first convolutional layer slide across the samples with a set stride. At each sliding position, the filter array is multiplied by the sample data and then summed to obtain a value. All the values ​​obtained during the sliding process are combined into a new array, which is called a feature map. Each value in the feature map is a feature of that feature map.

[0104] The feature map can be represented as a three-dimensional matrix of size H×W×C, which may include C two-dimensional matrices of size H×W. Here, H represents the pixel height of the image to be processed, W represents the pixel width of the image to be processed, and C represents the number of channels of the image to be processed. For example, an RGB image has 3 channels.

[0105] For example, for an H×W RGB image, the corresponding feature map can be represented as a three-dimensional matrix of H×W×3.

[0106] For example, for an H×W grayscale image, the corresponding feature map can be represented as a three-dimensional matrix of H×W×1.

[0107] 4) U-Net Neural Network

[0108] U-Net neural networks belong to the category of convolutional neural networks and have been widely used in image semantic segmentation. Also known as the U-Net network structure, U-Net neural networks are characterized by their U-shaped network structure and skip connections.

[0109] In the U-shaped network structure of U-Net, multiple downsampling operations are performed first, followed by multiple upsampling operations. Downsampling is used to extract image features, resulting in a feature map; downsampling methods can include multiple convolutional layers and pooling, which can be viewed as an image encoding process. Upsampling is used to improve the image resolution. By repeatedly upsampling the feature map output from the downsampling, a segmented image with the same size as the original input is gradually obtained. Upsampling methods can include nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation. Skip connections are used during the upsampling process to fuse the feature maps from the downsampling process.

[0110] Specifically, in this application's solution, the semantic category of each pixel in the image to be processed can be used as prior information. Different semantic information is then processed through different two-dimensional convolutional kernels and concatenated into the U-Net neural network to optimize image processing capabilities. Since this application's solution obtains feature maps based on different semantic information from the original image, and fine-tunes the feature mapping matrix obtained after processing the original image through the U-Net neural network based on these feature maps, it can obtain more accurate semantic segmentation results, improve the image quality of the corresponding semantic regions, and make the images of the corresponding semantic regions clearer.

[0111] The electronic device in this application embodiment can be a mobile terminal or a non-mobile terminal. For example, a mobile terminal can be a mobile phone, tablet computer, laptop computer, handheld computer, vehicle terminal, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., while a non-mobile terminal can be a personal computer (PC), television (TV), ATM, or self-service machine, etc., and this application embodiment does not make specific limitations.

[0112] The execution subject of the semantic information-based image processing method provided in this application embodiment can be the aforementioned electronic device, or a functional module and / or functional entity within the electronic device capable of implementing the semantic information-based image processing method. Furthermore, the solution of this application can be implemented through hardware and / or software, the specific implementation depending on actual usage requirements, and this application embodiment does not impose limitations. The following description uses an electronic device as an example, in conjunction with the accompanying drawings, to exemplarily illustrate the semantic information-based image processing method provided in this application embodiment.

[0113] Figure 2 This is a schematic flowchart of an image processing method based on semantic information provided in an embodiment of this application. (Refer to...) Figure 2 As shown, the method includes the following steps S101-S104.

[0114] S101, based on the semantic label map of the image to be processed, obtain the first feature data and the second feature data of the image to be processed; wherein, the semantic label map is a matrix that uses different labels to represent different semantic information; the first feature data and the second feature data respectively contain semantic information with different feature intensities.

[0115] In this embodiment, an image captured in a shooting scene can be used as the image to be processed. Then, a corresponding semantic label map is obtained based on the image to be processed. The semantic label map corresponds to the image to be processed, and different semantic labels are used in the semantic label map to identify different semantic regions in the image to be processed.

[0116] Optionally, a semantic label map can be obtained by manually semantically annotating the image to be processed. Of course, other image processing methods can also be used to obtain semantic label maps. The specific steps for obtaining semantic label maps will be described in detail below and will not be repeated here.

[0117] The semantic label map of the image to be processed can include multiple labels, each representing different semantic information. In other words, one label can represent one type of semantic information, and different labels represent different semantic information.

[0118] For example, Figure 3 Figure (a) shows a schematic diagram of the image to be processed. Figure 3 Figure (b) shows a schematic diagram of the semantic label map of the image to be processed, which is exemplarily labeled with four semantic labels: 0, 1, 2, 3; where 0 represents the semantic information of background, 1 represents the semantic information of cat, 2 represents the semantic information of tree, and 3 represents the semantic information of cloud. Figure 3 As shown in (b), these labels are located in corresponding regions of the image to be processed, each indicating the image features of its corresponding region. In practical implementation, the semantic information or image features of the corresponding region can be determined based on the semantic labels annotated in the image to be processed.

[0119] It should be noted that the above Figure 3 The semantic label diagram shown is illustrative and can be specifically determined according to actual usage requirements; this application does not limit its implementation. For example, in actual implementation, the semantic label diagram may include more semantic labels to represent more semantic segmentation category information; and semantic labels can be annotated pixel-by-pixel in the image to be processed to achieve more accurate image semantic segmentation.

[0120] In this embodiment, feature extraction can be performed based on the semantic label map of the image to be processed to obtain the feature data of the image to be processed. For feature extraction, the shallow structure of the convolutional neural network can extract some simple features of the original image, such as boundaries and colors, while the deep structure of the convolutional neural network can extract some abstract features of the original image, such as details and textures.

[0121] Since the solution provided in this application focuses on optimizing image sharpness for feature regions corresponding to specific semantic information in the image to be processed, two or more feature data can be extracted based on the semantic label map of the image to be processed. These feature data can contain semantic information with different feature intensities. For ease of explanation, this application embodiment uses two or more feature data, including first feature data and second feature data, as an example for illustration.

[0122] Optionally, in this embodiment of the application, image features can be extracted by performing convolution operations on the image to be processed.

[0123] For example, step S101 above may include the following possible implementations:

[0124] The semantic label map of the image to be processed is convolved using the first weight matrix to obtain the first feature data;

[0125] The second weight matrix is ​​used to perform convolution operation on the semantic label map of the image to be processed to obtain the second feature data.

[0126] For example, refer to Figure 4 The semantic label map of the image to be processed (denoted as X, here is an example semantic label map) can be input into the first feature extraction module. The first feature extraction module uses a first weight matrix (also known as convolution weight W). γ Semantic tags of the image to be processed Figure X Perform convolution to obtain the first feature data (denoted as γ): γ = W γ X.

[0127] See again Figure 4 The semantic tags of the image to be processed can be... Figure X The input is fed into the second feature extraction module, which uses the second weight matrix (also known as the convolution weight W). β Semantic tags of the image to be processed Figure X Perform convolution to obtain the second feature data (denoted as β): β = W β X.

[0128] It should be noted that the first feature extraction module and the second feature extraction module mentioned above can be modules that extract image features through convolution operations, or modules that extract image features through other methods. This application embodiment does not limit this.

[0129] Optionally, the first feature extraction module and / or the second feature extraction module can be a deconvolutional network (or a transposed convolutional network). A deconvolutional network can automatically extract rich implicit features from low-level boundaries to high-level targets. Using a deconvolutional network, high-level image features can be automatically extracted; these high-level features typically reflect the essence of the sample better than the original dataset. Furthermore, the features extracted from the low level are primarily low-level features such as color and edges; the features extracted from the mid-to-high levels can be texture features and relatively discriminative key features.

[0130] Optionally, in this embodiment, the first feature data includes semantic information of the first feature intensity, the second feature data includes semantic information of the second feature intensity, and the first feature intensity (corresponding to the convolution weight W) γ ) and second feature intensity (corresponding to convolution weight W) β The strengths of the first and second features are different. Optionally, the strength of the first feature is greater than the strength of the second feature; or, the strength of the second feature is greater than the strength of the first feature. For ease of explanation, the following example illustrates the case where the strength of the second feature is greater than the strength of the first feature. For example, assume the convolution weight W... γ Set the value to 0.3, and the convolution weight W... β The value is set to 0.7; correspondingly, the first feature intensity can be 0.3 and the second feature intensity can be 0.7.

[0131] For example, the first feature intensity can be represented by a weighting factor for the color feature, a weighting factor for the texture feature, and / or a weighting factor for the shape feature. Correspondingly, the second feature intensity can be represented by a weighting factor for the color feature, a weighting factor for the texture feature, and / or a weighting factor for the shape feature.

[0132] It should be noted that the representation of the first feature intensity and the second feature intensity should be consistent. For example, both the first feature intensity and the second feature intensity can be represented using the weight factor of the texture feature. For instance, assuming the first feature intensity is 0.3 and the second feature intensity is 0.7, then the weight factor of the texture feature corresponding to the semantic information (or semantic region) of the first feature intensity is 0.3, and the weight factor of the texture feature corresponding to the semantic information (or semantic region) of the second feature intensity is 0.7.

[0133] It can be understood that the larger the weight factor of the texture feature corresponding to a certain semantic region, the stronger the optimization of the texture feature of that semantic region, and the clearer the image of that semantic region.

[0134] For example, the second feature data may include preset semantic information, such as "cat," which is key information extracted from the original image. This preset semantic information has a second feature strength, such as the weight factor of the texture feature, and the second feature strength is set to 0.7. The first feature data may include all semantic information from the original image, such as "cat," "tree," "cloud," and "background." This semantic information all has a first feature strength, such as the weight factor of the texture feature, and the first feature strength is set to 0.3. In the second feature data, the second feature strength of the preset semantic information "cat" is greater than the first feature strength of "cat" in the first feature data.

[0135] In other words, by using the second weight matrix to perform convolution operation on the semantic label map of the image to be processed, the feature intensity corresponding to the preset semantic information (e.g., "cat") in the image to be processed is increased. That is, the feature region of the preset semantic information (e.g., "cat") in the second feature data is strengthened and optimized. Therefore, this is beneficial for optimizing the image clarity of the feature region corresponding to the specific semantic information (e.g., "cat") in the image to be processed.

[0136] It is understandable that the second feature data is feature data enhanced by specific semantic information of the image to be processed. The first feature data can be used as the baseline feature data. In the later stage, by fusing the first feature data with the second feature data, the image features corresponding to specific semantic information in the image to be processed can be enhanced and optimized, while ensuring that other image features in the image are not lost.

[0137] Optionally, in the embodiments of this application, the first feature data and the second feature data can be feature mapping matrices or normalized values ​​in the range of [0,1]. The specific values ​​can be determined according to actual usage requirements, and the embodiments of this application do not impose any limitations.

[0138] S102, based on the first feature data, the second feature data, and the first feature map of the image to be processed, a feature map based on different semantic information is obtained; wherein, the first feature map is the feature mapping matrix obtained after inputting the image to be processed into a two-dimensional convolutional network.

[0139] In this embodiment, the first feature map of the image to be processed can be represented as a three-dimensional matrix of size H×W×C, which may include C two-dimensional matrices of size H×W. For example, for a 256×256 RGB image to be processed, the corresponding first feature map can be represented as a three-dimensional matrix of size 256×256×3.

[0140] Optionally, in the embodiments of this application, the above step S102 may include the following possible implementation: adding the dot product result between the second feature data and the first feature map of the image to be processed to the first feature data to obtain a feature map based on different semantic information.

[0141] For example, see below. Figure 4 The above calculation process is illustrated schematically. For example... Figure 4 As shown, the image to be processed (also called the original image) is subjected to a two-dimensional convolution operation (conv2d) to obtain a first feature map (denoted as m1). Then, the second feature data β is multiplied by the first feature map m1 of the image to be processed to obtain β*m1. Then, the result of the multiplication operation β*m1 is added to the first feature data γ to obtain a feature map based on different semantic information, denoted as S. f Among them, S f =β*m1+γ.

[0142] See again Figure 4 As shown, the original image can be a reference frame image. Through image semantic segmentation, semantic labels can be obtained. Figure X It should be noted that when multiple frames of images are acquired, one of the frames can be used as a reference image for matching and alignment of the multiple frames. This reference image is the aforementioned reference frame image.

[0143] S103, the feature maps based on different semantic information and the second feature map of the image to be processed are fused to obtain the target feature map; wherein, the second feature map is the feature mapping matrix obtained after inputting the image to be processed into the U-Net neural network.

[0144] Optionally, in the embodiments of this application, step S103 may include the following possible implementation: performing pixel-by-pixel summation on the feature maps based on different semantic information and the second feature map of the image to be processed to obtain the target feature map (denoted as M). The solution of this application uses the semantic category of each pixel in the image to be processed as prior information, and after passing different semantic information through different two-dimensional convolution kernels, it concatenates them into the U-Net neural network, which can optimize image processing capabilities.

[0145] Optionally, in this embodiment of the application, before step S103 above, the image processing method based on semantic information provided in this embodiment of the application may further include: performing a two-dimensional convolution operation on the image to be processed to obtain an intermediate feature map. Then, the intermediate feature map is input into a U-Net neural network to output a second feature map.

[0146] In this embodiment of the application, the U-Net neural network can use a channel-based concat method to fuse or stitch upsampled feature maps H×W×C1 and downsampled feature maps H×W×C2 at the same scale. The resulting feature map can be represented as: H×W×(C1+C2).

[0147] For concat fusion along the channel dimension, the feature map obtained after concatenation can have more feature representations because the number of channels increases.

[0148] For example, suppose the U-Net neural network includes N downsampling processes and N upsampling processes. After inputting the intermediate feature map into the U-Net neural network, the U-Net neural network sequentially performs N downsampling processes and N upsampling processes on the intermediate feature map. The U-Net neural network uses a channel-level concat method to fuse or stitch the upsampled and downsampled feature maps at the same scale. For example, the feature map obtained from each upsampling process is concatenated with the feature map obtained from the corresponding level of downsampling process; and after feature fusion of the feature map obtained from each upsampling process and the feature map obtained from the corresponding level of downsampling process, the next upsampling process is performed based on the feature map obtained from the feature fusion. Finally, after performing the Nth upsampling process, a second feature map is obtained.

[0149] For example, such as Figure 5 As shown, assume N is 3. Figure 5 The structure shown depicts a segmentation result obtained by downsampling the original image three times and upsampling it three times. Specifically, first, a two-dimensional convolution operation is performed on the image to be processed (e.g., multiple frames of the original image) to obtain an intermediate feature map m0, which is then input into the U-Net neural network. Next, the intermediate feature map m0 is downsampled for the first time to obtain an intermediate feature map m... d1 Then, for the intermediate feature map m d1 A second downsampling is performed to obtain the intermediate feature map m. d2 Then, for the intermediate feature map m d2 A third downsampling is performed to obtain the intermediate feature map m. d3 .

[0150] Next, as Figure 5 As shown, the intermediate feature map m obtained from the third downsampling... d3 Perform the first upsampling, and then combine the result of the first upsampling with the intermediate feature map m obtained from the second downsampling. d2 Perform concat overlay to obtain intermediate feature map m u3Then, for the intermediate feature map m u3 Perform a second upsampling, and then compare the result of the second upsampling with the intermediate feature map m obtained from the first downsampling. d1 Perform concat overlay to obtain intermediate feature map m u2 Then, for the intermediate feature map m u2 A third upsampling is performed, and the result of the third upsampling is concatted with the intermediate feature map m0 to obtain the second feature map m. u1 .

[0151] Furthermore, feature maps S based on different semantic information can be... f and the second feature map m of the image to be processed u1 The feature maps are fused to obtain the target feature map M; where M = S f +m u1 .

[0152] For example, the target feature map can be a three-dimensional matrix of H×W×32, where the number of channels is 32.

[0153] For example, the target feature map can be a three-dimensional matrix of H×W×64, where the number of channels is 64.

[0154] It should be noted that the more channels a feature map has, the more accurate the semantic segmentation result and the higher the image clarity of the corresponding semantic region.

[0155] Since the proposed solution is based on feature maps with different semantic information, and fine-tunes the feature mapping matrix obtained after the U-Net neural network, it can obtain more accurate segmentation results, thereby improving the image quality of the corresponding semantic region and making the image of the corresponding semantic region clearer.

[0156] S104: Perform a two-dimensional convolution operation on the target feature map to obtain the target image.

[0157] The target image can be an H×W×3 RGB image.

[0158] For example, the target feature map can be input into a deconvolutional network to perform deconvolution operations and obtain the target image. The deconvolutional network can contain a multi-layered structure, where the lower-level outputs of the deconvolutional network can represent the approximate shape of the object, and the higher-level outputs can represent more details.

[0159] Figure 6 A schematic diagram of the overall architecture of the semantic information-based image processing method provided in this application embodiment is shown.

[0160] like Figure 6 As shown, based on the semantic labels of the image to be processed Figure X Obtain the first feature data γ (γ = W) of the image to be processed. γ X) and second feature data β (β = W) β X); where semantic tags Figure X It is a matrix that uses different labels to represent different semantic information; the first feature data γ and the second feature data β contain semantic information with different feature intensities. Further, based on the first feature data γ, the second feature data β, and the first feature map m of the image to be processed, feature maps S based on different semantic information are obtained. f (S f =β*m1+γ). Wherein, the first feature map m is the feature mapping matrix obtained after inputting the image to be processed into a two-dimensional convolutional network.

[0161] Furthermore, Figure 6 As shown, feature maps S based on different semantic information f and the second feature map m of the image to be processed u1 (Where, the second feature map is the feature mapping matrix obtained after inputting the image to be processed into the U-Net neural network) are fused to obtain the target feature map M (M = S f +m u1 Through the scheme of this application, feature maps S based on different semantic information f The feature mapping matrix obtained after the original image is processed by the U-Net neural network is fine-tuned to obtain more accurate image segmentation results.

[0162] at last, Figure 6 As shown, a two-dimensional convolution operation is performed on the target feature map M to obtain the target image.

[0163] In actual shooting, in response to the user's shooting operation, the electronic device first captures the image to be processed. Then, after optimization processing by the solution of this application, the target image is obtained. For example, the target image can be an H×W×3 RGB color image. Since the solution of this application obtains feature maps with different semantic information from the original image, and fine-tunes the feature maps (feature mapping matrix) obtained by inputting the original image into the U-Net neural network based on the feature maps with different semantic information, more accurate semantic segmentation results can be obtained. This can improve the image quality of the corresponding semantic region, making the image clarity of the corresponding semantic region higher. In other words, the specific semantic part in the target image is significantly optimized, and the texture is clearer.

[0164] For example, taking a portrait mode shooting scenario, assuming the captured image to be processed contains a person and a background, since semantic regions are usually not distinguished in this image, the feature intensity of the portrait semantic region and the feature intensity of the background semantic region are the same, and correspondingly, the image clarity of the respective semantic regions of the person and the background is consistent. This application's solution performs image optimization processing on the image to be processed, targeting different semantic information within the image. Since portrait mode requires highlighting portrait features, the image processing scheme provided in this application can be adopted, setting the feature intensity of the portrait semantic region to be greater than that of the background semantic region, enhancing the detail and texture optimization of the face region, making the texture of the face region clearer. It can be understood that the image clarity of the portrait semantic region is higher than that of the background semantic region. By adopting an image optimization solution based on semantic information, a clearer portrait image can be achieved.

[0165] The semantic information-based image processing method provided in this application uses the semantic category of each pixel in the image to be processed as prior information. Different semantic information is processed through different two-dimensional convolutional kernels and then concatenated into a U-Net neural network to optimize image processing capabilities. Because this application's solution fine-tunes the feature mapping matrix obtained after the U-Net neural network based on feature maps of different semantic information, it can obtain more accurate image segmentation results, improve the image quality of the corresponding semantic regions, and make the images of the corresponding semantic regions clearer. After optimization by this application's solution, specific semantic parts in the image are significantly optimized, and the texture is clearer.

[0166] Optionally, in the embodiments of this application, combined with Figure 2 ,like Figure 7 As shown, prior to step S101 above, the image processing method based on semantic information provided in this application embodiment may further include step S105 below.

[0167] S105, input the image to be processed into the semantic segmentation model, and use the semantic segmentation model to perform semantic segmentation on the image to be processed to obtain a semantic label map.

[0168] In this embodiment, the image to be processed is input into a semantic segmentation model to perform semantic segmentation on the original image. The label corresponding to the pixel region is determined based on the semantics in the pixel region, and the label is marked in the pixel region. In this way, each pixel region of the original image is marked with the corresponding label, thereby obtaining the semantic label map (also known as the category label map) corresponding to the original image.

[0169] Optionally, the semantic label map may include one or more semantic labels, representing one or more semantic segmentation category information. In the embodiments of this application, a mapping relationship between image features and labels can be established.

[0170] For example, semantic labels can be annotated pixel by pixel in the image to be processed to obtain a pixel-level image segmentation map, which can achieve more accurate image semantic segmentation.

[0171] In this embodiment, the semantic segmentation model may include a semantic segmentation function and a semantic labeling function. The semantic segmentation function yields a semantically segmented image, and the semantic labeling function yields a semantically labeled image. Thus, by performing semantic segmentation on the image to be processed using the semantic segmentation model, a semantically labeled image can be obtained.

[0172] For example, an RGB color image (H×W×3) or a grayscale image (H×W×1) is processed by a deep convolutional neural network to output a segmentation map, i.e., a semantic label map. This semantic label map can include the category label for each pixel.

[0173] The following is combined Figure 8 The image shows an example of semantic segmentation, which is used to illustrate semantic segmentation. First, as shown in the example diagram... Figure 8 As shown in (a) and (b), the image to be processed is input into the semantic segmentation model. The semantic segmentation model performs semantic segmentation on the image to be processed, and the output is as follows. Figure 8 The semantic segmentation image shown in (b). Further, as... Figure 8 As shown in (b) and (c), semantic labels are annotated on the semantic segmentation image to obtain the following results: Figure 8 The semantic label map shown in (c) illustrates this. Image semantic segmentation technology can identify pixel regions with the same semantic meaning (such as "cat", "tree", "cloud") in an image, thus enabling better analysis and understanding of the semantic information in the image.

[0174] In practical implementation, the semantic information-based image processing method provided in this application can be applied to scenarios where electronic devices capture images. In this application, when a user takes a photo with a mobile phone, the solution can be used to automatically optimize the captured image, improving the mobile phone's shooting effect and providing a better user experience.

[0175] For example, an electronic device can pre-set the LITE state to be enabled for one or more shooting modes (e.g., portrait mode) of the camera system, so that the camera system of the electronic device will by default use the image processing method based on semantic information provided in the embodiments of this application to optimize the captured image.

[0176] This setting is implemented internally by the system and does not require user intervention to trigger.

[0177] Taking the LITE setting for portrait mode as an example, the feature intensity of the portrait semantic region can be set to be greater than that of the background semantic region. Specifically, in a portrait mode shooting scenario (with the camera system in portrait mode), the original image will be processed as follows: feature maps with different semantic information will be obtained from the original image. Based on these feature maps, the feature maps (feature mapping matrix) obtained by inputting the original image into the U-Net neural network will be fine-tuned. This will result in more accurate semantic segmentation, thereby improving the image quality of the portrait semantic region and making it clearer.

[0178] It should be noted that this article uses setting the LITE enabled state for portrait mode as an example for illustrative purposes. It is understood that in actual implementation, the LITE enabled state can also be set for other shooting modes. Similarly, triggering the activation of this application solution by having the user set the LITE enabled state in the camera system (e.g., camera app) is also an illustrative example. The specific triggering and activation method of this application solution can also be other methods, and this application embodiment does not limit this.

[0179] Optionally, in this embodiment of the application, the timing of LITE mode can be determined based on parameters such as the contrast ratio, light intensity, and / or sensitivity (also known as exposure, which can be expressed as ISO) of the environment in which the electronic device is located, and the electronic device can automatically turn on the LITE mode when the conditions are met, without the need for user operation.

[0180] For example, in actual implementation, when the camera application of the electronic device is open, if the electronic device detects that the contrast ratio of the surrounding environment is less than or equal to a preset contrast threshold, the light intensity is greater than or equal to a preset intensity, and the ISO value is less than or equal to a preset ISO threshold, then the conditions for enabling LITE mode are met, and the electronic device can enable LITE mode. After enabling LITE mode, photos taken by the user through the camera application of the electronic device will be automatically optimized using the solution of this application, without the user's awareness. Since the solution of this application can obtain more accurate segmentation results, the image quality can be improved after optimization, resulting in higher image clarity.

[0181] Optionally, in the embodiments of this application, combined with Figure 7 ,like Figure 9 As shown, prior to step S105 above, the image processing method based on semantic information provided in this application embodiment may further include the following steps S106 and S107.

[0182] S106 continuously captures multiple frames of images using the camera of an electronic device.

[0183] S107, perform image registration on multiple frames to obtain the image to be processed.

[0184] Optionally, in the embodiments of this application, combined with Figure 9 ,like Figure 10 As shown, prior to step S104 above, the image processing method based on semantic information provided in this application embodiment may further include the following step S108.

[0185] S108 displays the target image on the screen of an electronic device.

[0186] In this embodiment of the application, in the camera application of the electronic device, assuming that the image processing function corresponding to the above-mentioned semantic information-based image processing method is in the enabled state, after the user triggers the image capture control, the electronic device can continuously capture multiple frames of images through the camera, and perform image registration on the multiple frames of images to obtain a frame of image to be processed; then, the above-mentioned semantic information-based image processing method provided in this embodiment of the application is used to perform image optimization processing to obtain the target image and improve the image clarity; then, the target image is displayed on the display screen of the electronic device.

[0187] This application employs a fine-tuned model based on a specific dataset to optimize the camera's image, resulting in clearer image output. The optimized semantic components are evident in the output image. For example, Figure 11 A comparison diagram of the optimized image and the unprocessed image of the proposed solution is shown. Figure 11 (a) shows the unprocessed image. Figure 11 Image (b) shows an image optimized using the semantic information-based image processing method provided in this application. It is evident that after optimization by the solution of this application, specific semantic components in the image are significantly improved, and the texture is clearer.

[0188] In this application, different convolutional kernels are used to incorporate semantic information, based on the shape and depth of the neural network structure. In other words, different semantic information is processed through different two-dimensional convolutional kernels and then concatenated into the U-Net neural network to optimize the image and improve its clarity.

[0189] The semantic information-based image processing method provided in this application uses the semantic category of each pixel in the image to be processed as prior information. Different semantic information is processed through different two-dimensional convolutional kernels and then concatenated into a U-Net neural network to optimize image processing capabilities. Because this application's solution fine-tunes the feature mapping matrix obtained after the U-Net neural network based on feature maps of different semantic information, it can obtain more accurate segmentation results, improve the image quality of the corresponding semantic regions, and make the image clarity of the corresponding semantic regions higher. After optimization by this application's solution, specific semantic parts in the image are significantly optimized, and the texture is clearer.

[0190] It should also be noted that in the embodiments of this application, "greater than" can be replaced with "greater than or equal to", "less than or equal to" can be replaced with "less than", or "greater than or equal to" can be replaced with "greater than", and "less than" can be replaced with "less than or equal to".

[0191] The various embodiments described herein can be independent solutions or combinations thereof based on their inherent logic, and all such solutions fall within the protection scope of this application.

[0192] It is understood that the methods and operations implemented by electronic devices in the above-described method embodiments can also be implemented by components (such as chips or circuits) that can be used in electronic devices.

[0193] The method embodiments provided in this application have been described above. The apparatus embodiments provided in this application will be described below. It should be understood that the description of the apparatus embodiments corresponds to the description of the method embodiments. Therefore, any content not described in detail can be referred to the method embodiments above. For the sake of brevity, it will not be repeated here.

[0194] The foregoing mainly describes the solutions provided by the embodiments of this application from the perspective of method steps. It is understood that, in order to achieve the above functions, the electronic device implementing this method includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of protection of this application.

[0195] This application embodiment can divide an electronic device into functional modules based on the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, other feasible division methods may exist. The following description uses the division of functional modules according to each function as an example.

[0196] Figure 12 This is a schematic block diagram of an image processing apparatus 800 based on semantic information provided in an embodiment of this application. The apparatus 800 can be used to perform the actions performed by the electronic device in the above method embodiments. The apparatus 800 includes an image feature extraction unit 810, an image feature fusion unit 820, and an image output unit 830.

[0197] The image feature extraction unit 810 is used to obtain first feature data and second feature data of the image to be processed based on the semantic label map of the image to be processed; wherein, the semantic label map is a matrix that uses different labels to represent different semantic information; the first feature data and the second feature data respectively contain different semantic information;

[0198] The image feature fusion unit 820 is used to obtain a feature map based on different semantic information based on the first feature data, the second feature data, and the first feature map of the image to be processed; wherein, the first feature map is a feature mapping matrix obtained by inputting the image to be processed into a two-dimensional convolutional network;

[0199] The image feature fusion unit 820 is also used to fuse feature maps based on different semantic information and a second feature map of the image to be processed to obtain a target feature map; wherein, the second feature map is a feature mapping matrix obtained after inputting the image to be processed into the U-Net neural network;

[0200] The image output unit 830 is used to perform two-dimensional convolution operations on the target feature map to obtain the target image.

[0201] The semantic information-based image processing apparatus provided in this application uses the semantic category of each pixel in the image to be processed as prior information. Different semantic information is processed through different two-dimensional convolutional kernels and then concatenated into a U-Net neural network to optimize image processing capabilities. Because this application's solution fine-tunes the feature mapping matrix obtained after the U-Net neural network based on feature maps of different semantic information, more accurate segmentation results can be obtained, improving the image quality of the corresponding semantic regions and resulting in higher image clarity. After optimization by this application's solution, specific semantic parts in the image are significantly optimized, and the texture is clearer.

[0202] Optionally, in some possible implementations, the image feature fusion unit 820 described above is specifically used for:

[0203] The dot product between the second feature data and the first feature map of the image to be processed is added to the first feature data to obtain the feature map based on different semantic information.

[0204] Optionally, in some possible implementations, the image feature extraction unit 810 described above is specifically used for:

[0205] The semantic label map of the image to be processed is convolved using the first weight matrix to obtain the first feature data;

[0206] The semantic label map of the image to be processed is convolved using a second weight matrix to obtain the second feature data.

[0207] The proposed solution uses the semantic category of each pixel in the image to be processed as prior information. Different semantic information is processed through different two-dimensional convolution kernels and then concatenated into the U-Net neural network, which can optimize image processing capabilities.

[0208] Optionally, in some possible implementations, the feature strength of the preset semantic information in the second feature data is greater than the feature strength of the preset semantic information in the first feature data.

[0209] Optionally, in some possible implementations, the image feature fusion unit 820 described above is specifically used for:

[0210] The target feature map is obtained by performing a pixel-by-pixel summation operation between the feature map based on different semantic information and the second feature map of the image to be processed.

[0211] Since the proposed solution is based on feature maps with different semantic information, and fine-tunes the feature mapping matrix obtained after the U-Net neural network, it can obtain more accurate segmentation results, thereby improving the image quality of the corresponding semantic region and making the image of the corresponding semantic region clearer.

[0212] Optionally, in some possible implementations, combining Figure 12 ,like Figure 13 As shown, the image processing apparatus 800 based on semantic information provided in this application embodiment may further include an image semantic segmentation unit 840;

[0213] The image semantic segmentation module 840 is used for:

[0214] The image to be processed is input into the semantic segmentation model;

[0215] The semantic segmentation model is used to perform semantic segmentation on the image to be processed to obtain the semantic label map.

[0216] Furthermore, the image feature extraction unit 810 can obtain the first feature data and the second feature data of the image to be processed based on the semantic label map of the image to be processed.

[0217] Optionally, in some possible implementations, combining Figure 13 ,like Figure 14 As shown, the semantic information-based image processing apparatus 800 provided in this application embodiment may further include an image acquisition unit 850;

[0218] The image acquisition unit 850 is used for:

[0219] Multiple frames of images are continuously acquired by the camera of the semantic information-based image processing device 800;

[0220] The multiple frames of images are registered to obtain the image to be processed.

[0221] Optionally, in some possible implementations, combining Figure 14 ,like Figure 15 As shown, the image processing method based on semantic information provided in this application embodiment may further include an image display unit 860;

[0222] The image display unit 860 is used to display the target image on the display screen of an electronic device after the image output unit 830 obtains the target feature map.

[0223] After the captured image is optimized using the method described in this application, specific semantic parts of the image are significantly improved, and the texture is clearer.

[0224] Alternatively, in some possible implementations, the image feature extraction unit 810 can also be used for:

[0225] Perform a two-dimensional convolution operation on the image to be processed to obtain an intermediate feature map;

[0226] The intermediate feature map is input into the U-Net neural network, and the second feature map is output.

[0227] Furthermore, the image feature fusion unit 820 can obtain the target feature map based on the feature maps based on different semantic information and the second feature map of the image to be processed.

[0228] Alternatively, in some possible implementations, the image feature extraction unit 810 may specifically be used for:

[0229] The intermediate feature map is input into the U-Net neural network;

[0230] The intermediate feature map is subjected to N downsampling processes and N upsampling processes sequentially through the U-Net neural network. After each upsampling process, the feature map obtained is fused with the feature map obtained from the corresponding level of downsampling process, and then the next upsampling process is performed based on the feature map obtained from the feature fusion.

[0231] After the Nth upsampling process is completed, the second feature map is obtained.

[0232] The semantic information-based image processing apparatus 800 according to the embodiments of this application can correspond to the execution of the method described in the embodiments of this application, and the above and other operations and / or functions of the units in the semantic information-based image processing apparatus 800 are respectively for implementing the corresponding process of the method, which will not be described again here for the sake of brevity.

[0233] Figure 16This is a schematic structural diagram of an electronic device 900 provided in an embodiment of this application. The electronic device 900 may include a processor 910, an external memory interface 920, an internal memory 921, a universal serial bus (USB) interface 930, a charging management module 940, a power management unit 941, a battery 942, antenna 1, antenna 2, a mobile communication module 950, a wireless communication module 960, an audio module 970, a speaker 970A, a receiver 970B, a microphone 970C, a headphone jack 970D, a sensor module 980, buttons 990, a motor 991, an indicator 992, a camera 993, a display screen 994, and a subscriber identification module (SIM) card interface 995, etc. The sensor module 980 may include a pressure sensor 980A, a gyroscope sensor 980B, a barometric pressure sensor 980C, a magnetic sensor 980D, an accelerometer sensor 980E, a distance sensor 980F, a proximity sensor 980G, a fingerprint sensor 980H, a temperature sensor 980I, a touch sensor 980J, an ambient light sensor 980K, and a bone conduction sensor 980L, etc.

[0234] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 900. In other embodiments of this application, the electronic device 900 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0235] Processor 910 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, memory, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). Different processing units may be independent devices or integrated into one or more processors. The controller may serve as the central nervous system and command center of electronic device 900. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0236] The processor 910 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 910 is a cache memory. This memory can store instructions or data that the processor 910 has just used or that are used repeatedly. If the processor 910 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 910, and thus improves the efficiency of the system.

[0237] In some embodiments, the processor 910 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc. It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation of the electronic device 900. In other embodiments of this application, the electronic device 900 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0238] The charging management module 940 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 940 receives charging input from the wired charger via a USB interface 930. In some wireless charging embodiments, the charging management module 940 receives wireless charging input via the wireless charging coil of the electronic device 900. While charging the battery 942, the charging management module 940 can also supply power to the electronic device via the power management unit 941.

[0239] The power management unit 941 connects the battery 942, the charging management module 940, and the processor 910. The power management unit 941 receives input from the battery 942 and / or the charging management module 940, providing power to the processor 910, internal memory 921, external memory, display screen 994, camera 993, and wireless communication module 960. The power management unit 941 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management unit 941 may be located within the processor 910. In other embodiments, the power management unit 941 and the charging management module 940 may be housed in the same device.

[0240] The wireless communication function of electronic device 900 can be realized through antenna 1, antenna 2, mobile communication module 950, wireless communication module 960, modem processor and baseband processor.

[0241] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 900 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0242] The mobile communication module 950 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 900. The mobile communication module 950 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 950 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 950 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 950 may be housed in the processor 910. In some embodiments, at least some functional modules of the mobile communication module 950 and at least some modules of the processor 910 may be housed in the same device.

[0243] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 970A, receiver 970B, etc.) or displays images or videos through the display screen 994. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 910 and may be housed in the same device as the mobile communication module 950 or other functional modules.

[0244] The wireless communication module 960 can provide wireless communication solutions for applications on the electronic device 900, including WLAN (such as Wi-Fi), BT, Global Navigation Satellite System (GNSS), FM, NFC, IR, or general 2.4G / 5G wireless communication technologies. The wireless communication module 960 can be one or more devices integrating at least one communication processing module. The wireless communication module 960 receives electromagnetic waves via antenna 2, modulates and filters the electromagnetic wave signals, and sends the processed signal to processor 910. The wireless communication module 960 can also receive signals to be transmitted from processor 910, modulate and amplify them, and then convert them into electromagnetic waves for radiation via antenna 2.

[0245] In some embodiments, the wireless communication module 960 can be a Wi-Fi and / or Bluetooth chip. The electronic device 900 can establish a connection with chips in electronic devices such as wireless headphones through this chip, enabling wireless communication and business processing between the electronic device 900 and other electronic devices. The Bluetooth chip typically supports BR / EDR Bluetooth and BLE.

[0246] In some embodiments, antenna 1 of electronic device 900 is coupled to mobile communication module 950, and antenna 2 is coupled to wireless communication module 960, enabling electronic device 900 to communicate with networks and other devices via wireless communication technology. Wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TDSCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. GNSS can include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0247] Electronic device 900 implements display functions through a GPU, a display screen 994, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 994 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 910 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0248] Display screen 994 is used to display images, videos, etc. Display screen 994 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 900 may include one or N displays 994, where N is a positive integer greater than 1.

[0249] Electronic device 900 can achieve shooting function through ISP, camera 993, video codec, GPU, display 994 and application processor.

[0250] The ISP (Image Signal Processor) is used to process data fed back from the camera 993. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's image sensor. The light signal is converted into an electrical signal, and the image sensor transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimizations on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be integrated into the camera 993.

[0251] Camera 993 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, electronic device 900 may include one or N cameras 993, where N is a positive integer greater than 1.

[0252] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 900 is selecting a frequency, the DSP is used to perform Fourier transforms on the frequency energy.

[0253] Video codecs are used to compress or decompress digital video. Electronic device 900 may support one or more video codecs. Thus, electronic device 900 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0254] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0255] The external memory interface 920 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 900. The external memory card communicates with the processor 910 through the external memory interface 920 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0256] Internal memory 921 can be used to store executable program code, including instructions. Processor 910 executes various functional applications and data processing of electronic device 900 by running the instructions stored in internal memory 921. Internal memory 921 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 900 (such as audio data, phonebook, etc.). Furthermore, internal memory 921 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0257] The processor 910 can be used to execute the above-described program code and call relevant modules to implement the functions of the electronic device in the embodiments of this application. For example, it can establish multiple communication links with another electronic device; when there is a preset service (such as file transfer service), it can transmit the data of the preset service with another electronic device through multiple communication links.

[0258] Electronic device 900 can implement audio functions, such as music playback and recording, through the speaker 970A, receiver 970B, microphone 970C, headphone jack 970D in audio module 970, and application processor.

[0259] The audio module 970 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 970 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 970 may be located in the processor 910, or some functional modules of the audio module 970 may be located in the processor 910.

[0260] The speaker 970A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. Electronic device 900 can listen to music or make hands-free calls through the speaker 970A.

[0261] The receiver 970B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 900 answers a telephone call or voice message, the receiver 970B can be brought close to the listener's ear to hear the voice.

[0262] Microphone 970C, also known as a "microphone," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 970C, inputting the sound signal into microphone 970C. Electronic device 900 may have at least one microphone 970C. In some embodiments, electronic device 900 may have two microphones 970C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 900 may also have three, four, or more microphones 970C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0263] The 970D headphone jack is used to connect wired headphones. The 970D headphone jack can be a USB 930 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0264] Pressure sensor 980A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 980A can be disposed on display screen 994. There are many types of pressure sensors 980A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 980A, the capacitance between the electrodes changes. Electronic device 900 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 994, electronic device 900 detects the touch operation intensity based on pressure sensor 980A. Electronic device 900 can also calculate the touch position based on the detection signal from pressure sensor 980A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example: when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0265] The gyroscope sensor 980B can be used to determine the motion attitude of the electronic device 900. In some embodiments, the gyroscope sensor 980B can determine the angular velocity of the electronic device 900 around three axes (e.g., the x, y, and z axes). The gyroscope sensor 980B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 980B detects the angle of the electronic device 900's shake, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the electronic device 900 through reverse movement, thus achieving image stabilization. The gyroscope sensor 980B can also be used in navigation and motion-sensing gaming scenarios.

[0266] The 980E accelerometer can detect the magnitude of acceleration of an electronic device 900 in various directions (typically three axes). When the electronic device 900 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of the electronic device, and is applicable to screen orientation switching, pedometers, and other applications.

[0267] The distance sensor 980F is used to measure distance. The electronic device 900 can measure distance via infrared or laser. In some embodiments, during a shooting scene, the electronic device 900 can utilize the distance sensor 980F to measure distance for rapid focusing.

[0268] The proximity sensor 980G may include, for example, a light-emitting diode (LED) and a photosensor, such as a photodiode. The LED may be an infrared LED. The electronic device 900 emits infrared light outward through the LED. The electronic device 900 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that an object is near the electronic device 900. When insufficient reflected light is detected, the electronic device 900 can determine that no object is near the electronic device 900. The electronic device 900 can use the proximity sensor 980G to detect when a user holds the electronic device 900 close to their ear for a phone call, so as to automatically turn off the screen to save power. The proximity sensor 980G can also be used in holster mode and pocket mode for automatic unlocking and locking of the screen.

[0269] The ambient light sensor 980K is used to detect ambient light levels. Electronic device 900 can adaptively adjust the brightness of its display screen 994 based on the detected ambient light level. The ambient light sensor 980K can also be used to automatically adjust the white balance when taking photos. The ambient light sensor 980K can also work in conjunction with the proximity sensor 980G to detect whether electronic device 900 is in a pocket, preventing accidental touches.

[0270] The barometric pressure sensor 980C is used to measure air pressure. In some embodiments, the electronic device 900 calculates altitude using the air pressure value measured by the barometric pressure sensor 980C to assist in positioning and navigation.

[0271] The magnetic sensor 980D includes a Hall sensor. The electronic device 900 can use the magnetic sensor 980D to detect its displacement. In some embodiments, the Hall sensor can utilize a magnet to form a linear trapezoidal magnetic field (or a ramp magnetic field). The displacement change of the Hall plate in the linear magnetic field corresponds to the change in magnetic field strength, and the resulting Hall potential is proportional to the displacement. By acquiring the Hall potential, the electronic device 900 can measure the magnitude of the displacement.

[0272] The fingerprint sensor 980H is used to collect fingerprints. The electronic device 900 can use the characteristics of the collected fingerprints to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0273] Temperature sensor 980I is used to detect temperature. In some embodiments, electronic device 900 uses the temperature detected by temperature sensor 980I to execute a temperature handling strategy. For example, when the temperature reported by temperature sensor 980I exceeds a threshold, electronic device 900 performs thermal protection by reducing the performance of a processor located near temperature sensor 980I to reduce power consumption. In other embodiments, when the temperature is below another threshold, electronic device 900 heats battery 942 to prevent abnormal shutdown of electronic device 900 due to low temperature. In still other embodiments, when the temperature is below yet another threshold, electronic device 900 boosts the output voltage of battery 942 to prevent abnormal shutdown due to low temperature.

[0274] Touch sensor 980J, also known as a "touch panel," can be located on display screen 994. The touch sensor 980J and display screen 994 together form a touchscreen, also known as a "touch screen." Touch sensor 980J detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 994. In other embodiments, touch sensor 980J may also be located on the surface of electronic device 900, in a different position than display screen 994.

[0275] The bone conduction sensor 980L can acquire vibration signals. In some embodiments, the bone conduction sensor 980L can acquire vibration signals from the vibrating bone segments of the human vocal cords. The bone conduction sensor 980L can also contact the human pulse to receive blood pressure signals. In some embodiments, the bone conduction sensor 980L can also be incorporated into headphones to form bone conduction headphones. The audio module 970 can parse the voice signals from the vibrating bone segments of the vocal cords acquired by the bone conduction sensor 980L to realize voice functionality. The application processor can parse heart rate information from the blood pressure signals acquired by the bone conduction sensor 980L to realize heart rate detection functionality.

[0276] Buttons 990 include a power button, volume buttons, etc. Buttons 990 can be mechanical buttons or touch-sensitive buttons. Electronic device 900 can receive button input and generate key signal inputs related to user settings and function control of electronic device 900.

[0277] Motor 991 can generate vibration alerts. Motor 991 can be used for incoming call vibration alerts and for touch vibration feedback. For example, different vibration feedback effects can be corresponding to touch operations applied to different applications (such as taking photos, playing audio, etc.). Motor 991 can also correspond to different vibration feedback effects for touch operations applied to different areas of the display screen 994. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0278] Indicator 992 can be an indicator light, used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0279] The SIM card interface 995 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 995 to establish contact with the electronic device 900. The electronic device 900 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 995 can support Nano SIM cards, Micro SIM cards, and other SIM cards. Multiple cards can be inserted into the same SIM card interface 995 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 995 is also compatible with different types of SIM cards. The SIM card interface 995 is also compatible with external memory cards. The electronic device 900 interacts with the network through the SIM card to achieve functions such as voice calls and data communication. In some embodiments, the electronic device 900 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 900 and cannot be separated from it.

[0280] The electronic device 900 can be a mobile terminal or a non-mobile terminal. For example, the electronic device 900 can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle terminal, wearable device, ultra-mobile personal computer (UMPC), netbook or personal digital assistant (PDA), wireless headset, wireless bracelet, wireless smart glasses, wireless watch, augmented reality (AR) / virtual reality (VR) device, desktop computer, smart home appliance (e.g., television, speaker, refrigerator, air purifier, air conditioner, rice cooker), etc. The electronic device 900 can also be collectively referred to as an Internet of Things (IoT) device. This application embodiment does not specifically limit the device type of the electronic device 900.

[0281] It should be understood that Figure 16 The electronic device 900 shown can correspond to Figure 12 , Figure 13 , Figure 14 and Figure 15 The image processing device 800 based on semantic information is shown.

[0282] in, Figure 16 The processor 910 in the illustrated electronic device 900 can correspond to Figure 12 The image feature extraction unit 810, the image feature fusion unit 820, and the image output unit 830 in the semantic information-based image processing device 800.

[0283] in, Figure 16 The processor 910 in the illustrated electronic device 900 can correspond to Figure 13 , Figure 14 and Figure 15 The image processing apparatus 800 based on semantic information includes an image semantic segmentation unit 840, an image feature extraction unit 810, an image feature fusion unit 820, and an image output unit 830.

[0284] in, Figure 16 The camera 993 in the electronic device 900 shown can correspond to Figure 14 and Figure 15 The image acquisition unit 850 in the semantic information-based image processing device 800.

[0285] in, Figure 16 The display screen 994 in the electronic device 900 shown can correspond to Figure 15 The image display unit 860 in the semantic information-based image processing apparatus 800.

[0286] In actual implementation, when the electronic device 900 is running, the processor 910 executes computer execution instructions in the memory 921 to perform the operation steps of the above method through the electronic device 900.

[0287] Optionally, in some embodiments, this application provides a chip coupled to a memory, the chip being used to read and execute computer programs or instructions stored in the memory to perform the methods described in the above embodiments.

[0288] Optionally, in some embodiments, this application provides an electronic device including a chip for reading and executing computer programs or instructions stored in a memory, such that the methods in the various embodiments are performed.

[0289] Optionally, in some embodiments, this application also provides a computer-readable storage medium storing program code that, when run on a computer, causes the computer to perform the methods described in the above embodiments.

[0290] Optionally, in some embodiments, this application also provides a computer program product including computer program code, which, when run on a computer, causes the computer to perform the methods described in the above embodiments.

[0291] In this embodiment, the electronic device includes a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on the operating system layer. The hardware layer may include hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory (also called main memory). The operating system layer can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer may include applications such as browsers, address books, word processing software, and instant messaging software.

[0292] This application does not impose any particular limitation on the specific structure of the execution subject of the method provided in this application embodiment. As long as it is possible to communicate according to the method provided in this application embodiment by running a program that records the code of the method provided in this application embodiment. For example, the execution subject of the method provided in this application embodiment can be an electronic device, or a functional module in an electronic device that can call and execute a program.

[0293] Various aspects or features of this application can be implemented as methods, apparatus, or articles of manufacture using standard programming and / or engineering techniques. As used herein, the term "article of manufacture" can encompass a computer program accessible from any computer-readable device, carrier, or medium. For example, computer-readable media can include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes), optical discs (e.g., compact discs (CDs), digital versatile discs (DVDs), etc.), smart cards, and flash memory devices (e.g., erasable programmable read-only memory (EPROMs), cards, sticks, or key drives, etc.).

[0294] The various storage media described herein may represent one or more devices and / or other machine-readable media used for storing information. The term "machine-readable media" may include, but is not limited to, wireless channels and various other media capable of storing, containing and / or carrying instructions and / or data.

[0295] It should be understood that the processor mentioned in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0296] It should also be understood that the memory mentioned in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM). For example, RAM can be used as an external cache. By way of example and not limitation, RAM can include a variety of forms, such as: static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0297] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, the memory (storage module) can be integrated into the processor.

[0298] It should also be noted that the memory described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0299] Those skilled in the art will recognize that the units and steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of protection of this application.

[0300] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0301] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or units may be electrical, mechanical, or other forms.

[0302] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0303] In addition, the functional units in the various embodiments of this application can be integrated into one unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0304] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium and includes several instructions that cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0305] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0306] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method based on semantic information, characterized in that, include: Based on the semantic label map of the image to be processed, first feature data and second feature data are obtained; wherein, the semantic label map corresponds to the image to be processed, and different semantic labels are used in the semantic label map to identify different semantic regions in the image to be processed; the first feature data includes semantic information of a first feature intensity, and the second feature data includes semantic information of a second feature intensity, wherein the first feature intensity and the second feature intensity are different; The dot product between the second feature data and the first feature map of the image to be processed is added to the first feature data to obtain feature maps based on different semantic information; wherein, the first feature map is obtained through a convolutional network and is used to identify the basic image features of the image to be processed. The feature map based on different semantic information is summed pixel by pixel with the second feature map to obtain the target feature map; the target image is obtained based on the target feature map; wherein, the second feature map is obtained by a neural network that supports image semantic segmentation and is used to identify the semantic segmentation features of the image to be processed.

2. The method according to claim 1, characterized in that, The first feature map is a feature mapping matrix obtained by inputting the image to be processed into a two-dimensional convolutional network.

3. The method according to claim 1, characterized in that, The second feature map is the feature mapping matrix obtained after inputting the image to be processed into the U-Net neural network.

4. The method according to claim 1, characterized in that, The semantic label map based on the image to be processed is used to obtain first feature data and second feature data, including: The semantic label map of the image to be processed is convolved using the first weight matrix to obtain the first feature data; The semantic label map of the image to be processed is convolved using a second weight matrix to obtain the second feature data.

5. The method according to claim 1, characterized in that, The second feature intensity is greater than the first feature intensity, and the semantic information of the second feature intensity corresponds to the target feature in the image to be processed.

6. The method according to any one of claims 1 to 5, characterized in that, Obtaining the target image based on the target feature map includes: The target image is obtained by performing a two-dimensional convolution operation on the target feature map.

7. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Perform a two-dimensional convolution operation on the image to be processed to obtain an intermediate feature map; The intermediate feature map is input into the U-Net neural network, and the output is the second feature map.

8. The method according to claim 7, characterized in that, The step of inputting the intermediate feature map into the U-Net neural network and outputting the second feature map includes: The intermediate feature map is input into the U-Net neural network; The intermediate feature map is subjected to N downsampling processes and N upsampling processes sequentially through the U-Net neural network. After each upsampling process, the feature map obtained is fused with the feature map obtained from the corresponding level of downsampling process, and then the next upsampling process is performed based on the feature map obtained from the feature fusion. After the Nth upsampling process is completed, the second feature map is obtained.

9. The method according to any one of claims 1 to 5, characterized in that, Before obtaining the first feature data and the second feature data based on the semantic label map of the image to be processed, the method further includes: The image to be processed is input into the semantic segmentation model; The semantic segmentation model is used to perform semantic segmentation on the image to be processed to obtain the semantic label map.

10. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Multiple frames of images are continuously captured using the camera of an electronic device; The multiple frames of images are registered to obtain the image to be processed.

11. The method according to any one of claims 1 to 5, characterized in that, After obtaining the target image, the method further includes: The target image is displayed on the screen of an electronic device.

12. An electronic device, characterized in that, The device includes a processor coupled to a memory, the processor being configured to execute a computer program or instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1 to 11.

13. A chip system, characterized in that, The chip system is coupled to a memory, and the chip system is used to read and execute a computer program stored in the memory to implement the method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 11.

Images