Image processing method, device, apparatus and storage medium

Abstract

Embodiments of the present disclosure provide an image processing method, device and equipment and storage medium. In determining the category to which each pixel point in a target image belongs, the target image can be first detected to obtain prediction position information of objects of different categories in the target image, at least one second feature can be determined according to the prediction position information, each second feature corresponds to one first feature extracted from the target image, a fusion feature is obtained based on the fusion of the first feature and the second feature, and then the category to which each pixel point in the target image belongs is determined based on the fusion feature. Wherein, the position of each object in the target image can be roughly determined according to the prediction position information, and the category to which each pixel point belongs is determined by combining the prediction position information to assist in determining the category information of each pixel point, and the category to which each pixel point belongs is more accurate.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to an image processing method, apparatus, device, and storage medium. Background Technology

[0002] In certain applications, it's necessary to determine the category of each pixel in an image (e.g., semantic segmentation). For instance, it might be necessary to determine whether a pixel belongs to a person, the sky, or grass before proceeding with further image processing based on that category. Accurately determining the category of each pixel in the image is a prerequisite for improving the accuracy of subsequent processing. Therefore, it's necessary to provide a scheme that can accurately determine the category of each pixel in the image to be processed. Summary of the Invention

[0003] This disclosure provides an image processing method, apparatus, device, and storage medium.

[0004] According to a first aspect of the present disclosure, an image processing method is provided, the method comprising:

[0005] Feature extraction is performed on the acquired target image to obtain at least one first feature, wherein the first feature is related to the object category to which each pixel in the target image belongs;

[0006] Based on the predicted location information of objects of different object categories in the target image, at least one second feature is obtained; wherein each second feature corresponds to a first feature, and the predicted location information is obtained by performing object detection on the objects in the target image;

[0007] Based on the fused feature obtained by fusing the first feature and the second feature, the object category to which each pixel in the target image belongs is determined.

[0008] According to a second aspect of the present disclosure, a method for training a semantic segmentation model is provided, the method comprising:

[0009] Obtain a sample image carrying a first label, wherein the first label is used to indicate the object category to which each pixel in the sample image belongs;

[0010] Based on the first label, determine the prior location information corresponding to objects of different object categories in the sample image;

[0011] Using a preset initial model based on the sample image and the prior location information, the object category of each pixel in the sample image is predicted to obtain the prediction result;

[0012] Based on the prediction results and the target loss determined by the first label, the initial model is iteratively optimized to obtain the semantic segmentation model.

[0013] According to a third aspect of the present disclosure, an image processing apparatus is provided, the apparatus comprising:

[0014] The first feature extraction module is used to extract features from the acquired target image to obtain at least one first feature, wherein the first feature is related to the object category to which each pixel in the target image belongs.

[0015] The second feature extraction module is used to obtain at least one second feature based on the predicted location information of objects of different object categories in the target image; wherein each second feature corresponds to a first feature, and the predicted location information is obtained by performing target detection on the objects in the target image;

[0016] The classification module is used to determine the object category of each pixel in the target image based on the fused feature obtained by fusing the first feature and the second feature.

[0017] According to a fourth aspect of the present disclosure, an electronic device is provided, the electronic device including a processor, a memory, and computer instructions stored in the memory that are executable by the processor, wherein when the processor executes the computer instructions, it can implement the method mentioned in the first aspect above.

[0018] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing computer instructions that, when executed, implement the method mentioned in the first aspect above.

[0019] In this embodiment of the disclosure, when determining the category of each pixel in a target image, at least one first feature can be extracted from the target image. This first feature is related to the object category of the pixel in the target image. Target detection can be performed on the target image to obtain predicted location information of objects of different categories in the target image. Then, multiple second features can be extracted from the target image based on this predicted location information, with each second feature corresponding to one first feature. Finally, the category of each pixel in the target image can be determined based on the fused feature obtained by fusing the first and second features. Specifically, the predicted location information can roughly determine the position of each object in the target image. The second features determined based on the predicted location information can characterize the weight of each first feature. By adjusting the weight of the first features extracted from the target image in conjunction with the predicted location information, the first features are corrected, so that the final fused feature can more accurately reflect the category information of the pixel, and thus the category of each pixel determined based on the fused feature is also more accurate.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0022] Figure 1 This is a flowchart of an image processing method according to an embodiment of the present disclosure.

[0023] Figure 2 This is a schematic diagram of an image processing method according to an embodiment of the present disclosure.

[0024] Figure 3 This is a schematic diagram of a training semantic segmentation model according to an embodiment of this disclosure.

[0025] Figure 4 This is a schematic diagram of a training target detection model according to an embodiment of the present disclosure.

[0026] Figure 5(a) is a schematic diagram of the maximum bounding rectangle of an image region of an object according to an embodiment of the present disclosure.

[0027] Figure 5(b) is a schematic diagram of a second detection box obtained by expanding the maximum bounding rectangle of the image region of the object according to an embodiment of the present disclosure.

[0028] Figure 6 This is a schematic diagram of an embodiment of the present disclosure in which an overlapping region is assigned to a second detection box.

[0029] Figure 7 This is a schematic diagram of an image processing method according to an embodiment of the present disclosure.

[0030] Figure 8 This is a schematic diagram of the logical structure of an image processing apparatus according to an embodiment of the present disclosure.

[0031] Figure 9 This is a schematic diagram of the logical structure of a device according to an embodiment of the present disclosure. Detailed Implementation

[0032] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0033] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. Additionally, the term “at least one” herein means any combination of at least two of any one or more of a plurality.

[0034] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0035] To enable those skilled in the art to better understand the technical solutions in the embodiments of this disclosure, and to make the above-mentioned objectives, features and advantages of the embodiments of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings.

[0036] In some applications, it's necessary to determine the category of each pixel in an image (e.g., semantic segmentation). For example, suppose there's an image to be processed, and we need to determine whether each pixel belongs to a person, the sky, or grass. Then, based on the category of each pixel, we can perform subsequent processing on the image, such as applying different color corrections to the person, sky, and grass regions respectively. Currently, to determine the category of each pixel in an image, we can first train a neural network in a supervised manner using a large number of labeled sample images. Then, we can use the trained neural network to predict the category of each pixel in the target image. Although this method can generally accurately predict the category of each pixel, the accuracy of the prediction results still needs further improvement.

[0037] Based on this, embodiments of this disclosure provide an image processing method. When determining the category of each pixel in a target image, target detection can be performed on the target image first to obtain predicted location information corresponding to objects of different categories in the target image. This predicted location information is then used to assist in the classification of pixels in the target image. For example, based on the predicted location information, second features corresponding to multiple first features extracted from the target image can be determined. These second features are features related to the location of each object and can essentially be considered as weights of the first features. Then, a fused feature can be obtained by fusing the multiple first features and the corresponding second features. Finally, the category of each pixel in the target image is determined based on the fused feature. Specifically, the predicted location information can roughly determine the position of each object in the image. By combining the predicted location information to adjust the weights of the first features extracted from the target image, the final fused feature can more accurately reflect the category information of each pixel, thus making the category of each pixel determined based on the fused feature more accurate.

[0038] The image processing method provided in this disclosure can be executed by various electronic devices, such as mobile phones, cameras, computers, cloud servers, etc., and there are no limitations on its implementation.

[0039] The following combination Figure 1 and Figure 2 The image processing method provided in the embodiments of this disclosure will be described, wherein, Figure 1 This is a flowchart of an image processing method provided in an embodiment of the present disclosure. Figure 2 This is a schematic diagram of an image processing method provided in an embodiment of this disclosure. Figure 1 As shown, the image processing method provided in this application may include the following steps:

[0040] S102. Perform feature extraction on the acquired target image to obtain multiple first features, wherein the first features are related to the category to which each pixel in the target image belongs;

[0041] In step S102, feature extraction can be performed on the acquired target image to obtain multiple first features. The target image can be any image to which the category of pixels in the image needs to be determined; the target image can be a color image or a grayscale image, and this application does not impose any restrictions.

[0042] Feature extraction of the target image can be achieved through a pre-trained feature extraction network, such as ResNet or AlexNet. Other methods are also possible, and this embodiment is not limited to any particular method. The extracted first feature is a feature related to the category of a pixel in the target image, which can be used to determine the category of the pixel in the target image. The first feature can be represented by a matrix, vector, or other forms, and this embodiment is not limited to any particular form.

[0043] S104. Based on the predicted location information of objects of different object categories in the target image, at least one second feature is obtained; wherein each second feature corresponds to a first feature, and the predicted location information is obtained by performing target detection on the objects in the target image;

[0044] In step S104, predicted location information corresponding to different categories of objects in the target image can be determined. This predicted location information can be various information used to indicate the position of each object in the target image. For example, the predicted location information can be the detection box corresponding to each object. The detection box can be a rectangle or other shapes, as long as it can roughly identify the position of each object in the target image. For example, if the detection box is a rectangle, the location information can be the pixel coordinates of the four vertices of the rectangle. If the detection box is a circle, the location information can be the pixel coordinates of the center and the radius of the circle.

[0045] The predicted location information can be determined in advance or in real time. Object detection can be performed on the target image to obtain detection boxes corresponding to each object, and then the location information of these detection boxes can be determined as the predicted location information. The predicted location information for different objects can be determined through a pre-trained object detection model or through other methods.

[0046] After obtaining the predicted location information, at least one second feature can be determined from the target image based on the predicted location information. Each second feature corresponds to a first feature, and each second feature represents the weight of the corresponding first feature. Since the predicted location information reflects the approximate location of each object in the image, the pixel regions corresponding to objects belonging to different categories in the image can be preliminarily determined based on the predicted location information. Furthermore, the reliability of the extracted multiple first features can be determined based on this prior information, thereby determining the weights corresponding to each of the multiple first features, i.e., the second features. In some embodiments, the determination of the second features corresponding to each of the multiple first features using the predicted location information can be achieved through a pre-trained neural network.

[0047] S106. Based on the fused feature obtained by fusing the first feature and the second feature, determine the object category to which each pixel in the target image belongs.

[0048] After obtaining the second feature, a fused feature can be obtained by fusing the first and second features. Then, the category of each pixel in the target image can be determined based on the fused feature. Further processing of the target image can then be performed based on the category of each pixel.

[0049] Since the second feature corresponding to each first feature represents the weight of that first feature, the first features can be fused according to the determined weights to obtain fused features. By incorporating the second features extracted based on the predicted location information of each object into the first features, the fused features obtained from the fusion of the two can more accurately reflect the category information of each pixel in the image, thus making the category of each pixel in the target image determined by the fused features more accurate.

[0050] In some embodiments, the image processing method described above can be executed using a pre-trained semantic segmentation model. For example... Figure 3 As shown, by inputting the target image and the predicted location information corresponding to different categories of objects in the target image into the semantic segmentation model, the semantic segmentation model can output the category to which each pixel in the target image belongs.

[0051] The training process of this semantic segmentation model is as follows: Figure 3 As shown, a large number of sample images carrying a first label can be acquired. The first label indicates the category to which each pixel in the sample image belongs. The first label can be obtained by the user annotating the category to which each pixel in the sample image belongs. After acquiring the sample images, prior location information corresponding to different categories of objects in the sample images can be determined based on the first label. This prior location information can be various information indicating the position of each object in the sample image; for example, the prior location information can be the detection box corresponding to each object. Similarly, the detection box corresponding to each object can be a rectangle or other shapes, as long as it can identify the approximate position of each object in the sample image. Since the first label indicates the category to which each pixel belongs, the image region corresponding to the set of pixels belonging to each category can be determined based on the first label. Furthermore, the prior location information corresponding to different categories of objects can be determined based on this image region.

[0052] After determining the prior location information corresponding to each object, the sample image and prior location information can be input into the preset initial model. The model can obtain the prediction result based on the sample image and prior location information, that is, the probability of each pixel in the sample image belonging to each preset object category. Then, the target loss can be determined based on the difference between the prediction result output by the model and the first label. Based on the target loss, the model parameters are continuously adjusted to train the semantic segmentation model.

[0053] In some embodiments, an object detection model can be pre-trained to predict the predicted location information of objects of different categories in a target image. For example... Figure 4 The diagram illustrates the training process of an object detection model. During training, a large number of sample images carrying a first label are acquired. This first label indicates the category to which each pixel in the sample image belongs. The first label can be obtained by the user annotating the category of each pixel in the sample image. Taking the prior location information represented by the location information of the detection boxes corresponding to each object as an example, after acquiring the sample images, the detection boxes corresponding to different categories of objects in the sample images can be determined based on the first label. Since the first label indicates the category to which each pixel belongs, the image region corresponding to the set of pixels belonging to each category can be determined based on the first label. Furthermore, the detection boxes corresponding to different categories of objects can be determined based on this image region.

[0054] After obtaining the detection boxes corresponding to different categories of objects in the sample image, the position information of the detection boxes in the sample image can be used as the prior position information, and this prior position information can be used as the second label of the sample image. Then, the object detection model is trained using the sample image and the second label. For example, the sample image can be input into a preset initial model, and the position information of each object output by the model can be used. Then, the difference between the position information output by the model and the position information indicated by the second label can be used as the target loss to continuously adjust the parameters in the model to train the object detection model.

[0055] In some embodiments, when determining the prior location information corresponding to different categories of objects in a sample image based on a first label, for each object in the sample image, the image region of the object in the sample image can be determined according to the first label, and then a detection box corresponding to the object can be determined based on the image region, wherein the detection box can cover the image region. For example, the image region composed of the object's pixels can be determined, and a rectangular, circular, or polygonal box that can cover the image region can be determined based on the image region. Then, the determined rectangular, circular, or polygonal box is used as the detection box corresponding to the object. After determining the detection boxes corresponding to each object, the location information of the detection boxes can be used as the prior location information.

[0056] In some embodiments, as shown in FIG5(a), when determining the detection box corresponding to each object based on the image region of each object in the sample image, the minimum bounding rectangle corresponding to the image region can be determined, and then the minimum bounding rectangle can be used as the detection box corresponding to each object. By determining the minimum bounding rectangle, it can be ensured that the detection box can cover the object, while not including too many regions corresponding to other objects.

[0057] In some embodiments, as shown in FIG5(b), in order to ensure that the detection box corresponding to each object can completely cover the image area of ​​the object, after determining the minimum bounding rectangle, the minimum bounding rectangle can be expanded around it according to a preset ratio to obtain the target rectangle, and then the target rectangle is used as the detection box.

[0058] In some embodiments, such as Figure 6 As shown, a sample image may contain multiple objects of different categories, and the detection boxes corresponding to these objects may overlap. Since each image region can only belong to one category, for multiple detection boxes with overlapping regions, the overlapping region can be assigned to one of the detection boxes, and the size of the other detection boxes within those boxes can be reduced until the overlapping region is outside the other detection boxes. In other words, after assigning the overlapping region to one detection box, the overlapping portion can be removed from the other detection boxes, ensuring that the final detection boxes corresponding to each object do not overlap.

[0059] In some embodiments, when assigning an overlapping region to one of the detection boxes, the distance between the center of each detection box and the center of the overlapping region can be determined for multiple detection boxes with overlapping regions. Then, the overlapping region is assigned to the detection box with the smallest distance. By assigning the overlapping region to the closer detection boxes, the most accurate allocation of detection boxes can be ensured. Figure 6 As shown, the center of the overlapping region is closest to the center of the middle rectangle. Therefore, the overlapping region is assigned to the middle rectangle, and then the left rectangle is reduced so that the overlapping region is outside the rectangle, thus obtaining the detection box corresponding to each object in the sample image.

[0060] In some embodiments, when obtaining at least one second feature based on the predicted location information corresponding to different categories of objects in the target image, a mask image can be determined first based on the predicted location information. The mask image can be considered as a rough label information of the target image, used to indicate the category of each pixel. Pixels belonging to the same object category in the mask image have the same pixel value, while pixels belonging to different object categories have different pixel values. Taking the predicted location information as represented by the location information of the detection boxes corresponding to the objects as an example, objects within the same detection box represent the same category, and therefore can be identified by a single pixel value. Different detection boxes represent objects of different categories, and therefore can be distinguished by setting different pixel values. After obtaining the mask image, max pooling and average pooling can be performed on the mask image, and the feature A obtained from max pooling and the feature B obtained from average pooling are concatenated to obtain the concatenated feature AB. Convolution processing is then performed on the concatenated feature to obtain at least one extracted second feature. Max pooling and average pooling can be performed in parallel. By extracting features from the predicted location information of each object in the target image and transforming it into valuable information to assist in classifying each pixel in the target image, more accurate classification results can be obtained.

[0061] In some embodiments, when determining the object category of each pixel in the target image based on the fused feature obtained by fusing the first feature and the second feature, each first feature can be multiplied by its corresponding second feature, and the result of the multiplication can be convolved. Then, the features obtained by the convolution process can be fused to obtain the fused feature. For example, the features obtained after the convolution process can be input into a pre-constructed FPN (Feature Pyramid Networks). The features at each level can be fused through the FPN to obtain the fused feature. Then, the fused feature can be decoded to obtain the object category of each pixel in the target image.

[0062] Furthermore, embodiments of this disclosure also provide a method for training a semantic segmentation model, which may include the following steps:

[0063] Obtain a sample image carrying a first label, wherein the first label is used to indicate the object category to which each pixel in the sample image belongs;

[0064] Based on the first label, determine the prior location information corresponding to objects of different object categories in the sample image;

[0065] Using a preset initial model based on the sample image and the prior location information, the object category of each pixel in the sample image is predicted to obtain the prediction result;

[0066] Based on the prediction results and the target loss determined by the first label, the initial model is iteratively optimized to obtain the semantic segmentation model.

[0067] The specific implementation details of the training method for the semantic segmentation model can be found in the description in the above embodiments, and will not be repeated here.

[0068] To further explain the image processing method provided in the embodiments of this disclosure, the following explanation is based on a specific embodiment.

[0069] To more accurately predict the category of each pixel in an image, this embodiment trains a semantic segmentation model. This model can more accurately determine the category of each pixel in an image. The entire method includes a training phase and an inference phase, which are described below. Figure 7 These two stages will be explained separately.

[0070] 1. Model Training Phase

[0071] The model training phase can utilize sample images carrying the first label to train a semantic segmentation model for predicting the category of each pixel in the image, and an object detection model for predicting the location information of the detection boxes corresponding to different categories of objects in the image. Specifically, this includes the following steps:

[0072] (1) Determine the second detection box corresponding to different categories of objects in the sample image.

[0073] A large number of sample images carrying a first label can be acquired. The first label indicates the category to which each pixel in the sample image belongs. For each object in the sample image, the image region corresponding to the object in the sample image can be determined based on the first label, and the minimum bounding rectangle corresponding to the image region can be determined. Then, the minimum bounding rectangle is expanded around the object according to a certain ratio to obtain the second detection box corresponding to the object.

[0074] If there is an overlapping area between the second detection boxes corresponding to each object, then for multiple second detection boxes with overlapping areas, the distance between the center of the overlapping area and the center of the multiple second detection boxes can be determined, the overlapping area is assigned to the second detection box with the smallest distance, and the other second detection boxes are reduced, and the overlapping area is removed from the other second detection boxes, so as to obtain the final second detection box corresponding to each second object. These second detection boxes do not overlap.

[0075] (2) Training of the semantic segmentation model

[0076] After determining the second detection boxes corresponding to different categories of objects in the sample image, the sample image and the location information of the second detection boxes in the sample image can be input into a preset initial model. This initial model may include a backbone network, such as a ResNet network, which extracts features from the sample image to obtain multiple features. Furthermore, the initial model may also include an attention mechanism network. This attention mechanism network further processes the location information of the second detection boxes, transforming it into valuable information for model training. For example, this attention mechanism network can perform max pooling and average pooling on the mask image obtained based on the location information of the second detection boxes in parallel. The two pooled features are concatenated, upsampled to a certain size, and fed into activation functions such as sigmoid. A series of convolutions are then performed to obtain weights corresponding to the multiple features. These weights are then used to weight and fuse the multiple features extracted by the backbone network to obtain the final features. Based on these final features, the category of each pixel in the sample image is predicted. Then, the model parameters can be continuously adjusted based on the prediction results of the sample images and the differences between the first labels corresponding to the sample images to train a semantic segmentation model.

[0077] (3) Training of the target detection model

[0078] Since the location information of the detection boxes of each object in the image can be simultaneously input into the trained semantic segmentation model to predict the category of each pixel, thus assisting in determining the category of each pixel, the aforementioned sample images can be reused to train an object detection model to determine the detection boxes corresponding to each object in the image. For example, after determining the second detection boxes corresponding to each object in the sample image, the location information of the second detection boxes can be used as the second label of the sample image. A pre-defined initial model can then be trained using the sample images carrying this second label to obtain the object detection model. This initial model can be a network such as Faster R-CNN or RetinaNet. Alternatively, the sample images can be input into the initial model, and the model parameters can be continuously adjusted based on the difference between the location information of the second detection boxes corresponding to each object output by the model and the label, thus obtaining the object detection model.

[0079] 2. Model Reasoning Stage

[0080] (1) Use the object detection model to determine the position information of the first detection box corresponding to each object in the target image to be processed.

[0081] After obtaining the target image to be processed, the target image can be input into the target detection model, and the target detection model can output the position information of the first detection box corresponding to each object in the target image.

[0082] (2) Use semantic segmentation models to determine the category of each pixel in the target image.

[0083] After determining the location information of the first detection box corresponding to each object in the target image, the location information and the target image can be input into the semantic segmentation model, and the semantic segmentation model can output the category to which each pixel in the target image belongs.

[0084] It is easy to understand that the solutions described in the above embodiments can be combined when there is no conflict, and not all of them will be listed in this disclosure.

[0085] Accordingly, embodiments of this disclosure also provide an image processing apparatus, such as... Figure 8 As shown, the device includes:

[0086] The first feature extraction module 81 is used to extract features from the acquired target image to obtain at least one first feature, wherein the first feature is related to the object category to which each pixel in the target image belongs.

[0087] The second feature extraction module 82 is used to obtain at least one second feature based on the predicted location information of objects of different object categories in the target image; wherein each second feature corresponds to a first feature, and the predicted location information is obtained by performing target detection on the objects in the target image;

[0088] The classification module 83 is used to determine the object category of each pixel in the target image based on the fused feature obtained by fusing the first feature and the second feature.

[0089] The specific steps of the image processing method performed by the above-mentioned device can be referred to the description in the above method embodiments, and will not be repeated here.

[0090] Furthermore, embodiments of this disclosure also provide a device, such as... Figure 9 As shown, the device includes a processor 91, a memory 92, and computer instructions stored in the memory 92 that can be executed by the processor 91. When the processor 91 executes the computer instructions, it implements the method described in any of the above embodiments.

[0091] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the foregoing embodiments.

[0092] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0093] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this disclosure can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the embodiments of this disclosure, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this disclosure.

[0094] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0095] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. When implementing the embodiments of this disclosure, the functions of each module can be implemented in one or more software and / or hardware. Alternatively, some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0096] This disclosure relates to the field of augmented reality (AR). It involves acquiring image information of target objects in a real-world environment and then using various visual algorithms to detect or identify the relevant features, states, and attributes of these objects, thereby achieving an AR effect that combines virtual and real elements to suit specific applications. For example, target objects may include human features such as faces, limbs, gestures, and movements; objects such as signs and markers; or venues such as sand tables, display areas, or displayed items. Visual algorithms may include visual localization, SLAM, 3D reconstruction, image registration, background segmentation, keypoint extraction and tracking of objects, and object pose or depth detection. Specific applications can include interactive scenarios related to real-world scenes or objects, such as guided tours, navigation, explanations, reconstruction, and virtual effect overlay displays, as well as human-related special effects processing, such as makeup enhancement, limb enhancement, special effects displays, and virtual model displays. Convolutional neural networks (CNNs) can be used to detect or identify the relevant features, states, and attributes of target objects. The aforementioned CNNs are network models trained using deep learning frameworks.

[0097] The above description is merely a specific implementation of the embodiments of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of the embodiments of this disclosure, and these improvements and modifications should also be considered within the protection scope of the embodiments of this disclosure.

Claims

1. An image processing method, characterized in that, The method includes: Feature extraction is performed on the acquired target image to obtain at least one first feature, wherein the first feature is related to the object category to which each pixel in the target image belongs; Based on the predicted location information of objects of different object categories in the target image, at least one second feature is obtained; wherein each second feature corresponds to a first feature, the predicted location information is obtained by performing object detection on the objects in the target image, the predicted location information is used to indicate the position of each object in the target image, each second feature is used to represent the weight of the corresponding first feature, and the weight is used to characterize the reliability of the extracted first feature; Based on the fused feature obtained by fusing the first feature and the second feature, the object category to which each pixel in the target image belongs is determined.

2. The method according to claim 1, characterized in that, The method is executed through a pre-trained semantic segmentation model, which is trained in the following manner: Obtain a sample image carrying a first label, wherein the first label is used to indicate the object category to which each pixel in the sample image belongs; Based on the first label, determine the prior location information corresponding to objects of different object categories in the sample image; Using a preset initial model based on the sample image and the prior location information, the object category of each pixel in the sample image is predicted to obtain the prediction result; Based on the prediction results and the target loss determined by the first label, the initial model is iteratively optimized to obtain the semantic segmentation model.

3. The method according to claim 1 or 2, characterized in that, The predicted location information is determined by a pre-trained target detection model, which is trained in the following manner: Obtain a sample image carrying a first label, wherein the first label is used to indicate the object category to which each pixel in the sample image belongs; Based on the first label, the prior location information corresponding to different categories of objects in the sample image is determined, and the prior location information is used as the second label of the sample image; The target detection model is obtained by training using the sample images and the second label.

4. The method according to claim 3, characterized in that, The step of determining the prior location information corresponding to different categories of objects in the sample image based on the first label includes: For each object in the sample image, the image region of each object in the sample image is determined based on the first label; A detection box corresponding to each object is determined based on the image region. The detection box can cover the image region, and the position information of the detection box in the sample image is used as the prior position information.

5. The method according to claim 4, characterized in that, Determining the detection box corresponding to each object based on the image region includes: Determine the minimum bounding rectangle corresponding to the image region; The minimum bounding rectangle is used as the detection box for each object; or the minimum bounding rectangle is expanded around it according to a preset ratio to obtain a target rectangle, and the target rectangle is used as the detection box.

6. The method according to claim 3, characterized in that, The sample image includes multiple objects. When the detection boxes corresponding to the multiple objects overlap, the method further includes: For multiple detection boxes with overlapping regions, the overlapping region is assigned to the detection box closest to the overlapping region; Reduce the area of ​​other detection frames among the plurality of detection frames until the overlapping area is outside the other detection frames.

7. The method according to claim 1 or 2, characterized in that, The method of obtaining at least one second feature based on the predicted location information corresponding to different categories of objects in the target image includes: A mask image is determined based on the predicted location information. In the mask image, the pixel values ​​of the pixels of objects belonging to the same object category are consistent, and the pixel values ​​of the pixels of objects belonging to different object categories are different. The mask image is subjected to max pooling and average pooling, and the features obtained by max pooling and average pooling are concatenated to obtain the concatenated features. The concatenated features are then subjected to convolution to obtain the second feature.

8. The method according to claim 1 or 2, characterized in that, The method of determining the object category of each pixel in the target image based on the fused feature obtained by fusing the first feature and the second feature includes: Each of the first features is multiplied by its corresponding second feature, and the result is convolved. Then, the features obtained from the convolution are fused to obtain the fused feature. The fused features are decoded to obtain the object category to which each pixel in the target image belongs.

9. A training method for a semantic segmentation model, characterized in that, The method includes: Obtain a sample image carrying a first label, wherein the first label is used to indicate the object category to which each pixel in the sample image belongs; Based on the first label, prior location information corresponding to objects of different object categories in the sample image is determined, and the prior location information is used to indicate the position of each object in the sample image. The sample image is used to extract features using a preset initial model to obtain at least one first feature, which is related to the object category to which each pixel in the sample image belongs. Based on the prior location information of objects of different object categories in the sample image, at least one second feature is obtained. Each second feature corresponds to one first feature, and each second feature is used to represent the weight of the corresponding first feature. The weight is used to characterize the reliability of the extracted first feature. Based on the fused feature obtained by fusing the first feature and the second feature, the object category to which each pixel in the sample image belongs is determined to obtain a prediction result. Based on the prediction results and the target loss determined by the first label, the initial model is iteratively optimized to obtain the semantic segmentation model.

10. An image processing apparatus, characterized in that, The device includes: The first feature extraction module is used to extract features from the acquired target image to obtain at least one first feature, wherein the first feature is related to the object category to which each pixel in the target image belongs. The second feature extraction module is used to obtain at least one second feature based on the predicted location information of objects of different object categories in the target image; wherein each second feature corresponds to a first feature, the predicted location information is obtained by performing object detection on the objects in the target image, the predicted location information is used to indicate the position of each object in the target image, each second feature is used to represent the weight of the corresponding first feature, and the weight is used to characterize the reliability of the extracted first feature; The classification module is used to determine the object category of each pixel in the target image based on the fused feature obtained by fusing the first feature and the second feature.

11. An electronic device, characterized in that, The electronic device includes a processor, a memory, and computer instructions stored in the memory that are executable by the processor, wherein the processor executes the computer instructions to implement the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when processed, implements the method as described in any one of claims 1-9.

Images