Method, apparatus, and storage medium for processing image shadows
By constructing a target multi-task network based on convergent features, the problems of high engineering deployment difficulty and low processing efficiency in image shadow removal methods are solved, achieving efficient and accurate shadow removal and mask segmentation, supporting autonomous driving perception tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHONGQING CHANGAN TECH CO LTD
- Filing Date
- 2023-07-07
- Publication Date
- 2026-06-02
AI Technical Summary
Existing image shadow removal methods suffer from problems such as high engineering deployment difficulty, low processing efficiency, or poor shadow removal effect.
A target multi-task network based on convergence features is constructed, including a convolution module, a residual module, an upsampling module, and an output module. The network is trained using a training dataset to obtain the target unshaded image and the shadow mask image of the image to be tested. The target image is determined by judging the shadow area ratio by setting a threshold.
It achieves efficient and automated image shadow removal, outputs high-precision images, is easy to deploy in engineering, and assists in autonomous driving perception tasks.
Smart Images
Figure CN116703782B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image shadow processing technology, specifically to a method, apparatus, and storage medium for processing image shadows. Background Technology
[0002] In recent years, various image processing systems have been widely applied in the field of intelligent driving. However, the coverage of shadows in images can cause significant difficulties for perception tasks in intelligent driving. Shadows disrupt the continuity of image grayscale, leading to deviations in the results of perception tasks. Therefore, in order to improve the visual appeal of images and reduce the difficulty of subsequent perception applications, it is necessary to detect and remove shadow-covered areas of images.
[0003] In recent years, various image shadow removal methods have emerged in the field of image shadow processing. For example, one method for removing shadows from pitted images uses a morphological grayscale segmentation method. This method first performs coarse segmentation using grayscale, then optimizes the segmentation results using morphological methods, and finally removes the shadows using a smoothing approach based on the segmentation results. However, this method, which obtains segmentation results through a single grayscale feature, often suffers from inaccuracies, leading to errors in shadow removal. Another example is a shadow removal method based on a two-stage image context recognition network, which achieves shadow removal through a two-stage approach of contextual feature transfer and result refinement; and a shadow removal method based on a partially supervised dual-network mutual stimulation learning, which completes shadow removal by constructing two networks: a shadow residual network and a shadow brightening network. While these two network-based processing methods can achieve good shadow removal results, their engineering deployment is challenging. In actual processing, multiple stages or multiple networks are required, resulting in significant latency and low processing efficiency.
[0004] Therefore, existing image shadow removal methods suffer from problems such as high engineering deployment difficulty, low processing efficiency, or poor shadow removal effect. Summary of the Invention
[0005] The purpose of this application is to provide a method, apparatus, and storage medium for processing image shadows, in order to solve the problems of high engineering deployment difficulty, low processing efficiency, or poor shadow removal effect of existing image shadow removal methods.
[0006] To achieve the above objectives, the first aspect of this application provides a method for processing image shadows, the method comprising:
[0007] Construct a target multi-task network based on convergence features;
[0008] Acquire the image to be tested;
[0009] The image to be tested is input into the target multi-task network to obtain the target deshading image and the target shadow mask image;
[0010] Determine the proportion of shadow area in the target shadow mask image;
[0011] Compare the shadow area ratio with a set threshold;
[0012] If the proportion of the shadow area is greater than a set threshold, the target image corresponding to the image under test is determined as the target image;
[0013] If the proportion of the shadow area is less than or equal to a set threshold, the image to be tested is determined as the target image;
[0014] The target image is the input image for the autonomous driving perception task.
[0015] In this embodiment of the application, the target multi-task network includes:
[0016] The module consists of a convolution module, a residual module, an upsampling module, and an output module.
[0017] In this embodiment, the convolution module includes a first convolution stage, a second convolution stage, and a third convolution stage. The first convolution stage includes two convolutional layers and a max pooling layer. The second and third convolutional stages each include two convolutional layers and an average pooling layer.
[0018] In this embodiment, the upsampling module includes a first upsampling stage, a second upsampling stage, and a third upsampling stage, each of which includes two convolutional layers.
[0019] In this embodiment of the application, inputting the image to be tested into the target multi-task network to obtain the target deshaded image and the target shadow mask image includes:
[0020] The basic image features of the image under test are extracted using a convolution module.
[0021] The basic features of the image are optimized and extracted using the residual module to obtain the deep features of the image;
[0022] Based on deep image features, feature convergence processing is performed through convolution and upsampling modules to obtain target features;
[0023] Based on the target features, the output module performs shadow mask segmentation and shadow removal generation tasks to obtain the target shadow removal image and target shadow mask image corresponding to the image under test.
[0024] In this embodiment of the application, based on deep image features, feature convergence processing is performed through a convolution module and an upsampling module to obtain the target features, including:
[0025] The deep features of the image and the features obtained in the third convolution stage are converged through the third upsampling stage to obtain the features after the first convergence.
[0026] The features obtained from the initial aggregation stage and the features obtained from the second convolution stage are aggregated through a second upsampling stage to obtain the features after secondary aggregation.
[0027] The features obtained after the second convergence are combined with the features obtained in the first convolution stage through the first upsampling stage to obtain the target features.
[0028] In this embodiment of the application, the residual module includes six layers of residual blocks, and the channel dimension in each residual block is expanded by a factor of two.
[0029] The output module includes a first convolutional layer and a second convolutional layer. The first convolutional layer is used to perform the shadow mask segmentation task, and the second convolutional layer is used to perform the shadow removal generation task.
[0030] In this embodiment of the application, constructing a target multi-task network based on convergence features includes:
[0031] Obtain the training and test datasets;
[0032] Construct an initial multi-task network based on convergence features;
[0033] Train an initial multi-task network based on the training dataset;
[0034] The trained multi-task network is tested using a test dataset to obtain the target multi-task network.
[0035] The training dataset includes multiple sets of training data, each set of training data including the original image and the corresponding shadow mask image and shadow-removed image.
[0036] A second aspect of this application provides an apparatus for processing image shadows, comprising:
[0037] The memory is configured to store instructions; and
[0038] The processor is configured to retrieve instructions from memory and, when executing the instructions, to implement the aforementioned method for processing image shadows.
[0039] A third aspect of this application provides a machine-readable storage medium, characterized in that the machine-readable storage medium stores instructions for causing a machine to perform the above-described method for processing image shadows.
[0040] The beneficial effects of this application are:
[0041] (1) This application adopts a target multi-task network based on convergence characteristics, which has a high degree of automation, high operating efficiency and is easy to deploy in engineering.
[0042] (2) This application can obtain the target deshading image and target shadow mask image of the image under test through a target multi-task network based on convergence features, and the accuracy of the output image is relatively high;
[0043] (3) This application can determine the input image for subsequent perception tasks through shadow mask images, which is beneficial for the subsequent perception tasks of autonomous driving.
[0044] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description
[0045] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:
[0046] Figure 1 A flowchart illustrating a method for processing image shadows provided in an embodiment of this application;
[0047] Figure 2 A schematic diagram of the image to be tested provided in a specific embodiment of this application;
[0048] Figure 3 A schematic diagram of a shadow-removed image provided in a specific embodiment of this application;
[0049] Figure 4 This is a schematic diagram of a shadow mask image provided in a specific embodiment of this application;
[0050] Figure 5 This is a schematic diagram of the structure of the target multi-task network provided in the embodiments of this application;
[0051] Figure 6 A flowchart illustrating a method for processing image shadows according to a specific embodiment of this application;
[0052] Figure 7 This is a structural block diagram of an apparatus for processing image shadows, provided in an embodiment of this application.
[0053] 710 - memory; 720 - processor. Detailed Implementation
[0054] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.
[0055] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.
[0056] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.
[0057] Figure 1 This is a flowchart illustrating a method for processing image shadows provided in an embodiment of this application. Figure 1 As shown in the figure, this application provides a method for processing image shadows, which may include the following steps.
[0058] Step 101: Construct a target multi-task network based on convergence features.
[0059] In this embodiment, to achieve shadow detection and removal in the image under test, considering the strong learning ability of deep network features, their ability to simultaneously complete multiple visual tasks, and their ease of engineering deployment, this embodiment constructs a target multi-task network based on convergent features. The target multi-task network is an end-to-end deep learning multi-task network that can output corresponding shadow mask images and de-shadowed images based on the input image under test. In one example, the initial multi-task network can be built and its parameters configured using the PyTorch framework. Then, the initial multi-task network can be trained using training data to obtain optimized network parameters. Finally, the trained multi-task network can be tested using test data to obtain the target multi-task network.
[0060] Step 102: Obtain the image to be tested;
[0061] Step 103: Input the image to be tested into the target multi-task network to obtain the target deshading image and the target shadow mask image.
[0062] In this embodiment, the image to be tested refers to the image for which shadow detection and removal are required. The image to be tested is typically an RGB three-channel image with shadows. With a target multi-task network constructed, automatic shadow detection and removal can be performed on multiple images to be tested based on the target multi-task network. Specifically, the image to be tested is input into the target multi-task network, and after processing by the target multi-task network, a target shadow mask image and a target de-shadowed image are output. Figure 2 This is a schematic diagram of the image to be tested provided in a specific embodiment of this application. Figure 3 This is a schematic diagram of a shadow-removed image provided in a specific embodiment of this application. Figure 4 This is a schematic diagram of a shadow mask image provided in a specific embodiment of this application. For example... Figure 2 The image to be tested, after being input into the target multi-task network, yields the corresponding unshaded image and shadow mask image, respectively, as shown below. Figure 3 and Figure 4 As shown in the diagram. The target shadow mask image is the shadow mask image corresponding to the image under test. In the shadow mask image, a pixel value of 0 represents the background, and a pixel value of 1 represents the shadow. The target deshading image is the deshading image corresponding to the image under test. The deshading image is typically a shadow-free image with RGB three channels. Thus, the deshading image and shadow mask image corresponding to the image under test can be quickly obtained through a target multi-task network based on convergent features, and the results are highly accurate.
[0063] Step 104: Determine the proportion of shadow area in the target shadow mask image;
[0064] Step 105: Compare the shadow area ratio with the set threshold;
[0065] Step 106: If the shadow area ratio is greater than the set threshold, determine the target image corresponding to the image to be tested as the target image;
[0066] Step 107: If the shadow area ratio is less than or equal to the set threshold, the image to be tested is determined as the target image;
[0067] The target image is the input image for the autonomous driving perception task.
[0068] In this embodiment, the target image is the input image for subsequent autonomous driving perception tasks. Since the image distribution may change after processing by the target multi-task network when the shadow of the image under test is small, using the de-shadowed image as the target image in this case would prevent subsequent perception applications from obtaining normal processing results. Therefore, after processing the image under test, considering the flexibility of subsequent perception applications in intelligent driving, the target image to be used for the perception task can be determined according to the needs of the subsequent perception applications. The target image can be either the image under test or the target de-shadowed image. Specifically, a threshold method can be used to determine the target image. Since the shadow portion and background portion in the target shadow mask image are completely distinguished, using the shadow mask image to determine the shadow area ratio in the image under test is more accurate and efficient. Therefore, the shadow area ratio in the target shadow mask image is first determined, and then this shadow area ratio is compared with a set threshold to determine the target image. The set threshold is the critical value for determining the shadow area ratio of the target image. The set threshold can be set according to actual conditions; for example, it can be set to 10%. If the shadow area ratio is greater than a set threshold, the target image with the shadow removed is determined as the target image and used as input for subsequent perception tasks; if the shadow area ratio is less than or equal to the set threshold, the unprocessed original image of the image to be tested is determined as the target image.
[0069] The above technical solution first constructs a target multi-task network based on convergence features and acquires the image to be tested. Then, the image to be tested is input into the target multi-task network to obtain a target deshaded image and a target shadow mask image. Next, the shadow area ratio in the target shadow mask image is determined. Finally, the shadow area ratio is compared with a set threshold. If the shadow area ratio is greater than the set threshold, the target deshaded image corresponding to the image to be tested is determined as the target image; if the shadow area ratio is less than or equal to the set threshold, the image to be tested is determined as the target image. The target image serves as the input image for the autonomous driving perception task. This application employs a multi-task network based on convergence features, which has a high degree of automation, high operating efficiency, and is easy to deploy in engineering. It can obtain high-precision shadow mask images and deshaded images, which is beneficial for assisting subsequent autonomous driving perception tasks.
[0070] In this embodiment of the application, step 101, constructing a target multi-task network based on convergence features, may include:
[0071] Obtain the training and test datasets;
[0072] Construct an initial multi-task network based on convergence features;
[0073] Train an initial multi-task network based on the training dataset;
[0074] The trained multi-task network is tested using a test dataset to obtain the target multi-task network.
[0075] The training dataset includes multiple sets of training data, each set of training data including the original image and the corresponding shadow mask image and shadow-removed image.
[0076] Specifically, before constructing the target multi-task network, data preparation is required to obtain training and testing datasets. In one example, training and testing datasets can be obtained from the ISTU dataset, which pre-classifies data into two categories: training data and testing data. Multiple sets of initial training and testing data are obtained from the ISTU dataset. Since labels, including shadow mask images and de-shadowed images, are needed when training the network, each set of initial training data contains the original image and its corresponding shadow mask and de-shadowed images. To meet the GPU memory requirements and conform to the network's format during training, all original images and their corresponding labels need to be cropped to the same size without overlap, thus obtaining the initial training dataset for the multi-task network. For example, 1330 sets of initial training data with an image size of 480*640 are obtained. Each image in the initial training data is then cropped to 240*320 without overlap, resulting in a training dataset of 5320 sets.
[0077] Furthermore, an initial multi-task network based on convergence features can be constructed using the PyTorch framework, with relevant parameters configured. In one example, the hyperparameters used to train the initial multi-task network include: an initial learning rate of 0.001; 100 training epochs; a batch size of 8; SGD as the optimizer; and an exponential descent strategy for the learning rate, with an energy value set to 0.9. The initial multi-task network is then trained using the training dataset to obtain optimized network parameters. Finally, the trained multi-task network is tested using the test dataset to obtain the target multi-task network. In one example, after constructing the initial multi-task network, it can be trained using training datasets with and without shadow labels within the PyTorch deep learning framework. Training includes two tasks: shadow region detection and shadow removal image generation. The shadow region detection task can employ the cross-entropy function L... cross The task of generating shadow-removed images can use the L1 norm loss function. The network loss during training is the linear sum of the two, which satisfies the following formula:
[0078] L=αL cross +L1;
[0079] Where α and β are adjustable parameters, and their values are both 1 here.
[0080] Thus, the above scheme can yield an end-to-end target multi-task network. By using feature convergence, the target multi-task network comprehensively utilizes the features of each level of the image, and can obtain high-precision shadow mask images and de-shadowed images. It has a high degree of automation, high operating efficiency, and is easy to deploy in engineering.
[0081] In this embodiment of the application, the target multi-task network includes:
[0082] The module consists of a convolution module, a residual module, an upsampling module, and an output module.
[0083] Specifically, considering subsequent engineering deployment issues, and to facilitate deployment and improve network operating efficiency, the target multi-task network in this embodiment mainly adopts the average cost operator. The target multi-task network includes a convolutional module, a residual module, an upsampling module, and an output module. The convolutional module includes three convolutional stages, each followed by a 2x downsampling, used to extract basic image features from the input image. Features obtained from each average cost stage are retained during network inference. The residual module includes six residual blocks, each with a 2x channel dimension expansion, used to optimize the extracted deep features, making the generated shadow-free image more realistic. The upsampling module includes three upsampling stages, each including two convolutional layers. Features obtained after the residual stages can be converged with features from each convolutional stage in turn during each upsampling stage. The output module includes a first convolutional layer and a second convolutional layer. The first convolutional layer performs shadow mask segmentation, and the second convolutional layer performs shadow-free generation.
[0084] Figure 5 This is a schematic diagram of the structure of the target multi-task network provided in an embodiment of this application. Figure 5 As shown in the embodiments of this application, the convolution module includes a first convolution stage, a second convolution stage, and a third convolution stage. The first convolution stage includes two convolutional layers and a max pooling layer. The second and third convolutional stages each include two convolutional layers and an average pooling layer.
[0085] In one example, the first convolutional stage includes a convolutional layer with 3 input channels, 32 output channels, and a 3x3 kernel; a convolutional layer with 32 input channels, 64 output channels, and a 1x1 kernel; and a 3x3 max pooling layer. The second convolutional stage includes a convolutional layer with 64 input channels, 128 output channels, and a 1x1 kernel; a convolutional layer with 128 input channels, 128 output channels, and a 1x1 kernel; and a 3x3 average pooling layer. The third convolutional stage includes a convolutional layer with 128 input channels, 256 output channels, and a 1x1 kernel; a convolutional layer with 256 input channels, 256 output channels, and a 1x1 kernel; and a 3x3 average pooling layer. The purpose of the convolutional stages is to extract basic image features, and the features obtained in each convolutional stage are preserved during network inference.
[0086] In this embodiment, the residual module includes six layers of residual blocks, and the channel dimension expansion factor in each residual block is twice.
[0087] Specifically, the residual module is the residual stage in the target multi-task network. The role of the residual stage is to optimize the extracted deep features, making the generated unshaded image more realistic. The residual stage contains six residual blocks. To save computational resources, compared with conventional residual blocks, the channel dimension expansion factor in each residual block in this embodiment is changed from 4 times to 2 times.
[0088] In this embodiment, the upsampling module includes a first upsampling stage, a second upsampling stage, and a third upsampling stage, each of which includes two convolutional layers.
[0089] In one example, the first upsampling stage includes a convolutional layer with 64 input channels, 64 output channels, and a 1x1 kernel; and a convolutional layer with 128 input channels, 64 output channels, and a 1x1 kernel. The second upsampling stage includes a convolutional layer with 128 input channels, 64 output channels, and a 1x1 kernel; and a convolutional layer with 128 input channels, 64 output channels, and a 1x1 kernel. The third upsampling stage includes a convolutional layer with 256 input channels, 64 output channels, and a 1x1 kernel; and a convolutional layer with 320 input channels, 64 output channels, and a 1x1 kernel.
[0090] In this embodiment, the output module includes a first convolutional layer and a second convolutional layer. The first convolutional layer is used to perform shadow mask segmentation tasks, and the second convolutional layer is used to perform shadow removal generation tasks.
[0091] In one example, the output module includes a first convolutional layer with 64 input channels, 2 output channels, and a 1x1 kernel size; and a second convolutional layer with 64 input channels, 3 output channels, and a 1x1 kernel size. The first convolutional layer is used for shadow mask segmentation, and the second convolutional layer is used for shadow-removed image generation.
[0092] In this embodiment of the application, inputting the image to be tested into the target multi-task network to obtain the target deshaded image and the target shadow mask image includes:
[0093] The basic image features of the image under test are extracted using a convolution module.
[0094] The basic features of the image are optimized and extracted using the residual module to obtain the deep features of the image;
[0095] Based on deep image features, feature convergence processing is performed through convolution and upsampling modules to obtain target features;
[0096] Based on the target features, the output module performs shadow mask segmentation and shadow removal generation tasks to obtain the target shadow removal image and target shadow mask image corresponding to the image under test.
[0097] After the image to be tested is input into the target multi-task network, the network first extracts the basic features of the image through a convolutional module. This convolutional module consists of three stages, after which the resulting feature map is downsampled by a factor of 2, and the feature map obtained at each stage is saved. After the third convolutional stage, to make the generated image more realistic, the target multi-task network can optimize the extraction of the basic image features through a residual module to obtain deep image features. Further, based on the deep image features obtained from the residual module, the target multi-task network performs feature convergence processing through convolutional and upsampling modules. This involves concatenating the feature map obtained from each convolutional stage with the deep image features, followed by convolution and upsampling. Finally, after three convolutional stages and three upsampling stages, a target feature of the same size as the image to be tested is obtained. The target feature is the feature image of the same size as the image to be tested. Finally, based on the target feature, the output module simultaneously performs shadow mask segmentation and deshading image generation tasks to obtain the target shadow mask image and the target deshading image corresponding to the image to be tested.
[0098] In this embodiment of the application, based on deep image features, feature convergence processing is performed through a convolution module and an upsampling module to obtain the target features, including:
[0099] The deep features of the image and the features obtained in the third convolution stage are converged through the third upsampling stage to obtain the features after the first convergence.
[0100] The features obtained from the initial aggregation stage and the features obtained from the second convolution stage are aggregated through a second upsampling stage to obtain the features after secondary aggregation.
[0101] The features obtained after the second convergence are combined with the features obtained in the first convolution stage through the first upsampling stage to obtain the target features.
[0102] Specifically, the deep image features obtained through the residual module are first converged through a convolutional layer in the third upsampling and third convolutional stages to obtain the initial converged features. These initial converged features are then converged again in the same manner in the second upsampling and second convolutional stages to obtain the second converged features. Finally, these second converged features are converged again in the first upsampling and first convolutional stages to obtain the target features. The target features are the feature map corresponding to the image under test obtained after multiple feature convergences. In one example, the feature convergence can be achieved by concatenating the channel dimensions and performing convolution, resulting in 64 feature channels obtained in the first downsampling stage. Thus, by utilizing features from various image layers through feature convergence, high-precision shadow mask images and de-shadowed images can be obtained.
[0103] Figure 6 This is a flowchart illustrating a method for processing image shadows according to a specific embodiment of this application. Figure 6 As shown in the figure, a specific embodiment of this application provides a flowchart of a method for processing image shadows, which may include the following steps:
[0104] Step 601: Obtain the image to be tested;
[0105] Step 602: Input the image to be tested into a target multi-task network based on convergence features;
[0106] Step 603: Obtain the shadow mask image and the de-shadowed image;
[0107] Step 604: Determine the shadow area ratio Q in the shadow mask image;
[0108] Step 605: Determine whether Q>T is true. If true, proceed to step 606; otherwise, proceed to step 607.
[0109] Step 606: Output the deshadowed image;
[0110] Step 607: Output the image to be tested.
[0111] The image to be tested is a three-channel natural image, and the output shadow image or the image to be tested serves as the input image for subsequent perception tasks. In this way, highly accurate de-shadowed images and shadow mask images can be extracted, aiding subsequent autonomous driving perception tasks.
[0112] Figure 7 This is a structural block diagram of an apparatus for processing image shadows, provided as an embodiment of this application. Figure 7 As shown, this application provides an apparatus for processing image shadows, which may include:
[0113] Memory 710 is configured to store instructions; and
[0114] The processor 720 is configured to retrieve instructions from the memory 710 and, when executing the instructions, to implement the aforementioned method for processing image shadows.
[0115] Specifically, in this embodiment of the application, the processor 720 can be configured to:
[0116] Construct a target multi-task network based on convergence features;
[0117] Acquire the image to be tested;
[0118] The image to be tested is input into the target multi-task network to obtain the target deshading image and the target shadow mask image;
[0119] Determine the proportion of shadow area in the target shadow mask image;
[0120] Compare the shadow area ratio with a set threshold;
[0121] If the proportion of the shadow area is greater than a set threshold, the target image corresponding to the image under test is determined as the target image;
[0122] If the proportion of the shadow area is less than or equal to a set threshold, the image to be tested is determined as the target image;
[0123] The target image is the input image for the autonomous driving perception task.
[0124] In this embodiment of the application, the target multi-task network includes:
[0125] The module consists of a convolution module, a residual module, an upsampling module, and an output module.
[0126] In this embodiment, the convolution module includes a first convolution stage, a second convolution stage, and a third convolution stage. The first convolution stage includes two convolutional layers and a max pooling layer. The second and third convolutional stages each include two convolutional layers and an average pooling layer.
[0127] In this embodiment, the upsampling module includes a first upsampling stage, a second upsampling stage, and a third upsampling stage, each of which includes two convolutional layers.
[0128] Furthermore, the processor 720 can also be configured as follows:
[0129] The basic image features of the image under test are extracted using a convolution module.
[0130] The basic features of the image are optimized and extracted using the residual module to obtain the deep features of the image;
[0131] Based on deep image features, feature convergence processing is performed through convolution and upsampling modules to obtain target features;
[0132] Based on the target features, the output module performs shadow mask segmentation and shadow removal generation tasks to obtain the target shadow removal image and target shadow mask image corresponding to the image under test.
[0133] Furthermore, the processor 720 can also be configured as follows:
[0134] The deep features of the image and the features obtained in the third convolution stage are converged through the third upsampling stage to obtain the features after the first convergence.
[0135] The features obtained from the initial aggregation stage and the features obtained from the second convolution stage are aggregated through a second upsampling stage to obtain the features after secondary aggregation.
[0136] The features obtained after the second convergence are combined with the features obtained in the first convolution stage through the first upsampling stage to obtain the target features.
[0137] In this embodiment of the application, the residual module includes six layers of residual blocks, and the channel dimension in each residual block is expanded by a factor of two.
[0138] The output module includes a first convolutional layer and a second convolutional layer. The first convolutional layer is used to perform the shadow mask segmentation task, and the second convolutional layer is used to perform the shadow removal generation task.
[0139] Furthermore, the processor 720 can also be configured as follows:
[0140] Obtain the training and test datasets;
[0141] Construct an initial multi-task network based on convergence features;
[0142] Train an initial multi-task network based on the training dataset;
[0143] The trained multi-task network is tested using a test dataset to obtain the target multi-task network.
[0144] The training dataset includes multiple sets of training data, each set of training data including the original image and the corresponding shadow mask image and shadow-removed image.
[0145] The above technical solution first constructs a target multi-task network based on convergence features and acquires the image to be tested. Then, the image to be tested is input into the target multi-task network to obtain a target deshaded image and a target shadow mask image. Next, the shadow area ratio in the target shadow mask image is determined. Finally, the shadow area ratio is compared with a set threshold. If the shadow area ratio is greater than the set threshold, the target deshaded image corresponding to the image to be tested is determined as the target image; if the shadow area ratio is less than or equal to the set threshold, the image to be tested is determined as the target image. The target image serves as the input image for the autonomous driving perception task. This application employs a multi-task network based on convergence features, which has a high degree of automation, high operating efficiency, and is easy to deploy in engineering. It can obtain high-precision shadow mask images and deshaded images, which is beneficial for assisting subsequent autonomous driving perception tasks.
[0146] This application also provides a machine-readable storage medium storing instructions that cause a machine to perform the above-described method for processing image shadows.
[0147] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, and optical storage) containing computer-usable program code.
[0148] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0149] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0150] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0151] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0152] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0153] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0154] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0155] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. A method for processing image shadows, characterized in that, The method includes: A target multi-task network based on convergence features is constructed. The target multi-task network includes a convolutional module, a residual module, an upsampling module, and an output module. The convolutional module includes a first convolutional stage, a second convolutional stage, and a third convolutional stage. The first convolutional stage includes two convolutional layers and a max pooling layer. The second and third convolutional stages each include two convolutional layers and an average pooling layer. The upsampling module includes a first upsampling stage, a second upsampling stage, and a third upsampling stage. Each upsampling stage includes two convolutional layers. Acquire the image to be tested; The image to be tested is input into the target multi-task network to obtain the target deshading image and the target shadow mask image; Determine the proportion of shadow area in the target shadow mask image; Compare the shadow area ratio with a set threshold; If the shadow area ratio is greater than a set threshold, the target deshaded image corresponding to the image under test is determined as the target image; If the shadow area ratio is less than or equal to a set threshold, the image under test is determined as the target image; The target image is the input image for the autonomous driving perception task; The step of inputting the image to be tested into the target multi-task network to obtain the target deshaded image and the target shadow mask image includes: The basic image features of the image under test are extracted using a convolution module; The residual module is used to optimize and extract the basic features of the image to obtain the deep features of the image; The deep features of the image and the features obtained in the third convolution stage are converged through the third upsampling stage to obtain the features after the first convergence. The features obtained after the initial convergence and the features obtained in the second convolution stage are converged through a second upsampling stage to obtain the features after secondary convergence. The features obtained after the second convergence are combined with the features obtained in the first convolution stage through the first upsampling stage to obtain the target features; Based on the target features, the output module performs shadow mask segmentation and shadow removal generation tasks to obtain the target shadow removal image and target shadow mask image corresponding to the image under test.
2. The method for processing image shadows according to claim 1, characterized in that, The residual module comprises six layers of residual blocks, and the channel dimension in each residual block is expanded by a factor of two. The output module includes a first convolutional layer and a second convolutional layer. The first convolutional layer is used to perform shadow mask segmentation tasks, and the second convolutional layer is used to perform shadow removal generation tasks.
3. The method for processing image shadows according to claim 1, characterized in that, The construction of the target multi-task network based on convergence features includes: Obtain the training and test datasets; Construct an initial multi-task network based on convergence features; The initial multi-task network is trained based on the training dataset; The trained multi-task network is tested using the test dataset to obtain the target multi-task network; The training dataset includes multiple sets of training data, each set of training data including the original image and the corresponding shadow mask image and shadow-removed image.
4. An apparatus for processing image shadows, characterized in that, include: The memory is configured to store instructions; as well as A processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the method for processing image shadows according to any one of claims 1 to 3.
5. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores instructions for causing the machine to perform the method for processing image shadows according to any one of claims 1 to 3.