Shadow area processing method and device, computer readable medium and electronic device

Abstract

The present disclosure provides a shadow area processing method, a shadow area processing device, a computer-readable medium and an electronic device, and relates to the field of image processing technology. The method comprises: performing shadow area detection on an image to be detected based on a shadow detection model to obtain a shadow area corresponding to the image to be detected; wherein the shadow detection model is obtained by training a constructed shadow detection neural network; the shadow detection neural network adjusts network parameters based on a brightness limit loss function; the brightness limit loss function is determined based on the cross entropy and relative entropy calculated based on the first detection result of the sample image and the second detection result of the sample image adjusted by brightness adjustment to obtain the adjusted image. The present disclosure enables the shadow detection model to overcome the influence of brightness factors and improve the accuracy of shadow area detection.

Description

Background Art

[0002] In recent years, computer vision systems have been widely used in production and life scenarios such as industrial visual inspection, video surveillance, medical image detection and intelligent driving. As a common physical phenomenon in nature, shadows bring many adverse effects to computer vision tasks, increase the difficulty of problem handling, and reduce the robustness of algorithms.

[0003] In order to prevent shadows from affecting computer vision tasks, it is often necessary to remove shadows from videos, images, and other data in advance. However, shadow removal is a technology that is closely related to light intensity. Different light intensities or different exposures of shooting devices when taking photos will result in different brightness of the shadow part and relative brightness to the non-shadow area. In this case, when detecting shadow areas based on neural networks, it is likely that the shadow areas will not be correctly distinguished due to changes in brightness or relative brightness, resulting in different detection results for the shadow areas. Summary of the invention

[0004] The purpose of the present disclosure is to provide a shadow area processing method, a shadow area processing device, a computer-readable medium and an electronic device, thereby avoiding the influence of brightness factors on shadow area recognition at least to a certain extent and improving the accuracy of shadow area detection.

[0005] According to a first aspect of the present disclosure, a shadow area processing method is provided, including: performing shadow area detection on an image to be detected based on a shadow detection model to obtain a shadow area corresponding to the image to be detected; wherein the shadow detection model is obtained by training a constructed shadow detection neural network; the shadow detection neural network adjusts network parameters based on a brightness limit loss function; the brightness limit loss function is determined based on a first detection result of a sample image and a second detection result of an adjusted image obtained by adjusting the brightness of the sample image to obtain a cross entropy and relative entropy.

[0006] According to a second aspect of the present disclosure, a shadow area processing device is provided, including: an area detection module, which is used to perform shadow area detection on an image to be detected based on a shadow detection model to obtain a shadow area corresponding to the image to be detected; wherein the shadow detection model is obtained by training a constructed shadow detection neural network; the shadow detection neural network adjusts network parameters based on a brightness limit loss function; the brightness limit loss function is determined based on a first detection result of a sample image and a second detection result of an adjusted image obtained by adjusting the brightness of the sample image and calculating cross entropy and relative entropy.

[0007] According to a third aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.

[0008] According to a fourth aspect of the present disclosure, an electronic device is provided, characterized in that it includes: a processor; and a memory for storing one or more programs, which enables the one or more processors to implement the above method when the one or more programs are executed by the one or more processors.

[0009] A shadow area processing method provided by an embodiment of the present disclosure can perform shadow area detection on a model to be detected based on a trained shadow detection model to obtain a shadow area corresponding to the image to be detected. The shadow detection model is obtained by training a constructed shadow detection neural network; the shadow detection neural network adjusts network parameters based on a brightness limit loss function; the brightness limit loss function is determined based on the cross entropy and relative entropy calculated based on the first detection result of the sample image and the second detection result of the adjusted image obtained by adjusting the brightness of the sample image. The shadow detection neural network is trained together with the adjusted image obtained by adjusting the brightness of the sample image and the sample image, and the network parameters are adjusted based on the loss function determined by the cross entropy and relative entropy calculated based on the first detection result and the second detection result, so that the shadow detection model can overcome the influence of the brightness factor and improve the accuracy of shadow area detection.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0012] Figure 1 A schematic diagram showing an exemplary system architecture to which embodiments of the present disclosure may be applied;

[0013] Figure 2 A schematic diagram showing an electronic device to which the embodiments of the present disclosure can be applied;

[0014] Figure 3 A flowchart schematically shows a shadow area processing method in an exemplary embodiment of the present disclosure;

[0015] Figure 4 A flowchart of a shadow detection model training method in an exemplary embodiment of the present disclosure is schematically shown;

[0016] Figure 5A sample image in an exemplary embodiment of the present disclosure is shown;

[0017] Figure 6 An adjustment image in an exemplary embodiment of the present disclosure is shown;

[0018] Figure 7 Another adjustment image in an exemplary embodiment of the present disclosure is shown;

[0019] Figure 8 A schematic diagram schematically shows the structure of a first sub-network and a second sub-network in a shadow detection neural network in an exemplary embodiment of the present disclosure;

[0020] Fig. 9 A schematic diagram schematically illustrates a training process of a shadow detection neural network in an exemplary embodiment of the present disclosure;

[0021] Fig.10 A schematic diagram schematically shows the composition of a shadow area processing device in an exemplary embodiment of the present disclosure;

[0022] Fig.11 A schematic diagram schematically shows the composition of another shadow area processing device in an exemplary embodiment of the present disclosure;

[0023] Fig.12 The figure schematically shows the composition of another shadow area processing device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0025] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0026] Figure 1 A schematic diagram of a system architecture of an exemplary application environment in which a shadow area processing method and device according to an embodiment of the present disclosure can be applied is shown.

[0027] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The terminal devices 101, 102, 103 may be various electronic devices with image processing functions, including but not limited to desktop computers, portable computers, smart phones, tablet computers, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0028] The shadow area processing method provided in the embodiment of the present disclosure is generally executed by the terminal devices 101, 102, and 103, and accordingly, the shadow area processing device is generally arranged in the terminal devices 101, 102, and 103. However, it is easy for those skilled in the art to understand that the shadow area processing method provided in the embodiment of the present disclosure can also be executed by the server 105, and accordingly, the shadow area processing device can also be arranged in the server 105, and this is not particularly limited in the present exemplary embodiment. For example, in an exemplary embodiment, the server 105 can train the constructed shadow detection neural network through the first training set and the second training set, and then send the obtained shadow detection model to the terminal devices 101, 102, and 103, and then perform shadow area detection on the image to be detected through the terminal devices 101, 102, and 103.

[0029] An exemplary embodiment of the present disclosure provides an electronic device for implementing a shadow area processing method, which may be Figure 1 The terminal device 101, 102, 103 or the server 105 in the embodiment of the present invention comprises at least a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the shadow area processing method by executing the executable instructions.

[0030] Below Figure 2 Taking the mobile terminal 200 in FIG. 1 as an example, the structure of the electronic device is exemplarily described. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Figure 2The structure in the figure can also be applied to fixed type devices. In other embodiments, the mobile terminal 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure can be implemented in hardware, software, or a combination of software and hardware. The interface connection relationship between the components is only shown schematically and does not constitute a structural limitation of the mobile terminal 200. In other embodiments, the mobile terminal 200 may also adopt the same Figure 2 Different interface connection methods, or a combination of multiple interface connection methods.

[0031] like Figure 2 As shown, the mobile terminal 200 may specifically include: a processor 210, an internal memory 221, an external memory interface 222, a Universal Serial Bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 271, a receiver 272, a microphone 273, an earphone interface 274, a sensor module 280, a display screen 290, a camera module 291, an indicator 292, a motor 293, a button 294, and a subscriber identification module (SIM) card interface 295, etc. The sensor module 280 may include a depth sensor 2801, a pressure sensor 2802, a gyroscope sensor 2803, etc.

[0032] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor and / or a neural network processor (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0033] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. NPU can realize applications such as intelligent cognition of the mobile terminal 200, such as image recognition, face recognition, voice recognition, text understanding, etc. In some embodiments, the above-mentioned NPU can be used to construct a shadow detection neural network and train the constructed shadow detection neural network.

[0034] The processor 210 is provided with a memory. The memory can store instructions for implementing six modular functions: detection instructions, connection instructions, information management instructions, analysis instructions, data transmission instructions, and notification instructions, and the execution is controlled by the processor 210.

[0035] The mobile terminal 200 implements the display function through a GPU, a display screen 290, and an application processor. The GPU is a microprocessor for image processing, which is connected to the display screen 290 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information. In some embodiments, the sample images in the first training set can be brightness adjusted by the GPU to obtain a second training set.

[0036] The depth sensor 2801 is used to obtain the depth information of the scene. The pressure sensor 2802 is used to sense the pressure signal and can convert the pressure signal into an electrical signal. The gyroscope sensor 2803 can be used to determine the motion posture of the mobile terminal 200.

[0037] In addition, sensors with other functions may be set in the sensor module 280 according to actual needs, such as an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0038] The mobile terminal 200 may also include other devices that provide auxiliary functions. For example, the key 294 includes a power key, a volume key, etc., and the user can generate key signal input related to the user settings and function control of the mobile terminal 200 through key input. For example, the indicator 292, the motor 293, the SIM card interface 295, etc.

[0039] Shadow removal is a technology that is closely related to light intensity. Different light intensities or different exposures of the shooting equipment when taking pictures will result in different brightness of the shadow part and the relative brightness of the shadow area and the non-shadow area. In this case, when detecting the shadow area based on the neural network, it is likely that the shadow area cannot be correctly distinguished due to changes in brightness or relative brightness, resulting in different detection results for the shadow area.

[0040] In the related shadow detection technology, shadow detection technology under fixed environment and conditions has made great progress. However, in the related technology, the brightness of the image to be processed is less concerned, so for the same image, in some unconventional brightness conditions, it is very likely that the shadow area detection problem will be wrong.

[0041] Based on one or more of the above problems, this exemplary embodiment provides a shadow area processing method. The shadow area processing method can be applied to the above server 105, and can also be applied to one or more of the above terminal devices 101, 102, 103, which is not particularly limited in this exemplary embodiment. Figure 3 As shown, the shadow area processing method may include the following steps S310: performing shadow area detection on the image to be detected based on the shadow detection model to obtain the shadow area corresponding to the image to be detected.

[0042] The shadow detection model is obtained by training the constructed shadow detection neural network; the shadow detection neural network adjusts the network parameters based on the brightness limit loss function; the brightness limit loss function is determined based on the cross entropy and relative entropy calculated based on the first detection result of the sample image and the second detection result of the adjusted image obtained by adjusting the brightness of the sample image.

[0043] In an exemplary embodiment, the shadow detection neural network may include a first sub-network and a second sub-network having the same network structure, and the first sub-network and the second sub-network share network parameters.

[0044] In an exemplary embodiment, the first sub-network and the second sub-network both include a first-stage network and a second-stage network, wherein the first-stage network includes an Encoder structure, and the second-stage network includes a Decoder structure.

[0045] It should be noted that, in some embodiments, the Encoder structure included in the first-stage network may include a ResNeXt101 structure, and the Decoder structure may include an extended path structure in a U-net network.

[0046] In addition, in the decoder structure, i.e., the upsampling circuit of the neural network, different levels of convolutional layers can be used, and the corresponding weight maps can be generated using the attention mechanism respectively. Then, the shadow segmentation results determined by the original network and the weight map can be fused together through attention fusion to obtain a more refined shadow area detection result. For example, the SE-Block attention module can be added to the last few layers of the shadow detection neural network.

[0047] At the same time, in the decoder structure, that is, the upsampling circuit of the neural network, when upsampling different feature layers, considering that the convolution layer can only preliminarily combine the image information in the receptive field, a non-local neural network can be added after the upper feature image, which can provide accurate global information for each pixel, automatically build pixel relationships, and solve the problem of large-scale recognition errors in shadow areas when detecting difficult case images. It should be noted that non-local operations only introduce global information, but do not change the size and depth of the feature layer.

[0048] In an exemplary embodiment, referring to Figure 4 As shown, the training process of the shadow detection model may include the following steps S410 and S420:

[0049] In step S410, a first training set including sample images is obtained, and brightness adjustment is performed on the sample images in the first training set to obtain a second training set including adjusted images.

[0050] In an exemplary embodiment, the samples required for training may be processed before training. Specifically, in order to avoid the influence of brightness on the shadow detection of the neural network, the sample images included in the first training set may be brightness adjusted to obtain adjusted images, and all the adjusted images may be used as the second training set. Figure 5 The sample image shown in the figure can have its brightness enhanced or reduced. Figure 5 The sample image shown is enhanced by 1.5 times to obtain the following Figure 6 Adjust the image as shown; Figure 5 The sample image shown is reduced in brightness by 0.5 times to obtain the following Figure 7 Adjusted image shown.

[0051] In step S420, a shadow detection neural network is constructed, and the shadow detection neural network is trained based on each sample image included in the first training set and each adjusted image included in the second training set to obtain a shadow detection model.

[0052] In an exemplary embodiment, the shadow detection neural network includes a first subnetwork and a second subnetwork with the same network structure, and the first subnetwork and the second subnetwork share network parameters. In this case, when the shadow detection neural network is trained based on the first training set and the second training set, the sample image in the first training set can be input into the first subnetwork for processing to obtain the first detection result, and the adjustment image corresponding to the sample image in the second training set can be input into the second subnetwork for processing to obtain the second detection result. Then, the cross entropy is calculated for the first detection result and the second detection result respectively to obtain the first cross entropy and the second cross entropy, and the relative entropy of the two is calculated based on the first detection result and the second detection result. After obtaining the first cross entropy, the second cross entropy and the relative entropy, the brightness limit function can be determined based on the first cross entropy, the second cross entropy and the relative entropy, and the network parameters of the shadow detection neural network are adjusted according to the determined brightness limit loss function, thereby obtaining a shadow detection model.

[0053] The cross entropy can be calculated based on the following formula (1):

[0054] C=∑ i -[wy i loga i +(1-w)(1-y i )log(1-a i )] Formula (1)

[0055] Among them, C represents the cross entropy, y i Represents the true value of the pixel category, y i ∈[0,1]; a i is the predicted value of the pixel category, with a i ∈[0, 1]; w is a preset balance factor. It should be noted that, in some embodiments, the preset balance factor w may be 0.15.

[0056] The above relative entropy can be calculated based on the following formula (2):

[0057]

[0058] Wherein, Lc represents the relative entropy of the first detection result and the second detection result; qi and q' i They represent the results obtained by performing softmax on the first detection result and the second detection result. Specifically, they can be obtained based on the following formulas (3) and (4):

[0059]

[0060]

[0061] Among them, qi and q' i They represent the results obtained by performing softmax on the first detection result and the second detection result respectively; z i and z' i They respectively represent the predicted values ​​of the pixel categories corresponding to the first detection result and the second detection result; K represents the number of pixels corresponding to the first detection result and the second detection result.

[0062] In an exemplary embodiment, after the cross entropy and relative entropy are calculated according to the above formulas (1) to (4), the brightness limit loss function can be determined based on the following formulas (5) and (6):

[0063] L m =C+C′ Formula (5)

[0064] L=L m +α×L c Formula (6)

[0065] Among them, L m represents the sum of the first detection result and the second detection result, C represents the cross entropy corresponding to the first detection result, C' represents the cross entropy corresponding to the second detection result; Lc represents the relative entropy corresponding to the first detection result and the second detection result; α represents a preset weight. In an exemplary embodiment, the weight α can be 2.

[0066] After obtaining the shadow detection model, shadow area detection can be performed directly on the image to be detected based on the shadow detection model. Since the cross entropy of the first detection result and the cross entropy of the second detection result, as well as the relative entropy between the first detection result and the second detection result are considered at the same time during the training process, the problem of not being able to correctly distinguish the shadow area due to changes in brightness or relative brightness can be overcome to a certain extent.

[0067] In an exemplary embodiment, after the shadow area is detected, the boundary of the shadow area can be post-processed based on the conditional random field (CRF) to obtain the processed shadow area. CRF is a segmentation post-processing method, which can perform post-processing by calculating the probability value of a certain type between points. Since the shadow area detection is equivalent to a binary segmentation problem (shadow and non-shadow), the boundary of the shadow area can be processed based on CRF.

[0068] In an exemplary embodiment, after detecting the shadow area and the non-shadow area in the image to be detected, the enhancement coefficient corresponding to the image to be detected can be calculated based on the brightness of the shadow area and the non-shadow area in the image to be detected, and then the shadow of the image to be detected is removed by the enhancement coefficient to obtain the target image after the shadow is removed. For example, the shadow area of ​​the image can be enhanced by a low-light image enhancement algorithm. It should be noted that the low-light image enhancement algorithm is more excellent than the process of using the average brightness of the non-shadow part for compensation.

[0069] In an exemplary embodiment, when calculating the enhancement coefficient, the first average brightness of the shadow area and the second average brightness of the non-shadow area in the image to be detected can be calculated respectively, and then the enhancement coefficient corresponding to the image to be detected can be determined based on the first average brightness and the second average brightness.

[0070] For example, in one embodiment, the enhancement coefficient corresponding to the image to be detected can be calculated by the following formula (7):

[0071]

[0072] Where mu represents the enhancement coefficient; L Dark Indicates the first average brightness of the shadow area; L Bright Indicates the second average brightness of the non-shadow area.

[0073] In an exemplary embodiment, when shadow removal is performed on an image to be detected based on an enhancement coefficient, the image to be detected can be enhanced based on the enhancement coefficient to obtain an enhanced image, and then the shadow area in the enhanced image is combined with the non-shadow area in the image to be detected to obtain a target image after shadow removal.

[0074] In addition, after detecting the shadow area, in addition to performing the shadow removal operation, other processing may be performed on the shadow area, such as deepening the color of the shadow part, adding shadow light effects, etc., and the present disclosure does not impose any special restrictions on this.

[0075] The following Figure 8 The neural network structure shown takes the results of the first subnetwork and the second subnetwork of the shadow detection neural network as an example to explain the technical solution of the embodiment of the present disclosure in detail.

[0076] Reference Figure 8 As shown in the figure, the neural network structure uses ResNeXt101 as the front-end feature extraction module, that is, the first-stage network; drawing on the results of U-net, for the obtained feature layer of ResNeXt101, the extended path structure of U-net is used to gradually expand the feature layer size while reducing the feature layer depth, and finally a 1×1 convolutional layer and bilinear interpolation method are used to obtain the detection result.

[0077] It should be noted that for the 128×128×64 upper feature image, a non-local neural network can be added so that the upper feature image can provide accurate global information for each pixel and automatically build pixel relationships. In addition, the SE-Block attention module can be added to the last few layers of the network in the network sampling route to make the shadow area detection result more refined.

[0078] In the right Figure 8 When the first sub-network and the second sub-network shown are trained, the SBU shadow detection dataset can be used as a sample for training. SBU is the largest shadow detection dataset currently, which includes 4089 training images and 638 test images. Therefore, when training, 200 images can be selected from the test set as the sample set, and the images can be pre-processed and scaled to 256×256, and enhancement processes such as random horizontal flipping and rotation can be performed.

[0079] Reference Fig. 9 As shown, during training, for a sample image in a sample set, the brightness of the sample image can first be adjusted to obtain an adjusted image. The adjustment range can be 0.5-1.5 times. Then the sample image and the adjusted image are respectively input into the first sub-network and the second sub-network of the shared network parameters to obtain the first detection result and the second detection result.

[0080] After obtaining the first detection result and the second detection result, the first cross entropy can be calculated based on the first detection result and the true value, and the second cross entropy can be calculated based on the second detection result and the true value. When calculating specifically, the method shown in formula (1) can be used for calculation.

[0081] At the same time, because under normal circumstances, the second detection result corresponding to the adjusted image after brightness adjustment is more different from the true value than the first detection result corresponding to the sample image, but because the sample image and the adjusted image have the same content and the same target true value, in order to eliminate the influence caused by the different brightness, the relative entropy between the first detection result and the second detection result can also be calculated as part of the brightness limit loss function. When calculating specifically, the method shown in formulas (2) to (4) can be used for calculation.

[0082] After obtaining the above-mentioned first cross entropy, second cross entropy and relative entropy, the brightness limit loss function corresponding to the first sub-network and the second sub-network can be calculated based on formula (5) and formula (6), and then the network parameters of the first sub-network and the second sub-network that share the network parameters can be adjusted based on the brightness limit loss function. After adjusting the parameters of the first sub-network and the second sub-network based on the training set, the trained shadow detection model can be obtained. It should be noted that since the first sub-network and the second sub-network share network parameters, the shadow detection model can be the first sub-network or the second sub-network.

[0083] In summary, in this exemplary embodiment, a shadow area detection method focusing on brightness factors is provided. The different brightness caused by illumination is an important objective factor in the shadow removal task, and different brightness will have an adverse effect on the segmentation detection of shadows. In order to address this problem, a brightness limit loss function is designed in the neural network, and the original sample image and the adjusted image after brightness transformation are predicted at the same time, and then the loss between their prediction results is analyzed, and a new loss function is formed with the cross entropy loss used by the network to realize the training of the shadow detection model.

[0084] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0085] For further reference, Fig.10 As shown, in the implementation of this example, a shadow area processing device 1000 is also provided, including an area detection module 1010. Among them:

[0086] The region detection module 1010 can be used for the shadow detection model to perform shadow region detection on the image to be detected, and obtain the shadow region corresponding to the image to be detected.

[0087] The shadow detection model is obtained by training the constructed shadow detection neural network; the shadow detection neural network adjusts the network parameters based on the brightness limit loss function; the brightness limit loss function is determined based on the cross entropy and relative entropy calculated based on the first detection result of the sample image and the second detection result of the adjusted image obtained by adjusting the brightness of the sample image.

[0088] In an exemplary embodiment, the shadow detection neural network includes a first sub-network and a second sub-network having the same network structure, and the first sub-network and the second sub-network share network parameters.

[0089] In an exemplary embodiment, the first sub-network and the second sub-network both include a first-stage network and a second-stage network; the first-stage network includes an Encoder structure, and the second-stage network includes a Decoder structure.

[0090] In an exemplary embodiment, the Encoder structure includes a ResNeXt101 structure, and the Decoder structure includes an extended path structure in a U-net network.

[0091] In an exemplary embodiment, see Fig.11 As shown, the shadow area processing device 1100 also includes a shadow removal module 1110, which can be used to calculate the enhancement coefficient corresponding to the image to be detected based on the brightness of the shadow area and the non-shadow area in the image to be detected; and to remove the shadow of the image to be detected based on the enhancement coefficient to obtain the target image.

[0092] In an exemplary embodiment, the shadow removal module 1110 can be used to respectively calculate the first average brightness of the shadow area and the second average brightness of the non-shadow area in the image to be detected; and determine the enhancement coefficient corresponding to the image to be detected based on the first average brightness and the second average brightness.

[0093] In an exemplary embodiment, the shadow removal module 1110 may be used to enhance the image to be detected based on the enhancement coefficient to obtain an enhanced image; and combine the shadow area in the enhanced image with the non-shadow area in the image to be detected to obtain a target image.

[0094] In an exemplary embodiment, the region detection module 1010 may be used to post-process the boundary of the shadow region based on the conditional random field to obtain a processed shadow region.

[0095] In an exemplary embodiment, see Fig.12 As shown, the shadow area processing device 1200 further includes a training set acquisition module 1210 and a model training module 1220. Among them:

[0096] The training set acquisition module 1210 may be used to acquire a first training set including sample images, and perform brightness adjustment on the sample images in the first training set to obtain a second training set including adjusted images.

[0097] The model training module 1220 can be used to construct a shadow detection neural network, and train the shadow detection neural network based on the sample images included in the first training set and the adjusted images included in the second training set to obtain a shadow detection model.

[0098] In an exemplary embodiment, the model training module 1220 can be used to input a sample image in a first training set into a first subnetwork to obtain a first detection result, and input an adjusted image corresponding to the sample image in a second training set into a second subnetwork to obtain a second detection result; calculate a first cross entropy based on the first detection result, calculate a second cross entropy based on the second detection result, and calculate a relative entropy based on the first detection result and the second detection result; determine a brightness limit loss function based on the first cross entropy, the second cross entropy and the relative entropy; adjust the network parameters of the shadow detection neural network based on the brightness limit loss function to obtain a shadow detection model.

[0099] The specific details of each module in the above device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0100] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0101] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of the present specification, for example, it can execute Figure 3 and Figure 4 Any one or more steps in .

[0102] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0103] In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0104] In addition, program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0105] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0106] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A shadow area processing method, characterized in that: include: Perform shadow area detection on the image to be detected based on the shadow detection model to obtain the shadow area corresponding to the image to be detected; The shadow detection model is obtained by training a constructed shadow detection neural network; the shadow detection neural network adjusts network parameters based on a brightness limit loss function; the brightness limit loss function is determined based on a cross entropy and a relative entropy calculated based on a first detection result of a sample image and a second detection result of an adjusted image obtained by adjusting the brightness of the sample image; the adjustment range of the brightness adjustment of the adjusted image is 0.5-1.5 times; The method further comprises: Acquire a first training set including the sample images, and perform brightness adjustment on the sample images in the first training set to obtain a second training set including adjusted images; Constructing a shadow detection neural network, and training the shadow detection neural network based on the sample images included in the first training set and the adjusted images included in the second training set to obtain a shadow detection model; The shadow detection neural network includes a first sub-network and a second sub-network with the same network structure, and the first sub-network and the second sub-network share network parameters; The step of training the shadow detection neural network based on the sample images included in the first training set and the adjusted images included in the second training set to obtain a shadow detection model includes: Input the sample image in the first training set into a first sub-network to obtain a first detection result, and input the adjusted image corresponding to the sample image in the second training set into a second sub-network to obtain a second detection result; Calculating a first cross entropy based on the first detection result, calculating a second cross entropy based on the second detection result, and calculating a relative entropy based on the first detection result and the second detection result; Determine the brightness limit loss function based on the first cross entropy, the second cross entropy and the relative entropy; The network parameters of the shadow detection neural network are adjusted based on the brightness limit loss function to obtain a shadow detection model.

2. The method according to claim 1, characterized in that The shadow detection neural network includes a first sub-network and a second sub-network having the same network structure, and the first sub-network and the second sub-network share network parameters.

3. The method according to claim 2, characterized in that The first sub-network and the second sub-network both include a first-stage network and a second-stage network; The first-stage network includes an Encoder structure, and the second-stage network includes a Decoder structure.

4. The method according to claim 3, characterized in that The Encoder structure includes a ResNeXt101 structure, and the Decoder structure includes an extended path structure in a U-net network.

5. The method according to claim 1, characterized in that The method further comprises: Calculating an enhancement coefficient corresponding to the image to be detected based on the brightness of the shadow area and the non-shadow area in the image to be detected; The target image is obtained by performing shadow removal on the image to be detected based on the enhancement coefficient.

6. The method according to claim 5, characterized in that The step of calculating the enhancement coefficient corresponding to the image to be detected based on the brightness of the shadow area and the non-shadow area in the image to be detected comprises: Respectively calculating a first average brightness of the shadow area and a second average brightness of the non-shadow area in the image to be detected; An enhancement coefficient corresponding to the image to be detected is determined based on the first average brightness and the second average brightness.

7. The method according to claim 5, characterized in that The step of performing shadow removal on the image to be detected based on the enhancement coefficient to obtain a target image includes: Enhance the image to be detected based on the enhancement coefficient to obtain an enhanced image; The shadow area in the enhanced image is combined with the non-shadow area in the image to be detected to obtain a target image.

8. The method according to claim 1, characterized in that After obtaining the shadow area corresponding to the image to be detected, the method further includes: The boundary of the shadow area is post-processed based on the conditional random field to obtain a processed shadow area.

9. A shadow area processing device, characterized in that: include: A region detection module is used to perform shadow region detection on the image to be detected based on the shadow detection model to obtain the shadow region corresponding to the image to be detected; The shadow detection model is obtained by training a constructed shadow detection neural network; the shadow detection neural network adjusts network parameters based on a brightness limit loss function; the brightness limit loss function is determined based on a cross entropy and a relative entropy calculated based on a first detection result of a sample image and a second detection result of an adjusted image obtained by adjusting the brightness of the sample image; the adjustment range of the brightness adjustment of the adjusted image is 0.5-1.5 times; The shadow area processing device also includes a training set acquisition module and a model training module; wherein: The training set acquisition module is used to acquire a first training set including the sample images, and perform brightness adjustment on the sample images in the first training set to obtain a second training set including adjusted images; The model training module is used to construct a shadow detection neural network, and the shadow detection neural network is trained based on each of the sample images included in the first training set and each of the adjusted images included in the second training set to obtain a shadow detection model; the shadow detection neural network includes a first subnetwork and a second subnetwork with the same network structure, and the first subnetwork and the second subnetwork share network parameters; The model training module is used to input the sample image in the first training set into the first subnetwork to obtain a first detection result, and input the adjusted image corresponding to the sample image in the second training set into the second subnetwork to obtain a second detection result; calculate the first cross entropy based on the first detection result, calculate the second cross entropy based on the second detection result, and calculate the relative entropy based on the first detection result and the second detection result; determine the brightness limit loss function based on the first cross entropy, the second cross entropy and the relative entropy; adjust the network parameters of the shadow detection neural network based on the brightness limit loss function to obtain a shadow detection model.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

11. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 8 by executing the executable instructions.

Images