Method, device, equipment and medium for extracting shadow area of ​​object to be detected

Abstract

The present application discloses a method, device, equipment and medium for extracting the shadow area of ​​an object to be detected, the method comprising: receiving a remote sensing image, obtaining an extension image of the object to be detected in the remote sensing image; performing a preset semantic segmentation process on the extension image to obtain a target initial shadow image of the object to be detected; performing a noise removal process on the target initial shadow image to obtain the shadow area of ​​each object to be detected. The present application solves the technical problem that the effect of extracting the shadow area of ​​an object to be detected in the prior art is unstable.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology in financial technology (Fintech), and in particular to a method, device, equipment and medium for extracting the shadow area of ​​an object to be detected. Background Art

[0002] With the continuous development of financial technology, especially Internet technology finance, more and more technologies are applied in the financial field, but the financial industry also puts forward higher requirements on technology. For example, the financial industry also has higher requirements for the extraction of shadow area of ​​detected objects.

[0003] That is, today is the era of crude oil. Real-time understanding of crude oil reserve data of various countries is directly related to national security. At present, crude oil reserve estimation or forecast is usually carried out by monitoring crude oil ports of various countries through satellites to obtain remote sensing images to calculate the capacity of each large object to be detected in the port, such as a large oil tank. In the process of calculating the capacity of each large oil tank in the port, shadow area extraction is the most important link. At present, the shadow area of ​​large oil tanks is often extracted by color threshold segmentation in RGB (three primary colors, red, green and blue, Red, Green, Blue) color space, which leads to unstable shadow extraction effect of objects to be detected. Summary of the invention

[0004] The main purpose of the present application is to provide a method, device, equipment and medium for extracting the shadow area of ​​an object to be detected, aiming to solve the technical problem of unstable extraction effect in the current process of extracting the shadow area of ​​an object to be detected in the prior art.

[0005] To achieve the above object, the present application provides a method for extracting the shadow area of ​​an object to be detected, the method comprising:

[0006] Receiving a remote sensing image and obtaining an external image of an object to be detected in the remote sensing image;

[0007] Performing a preset semantic segmentation process on the external image to obtain an initial shadow image of the target object to be detected;

[0008] The target initial shadow image is subjected to noise removal processing to obtain the shadow area of ​​each object to be detected.

[0009] Optionally, the step of performing preset semantic segmentation processing on the external extension image to obtain the target initial shadow image of the object to be detected includes:

[0010] Inputting the external extension image into a preset semantic segmentation network model to perform preset semantic segmentation processing on the external extension image to obtain an initial shadow image of the target of the object to be detected;

[0011] The preset semantic segmentation network model is a model of predicted shadow images obtained by training a preset basic model to be trained based on preset image data of an object to be detected with a preset shadow label.

[0012] Optionally, before the step of inputting the externalized image into a preset semantic segmentation network model to obtain the target initial shadow image of each object to be detected, the method includes:

[0013] Acquire preset image data of the object to be detected, and iteratively train the preset basic model to be trained based on the preset image data of the object to be detected, so as to update the preset prediction model to be trained through iterative training;

[0014] It is determined whether the preset prediction model to be trained after iterative training and updating satisfies the preset training completion condition. If the preset training completion condition is met, the preset semantic segmentation network model is obtained.

[0015] Optionally, the step of acquiring preset image data of the object to be detected, iteratively training the preset basic model to be trained based on the preset image data of the object to be detected, and updating the preset prediction model to be trained by iterative training includes:

[0016] Acquire preset image data of an object to be detected, and input the preset image data of the object to be detected into the preset basic model to be trained to obtain predicted probability image data, wherein the preset basic model to be trained includes a skip connection layer;

[0017] Acquire preset shadow label data of the preset image data of the object to be detected, and compare the preset shadow label data with the predicted probability image data to obtain difference data;

[0018] The preset prediction model to be trained is updated by iterative training according to the difference data.

[0019] Optionally, the step of receiving a remote sensing image and obtaining an external extension image of the object to be detected in the remote sensing image includes:

[0020] Receiving a remote sensing image, obtaining an image of a bounding rectangular frame of an object to be detected in the remote sensing image, and obtaining position information of the image of the bounding rectangular frame;

[0021] According to the position information of the circumscribed rectangular frame image, an image including the circumscribed rectangular frame image and having a size that is a preset multiple of the circumscribed rectangular frame image is cropped from the remote sensing image with the circumscribed rectangular frame image as the center, and the image including the circumscribed rectangular frame image and having a size that is a preset multiple of the circumscribed rectangular frame image is set as the external extension image.

[0022] Optionally, the step of performing noise removal processing on the target initial shadow image to obtain the shadow area of ​​each object to be detected includes:

[0023] Determine an outward-extended rectangular frame of each object to be detected according to the outward-extended image of each object to be detected;

[0024] Determine an intersecting image in the initial shadow image that intersects with the outward-extended rectangular frame, and remove the intersecting image to obtain a first processed image;

[0025] A first preset shadow area threshold is obtained, and images having an area greater than the preset shadow area threshold are extracted from the first processed image to obtain the shadow area of ​​each object to be detected.

[0026] Optionally, the step of obtaining a first preset shadow area threshold, extracting images having an area greater than the preset shadow area threshold from the first processed image, and obtaining the shadow area of ​​each object to be detected includes:

[0027] Obtaining a first preset shadow area threshold, extracting an image whose area is larger than the first preset shadow area threshold from the first processed image, and obtaining a second processed image;

[0028] A second preset shadow area threshold is obtained, an image having an area greater than the second preset shadow area threshold is extracted from the second processed image to obtain a third processed image, an image intersecting with the bounding box of the bounding box image is extracted from the third processed image, and a shadow area of ​​each object to be detected is obtained.

[0029] Optionally, the step of inputting the external extension image into a preset semantic segmentation network model to perform preset semantic segmentation processing on the external extension image to obtain the target initial shadow image of the object to be detected includes:

[0030] Preprocessing the external extension image to obtain a preprocessed image;

[0031] The pre-processed image is input into a preset semantic segmentation network model to perform preset semantic segmentation processing on the external extension image to obtain the target initial shadow image of the object to be detected.

[0032] The present application also provides a device for extracting the shadow area of ​​an object to be detected, the device comprising:

[0033] A receiving module, used for receiving a remote sensing image and obtaining an external image of an object to be detected in the remote sensing image;

[0034] A first acquisition module is used to perform preset semantic segmentation processing on the external image to obtain an initial shadow image of the target object to be detected;

[0035] The noise removal module is used to perform noise removal processing on the target initial shadow image to obtain the shadow area of ​​each object to be detected.

[0036] Optionally, the first acquisition module includes:

[0037] A speech segmentation unit, used for inputting the external extension image into a preset semantic segmentation network model to perform preset semantic segmentation processing on the external extension image to obtain a target initial shadow image of the object to be detected;

[0038] The preset semantic segmentation network model is a model of predicted shadow images obtained by training a preset basic model to be trained based on preset image data of an object to be detected with a preset shadow label.

[0039] Optionally, the device for extracting the shadow area of ​​the object to be detected includes:

[0040] A second acquisition module is used to acquire preset image data of the object to be detected, and iteratively train the preset basic model to be trained based on the preset image data of the object to be detected, so as to update the preset prediction model to be trained through iterative training;

[0041] The judgment module is used to judge whether the preset prediction model to be trained after iterative training and updating meets the preset training completion conditions. If the preset training completion conditions are met, the preset semantic segmentation network model is obtained.

[0042] Optionally, the second acquisition module includes:

[0043] A first acquisition unit is used to acquire preset image data of an object to be detected, and input the preset image data of the object to be detected into the preset basic model to be trained to obtain predicted probability image data, wherein the preset basic model to be trained includes a skip connection layer;

[0044] A second acquisition unit is used to acquire preset shadow label data of the preset image data of the object to be detected, and compare the preset shadow label data with the predicted probability image data to obtain difference data;

[0045] A training unit is used to iteratively train and update the preset prediction model to be trained according to the difference data.

[0046] Optionally, the receiving module includes:

[0047] A receiving unit, used to receive a remote sensing image, obtain a circumscribed rectangular frame image of an object to be detected in the remote sensing image, and obtain position information of the circumscribed rectangular frame image;

[0048] The extension unit is used to, based on the position information of the circumscribed rectangular frame image and with the circumscribed rectangular frame image as the center, crop from the remote sensing image to obtain an image including the circumscribed rectangular frame image and having a size that is a preset multiple of the circumscribed rectangular frame image, and set the image including the circumscribed rectangular frame image and having a size that is a preset multiple of the circumscribed rectangular frame image as the extension image.

[0049] Optionally, the noise removal module includes:

[0050] A first determining unit, configured to determine an outer extension rectangular frame of each object to be detected according to the outer extension image of each object to be detected;

[0051] A second determining unit is used to determine an intersecting image in the initial shadow image that intersects with the outer extension rectangular frame, and remove the intersecting image to obtain a first processed image;

[0052] The third acquisition unit is used to acquire a first preset shadow area threshold, extract images with an area larger than the preset shadow area threshold from the first processed image, and obtain the shadow area of ​​each object to be detected.

[0053] Optionally, the third acquiring unit includes:

[0054] A first acquisition subunit is used to acquire a first preset shadow area threshold, extract an image with an area larger than the first preset shadow area threshold from the first processed image, and obtain a second processed image;

[0055] The second acquisition subunit is used to obtain a second preset shadow area threshold, extract an image whose area is greater than the second preset shadow area threshold from the second processed image to obtain a third processed image, extract an image that intersects with the external frame of the external frame image from the third processed image, and obtain the shadow area of ​​each object to be detected.

[0056] Optionally, the first acquisition module includes:

[0057] A preprocessing unit, used for preprocessing the external extension image to obtain a preprocessed image;

[0058] The input unit is used to input the pre-processed image into a preset semantic segmentation network model to perform preset semantic segmentation processing on the external extension image to obtain the target initial shadow image of the object to be detected.

[0059] The present application also provides a device for extracting the shadow area of ​​an object to be detected, which is a physical device. The device for extracting the shadow area of ​​an object to be detected includes: a memory, a processor, and a program of the method for extracting the shadow area of ​​the object to be detected stored in the memory and executable on the processor. When the program of the method for extracting the shadow area of ​​the object to be detected is executed by the processor, the steps of the method for extracting the shadow area of ​​the object to be detected as described above can be implemented.

[0060] The present application also provides a medium, on which is stored a program for implementing the above-mentioned method for extracting the shadow area of ​​the object to be detected. When the program for the method for extracting the shadow area of ​​the object to be detected is executed by a processor, the steps of the above-mentioned method for extracting the shadow area of ​​the object to be detected are implemented.

[0061] The present application receives a remote sensing image to obtain an external extension image of the object to be detected in the remote sensing image; performs a preset semantic segmentation process on the external extension image to obtain a target initial shadow image of the object to be detected; performs a noise removal process on the target initial shadow image to obtain the shadow area of ​​each object to be detected. In the present application, after obtaining the external extension image of the object to be detected based on the remote sensing image, the external extension image is subjected to a preset semantic segmentation process to obtain the target initial shadow image of the object to be detected (since the preset semantic segmentation process is not affected by natural factors, the extraction accuracy of the shadow area of ​​the object to be detected is improved), and then, after the target initial shadow image is subjected to noise removal, the shadow area of ​​each object to be detected is accurately obtained. In the present application, the preset semantic segmentation network model is not affected by the lighting when the remote sensing image is collected, thereby avoiding the instability of the shadow extraction effect of the object to be detected. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0063] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0064] Figure 1 This is a flow chart of a first embodiment of a method for extracting the shadow area of ​​an object to be detected in the present application;

[0065] Figure 2 This is a flow chart of a second embodiment of the method for extracting the shadow area of ​​an object to be detected in the present application;

[0066] Figure 3 A schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present application;

[0067] Figure 4 This is a schematic diagram of the first scene in the method for extracting the shadow area of ​​an object to be detected in this application;

[0068] Figure 5 A schematic diagram of a second scene of the method for extracting the shadow area of ​​an object to be detected in this application;

[0069] Figure 6 This is a schematic diagram of a third scene in the method for extracting the shadow area of ​​an object to be detected in this application.

[0070] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0071] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0072] The present application embodiment provides a method for extracting the shadow area of ​​an object to be detected. In the first embodiment of the method for extracting the shadow area of ​​an object to be detected, refer to Figure 1 , the method for extracting the shadow area of ​​the object to be detected includes:

[0073] Step S10, receiving a remote sensing image, and obtaining an extension image of the object to be detected in the remote sensing image;

[0074] Step S20, performing a preset semantic segmentation process on the external image to obtain an initial shadow image of the target object to be detected;

[0075] Step S30 , performing noise removal processing on the target initial shadow image to obtain the shadow area of ​​each object to be detected.

[0076] The specific steps are as follows:

[0077] Step S10, receiving a remote sensing image, and obtaining an extension image of the object to be detected in the remote sensing image;

[0078] In today's era, real-time knowledge of various data reserves of other countries, such as data reserves of objects to be detected, is directly related to the national economy and people's livelihood, and real-time knowledge of crude oil reserve data of various countries is even more directly related to national security. That is, in this implementation, the object to be detected is an oil tank, especially a large oil tank, as an example for specific explanation (but not limited to oil tanks, for example, the object to be detected can also be a container, etc.). At present, crude oil reserve estimation or forecasting is usually carried out by monitoring crude oil ports of various countries through satellites to obtain remote sensing images. Specifically, to obtain remote sensing images of ports, to calculate the capacity of each large oil tank in the port based on the remote sensing images of ports, in the process of calculating the capacity of each large oil tank in the port, shadow area extraction is the most important link, that is, the calculation of the capacity of each large oil tank in the port can generally be divided into three steps: oil tank detection and positioning, oil tank shadow Shadow area extraction and calculation of oil tank volume, among which shadow area extraction of oil tank is a very important link. At present, the shadow area of ​​oil tank is often extracted by color threshold segmentation in RGB (three primary colors, red, green and blue, Red, Green, Blue) color space. The color threshold segmentation in RGB color space causes the shadow extraction of oil tank to be greatly affected by the illumination when the image was previously collected, and the extraction effect is unstable. That is, when the RGB color space is used to express the shadow area of ​​oil tank, the threshold value calculated when extracting the shadow area of ​​oil tank images extracted under different illumination changes too much. For example, at sunrise, the calculated A threshold value is too different from the B threshold value calculated at sunset or fluctuates too much, which will cause the size of the extracted shadow area to be unstable, thereby affecting the extraction effect.

[0079] In this embodiment, a remote sensing image sent by a satellite is received, and an external frame image of each object to be detected is determined based on the remote sensing image to obtain the position information of each external frame image. Specifically, if the object to be detected is a circular oil tank, an external rectangular frame image of each object to be detected is determined based on the port remote sensing image to obtain the position information of each external frame image. If the object to be detected is a container, an outer container image of each object to be detected is determined based on the remote sensing image to obtain the position information of each outer container image. It should be noted that in this embodiment, if the object to be detected is a circular oil tank, an external rectangular frame image of each object to be detected is determined based on the port remote sensing image. , taking the acquisition of the position information of each of the external frame images as an example for specific explanation, specifically, through the preset Rotated-Faster-R-CNN model (a model that has been trained to accurately locate the position information of the external frame images of oil tanks in the port remote sensing images), the remote sensing image is analyzed to determine the external frame image of each object to be detected to obtain the position information of each of the external frame images. It should be noted that the position information of each of the external frame images can be expressed by (c_x, c_y, w, h), wherein (c_x, c_y) represents the center coordinate point of each external frame image, (w) represents the width of each external frame, and (h) represents the height of each external frame.

[0080] In this embodiment, since it is necessary to extract the shadow area of ​​the object to be detected, it is necessary to first extract the image including the shadow area. Specifically, based on the position information of the circumscribed frame image, an external image of the object to be detected including the shadow of the object to be detected is obtained with the circumscribed frame image as the center.

[0081] Specifically, the step of receiving a remote sensing image and obtaining an external extension image of the object to be detected in the remote sensing image includes:

[0082] Step S11, receiving a remote sensing image, obtaining a circumscribed rectangular frame image of an object to be detected in the remote sensing image, and obtaining position information of the circumscribed rectangular frame image;

[0083] Receive a remote sensing image sent by a satellite, and determine the circumscribed rectangular frame image of each object to be detected based on the remote sensing image to obtain the position information of each of the circumscribed rectangular frame images. Specifically, analyze the remote sensing image through a preset Rotated-Faster-R-CNN model (a model that has been trained to accurately locate the position information of the circumscribed rectangular frame image of the oil tank in the remote sensing image), determine the circumscribed rectangular frame image of each object to be detected, and obtain the position information of each of the circumscribed rectangular frame images. It should be noted that the position information of each of the circumscribed rectangular frame images can be represented by (c_x, c_y, w, h), where (c_x, c_y) represents the center coordinate point of each circumscribed rectangular frame image, (w) represents the width of each circumscribed rectangular frame, and (h) represents the height of each circumscribed rectangular frame.

[0084] Step S12, according to the position information of the circumscribed rectangular frame image, with the circumscribed rectangular frame image as the center, cropping from the remote sensing image to obtain an image including the circumscribed rectangular frame image and having a size that is a preset multiple of the circumscribed rectangular frame image, and setting the image including the circumscribed rectangular frame image and having a size that is a preset multiple of the circumscribed rectangular frame image as the external extension image.

[0085] According to the position information of the circumscribed rectangular frame image, an image including the circumscribed rectangular frame image and having a size that is a preset multiple of the circumscribed rectangular frame image is cropped from the remote sensing image with the circumscribed rectangular frame image as the center: Specifically, according to the position information of each circumscribed rectangular frame image, an image of each object to be detected in each remote sensing image is expanded outward. There are multiple objects to be detected in each remote sensing image. Therefore, when an object to be detected is expanded outward (with the circumscribed rectangular frame image as the center, an image including the circumscribed rectangular frame image and having a size that is a preset multiple of the circumscribed rectangular frame image is cropped from the remote sensing image), it may include images or shadow images of other objects to be detected. The image including the circumscribed rectangular frame image and having a size that is a preset multiple of the circumscribed rectangular frame image is set as the expanded image. Figure 6 As shown, (a) is the circumscribed rectangular frame image before external expansion, and (b) is the external expansion image after external expansion.

[0086] In this embodiment, an image including the circumscribed rectangular frame image is cropped from the remote sensing image with the circumscribed rectangular frame image as the center, and the image whose size is a preset multiple of the circumscribed rectangular frame image, and the preset multiple can be 1-2. In particular, the preset expansion ratio can be 1.8. The purpose of setting the preset expansion ratio is to obtain an image including the shadow of the object to be detected (i.e., to avoid incomplete shadows) while not over-processing redundant images to avoid increasing the processing burden. After obtaining the external extension image, specifically, the width of the external extension rectangular frame width = (1 + rates) * w, the height of the external extension rectangular frame height = (1 + rates) * h, the new rectangular frame can be expressed as (new_x, new_y, new_w, new_h), rates represents the difference between the preset multiple and 1, as shown in Figure 4 shown.

[0087] It should be noted that, in order to process only the outer extension image, after obtaining the outer extension image, the outer extension image is cropped so as to input the outer extension image into the preset semantic segmentation network model.

[0088] Step S20, performing a preset semantic segmentation process on the external image to obtain an initial shadow image of the target object to be detected;

[0089] The external extension image is subjected to preset semantic segmentation processing, wherein semantics in the field of image processing may refer to the content of the image, and the external extension image is subjected to preset semantic segmentation processing may refer to segmenting pixels in the external extension image by preset label features or preset coding features, wherein specifically, the pixels in the external extension image may be segmented by machine learning or neural networks.

[0090] Specifically, the step of performing preset semantic segmentation processing on the external extension image to obtain the target initial shadow image of the object to be detected includes:

[0091] Step S21, inputting the external extension image into a preset semantic segmentation network model to perform preset semantic segmentation processing on the external extension image to obtain an initial shadow image of the target of the object to be detected;

[0092] The preset semantic segmentation network model is a model of predicted shadow images obtained by training a preset basic model to be trained based on preset image data of an object to be detected with a preset shadow label.

[0093] In the present embodiment, it should be noted that the preset semantic segmentation network model is based on the preset image data of the object to be detected with a preset shadow label, and is a model for accurately predicting the initial shadow image obtained by training the preset basic model to be trained. Since the preset semantic segmentation network model is a model that can accurately predict the initial shadow image after training, after the external extension image is input into the preset semantic segmentation network model, the target external extension image can be subjected to preset semantic segmentation processing to accurately obtain the target initial shadow image of the object to be detected. It should be noted that the target initial shadow image is the current initial shadow image of the object to be detected that needs to be processed, or the latest initial shadow image of the object to be detected that needs to be processed the most recently. Since the object to be detected can be both an oil tank and a container, etc., in this embodiment, the preset semantic segmentation network model has multiple sub-models, such as a preset oil tank semantic segmentation network sub-model and a preset container semantic segmentation network sub-model. Therefore, after the external extension image is input into the preset semantic segmentation network model, it is necessary to obtain the category of the external extension image, and then, according to the category of the external extension image, the external extension image is input into the corresponding sub-model such as the preset oil tank semantic segmentation network sub-model. In this embodiment, since the preset semantic segmentation network model includes multiple sub-models, it can improve the convenience of detecting the object to be detected.

[0094] Wherein, before the step of inputting the external extension image into a preset semantic segmentation network model to obtain the target initial shadow image of each object to be detected, the method includes:

[0095] Step A1, obtaining preset image data of an object to be detected, and iteratively training the preset basic model to be trained based on the preset image data of the object to be detected, so as to update the preset prediction model to be trained by iterative training;

[0096] In this embodiment, the preset semantic segmentation network model is accurately obtained. Specifically, the preset image data of the object to be detected is first obtained. The preset image data of the object to be detected includes both each preset image of the object to be detected and the preset shadow label corresponding to each preset image of the object to be detected. That is, in this embodiment, the corresponding mask image (including the preset shadow label) is first obtained based on the original image of the preset image of the object to be detected. In the mask image, the shadow area is marked as "1" and the background area is marked as "0". The preset image data of the object to be detected is obtained, and the preset basic model to be trained is iteratively trained based on each preset image of the object to be detected in the preset image data of the object to be detected, so as to update the preset prediction model to be trained by iterative training. Specifically, the training result (or predicted probability image data) of the iterative training is compared with the expected result of the preset shadow label in the corresponding mask image, so as to update the preset prediction model to be trained by iterative training, and specifically, the network weight variables in the preset prediction model to be trained are updated by iterative training.

[0097] Step A2, determining whether the preset prediction model to be trained after iterative training and updating meets the preset training completion conditions, and if the preset training completion conditions are met, obtaining the preset semantic segmentation network model.

[0098] Determine whether the preset prediction model to be trained after iterative training updates meets the preset training completion conditions. If the preset training completion conditions are met, obtain the preset semantic segmentation network model. Specifically, the preset training completion conditions may be that the iterations reach a preset number of times, or that the preset loss function converges. It should be noted that, for each iterative training, the result of the iterative training is compared with the expected result of the preset shadow label in the corresponding mask image to determine the difference data, and adjust the preset prediction model to be trained in a directional manner according to the difference data, in particular, adjust the network weight variables to finally obtain the preset semantic segmentation network model. It should be noted that the preset prediction model to be trained includes a feature extraction part and an upsampling part, etc., wherein the feature extraction part includes a convolution layer, a pooling layer, etc., and the upsampling part includes a deconvolution layer, etc., and the feature extraction part and the upsampling part may belong to the same network architecture, or may belong to different network architectures. For example, the feature extraction part and the upsampling part may both belong to the Unet network architecture (belonging to a plain network), or the feature extraction part may belong to the Unet network architecture, and the upsampling part may belong to the ResNet network architecture (belonging to a non-plain network).

[0099] Among them, refer to Figure 2 The step of inputting the external image into a preset semantic segmentation network model to obtain the target initial shadow image of each object to be detected includes:

[0100] Step S211, preprocessing the external extension image to obtain a preprocessed image;

[0101] It should be noted that due to the different outward extension ratios (preset multiples) of different images, such as the outward extension ratio of the current outward extension image is 1.8, and the outward extension ratio (preset multiples) of another outward extension image is 1.5, that is, the sizes of the outward extension images obtained by the two are different, and the preset semantic segmentation network model processes images of a certain size or a certain brightness, etc. In order to improve the convenience of model processing, in this embodiment, after obtaining the outward extension image, before inputting the outward extension image into the preset semantic segmentation network model, the outward extension image is preprocessed (that is, the outward extension image is stretched to a preset size or the brightness of the outward extension image is adjusted, etc.) to obtain a preprocessed image.

[0102] Step S212: input the pre-processed image into a preset semantic segmentation network model to perform preset semantic segmentation processing on the external image to obtain an initial shadow image of the target of the object to be detected.

[0103] After the preprocessed image is obtained, the preprocessed image is input into a preset semantic segmentation network model to perform preset semantic segmentation processing on the externalized image to obtain an initial shadow image of the target of each object to be detected.

[0104] Step S30 , performing noise removal processing on the target initial shadow image to obtain the shadow area of ​​each object to be detected.

[0105] The target initial shadow image is subjected to noise removal processing, that is, noise shadow removal, to obtain the shadow area of ​​each object to be detected.

[0106] The present application receives a remote sensing image to obtain an external extension image of the object to be detected in the remote sensing image; performs a preset semantic segmentation process on the external extension image to obtain a target initial shadow image of the object to be detected; performs a noise removal process on the target initial shadow image to obtain the shadow area of ​​each object to be detected. In the present application, after obtaining the external extension image of the object to be detected based on the remote sensing image, the external extension image is subjected to a preset semantic segmentation process to obtain the target initial shadow image of the object to be detected (since the preset semantic segmentation process is not affected by natural factors, the extraction accuracy of the shadow area of ​​the object to be detected is improved), and then, after the target initial shadow image is subjected to noise removal, the shadow area of ​​each object to be detected is accurately obtained. In the present application, the preset semantic segmentation network model is not affected by the lighting when the remote sensing image is collected, thereby avoiding the instability of the shadow extraction effect of the object to be detected.

[0107] Further, refer to Figure 2 Based on the first embodiment of the present application, in another embodiment of the present application, the steps of obtaining the preset image data of the object to be detected, iteratively training the preset basic model to be trained based on the preset image data of the object to be detected, and updating the preset prediction model to be trained by iterative training include:

[0108] Step B1, obtaining preset image data of an object to be detected, and inputting the preset image data of the object to be detected into the preset basic model to be trained to obtain predicted probability image data, wherein the preset basic model to be trained includes a skip connection layer;

[0109] After obtaining the preset image data of the object to be detected and inputting the preset image data of the object to be detected into the preset basic model to be trained, the predicted probability image data is obtained. In this embodiment, the preset basic model to be trained includes a skip connection layer, and in particular, the feature extraction part includes a skip connection layer. Specifically, in the preset basic model to be trained, if the feature extraction part includes a Unet network coding stage or a Unet network architecture (excluding a skip connection layer), it is changed to a ResNet network coding stage or a ResNet network architecture (including a skip connection layer). Specifically, the role of the skip connection layer is: after the preset number of convolutions, the information discarded before the preset number of convolutions is processed and added to the data of the preset number of convolutions to avoid information loss caused by the convolution of the image data. In this embodiment, in the network decoding process or in the upsampling part, the Unet decoding network stage is still used to obtain a probability map of size (C, W, H), where C is the number of predicted categories (C = target category number N + background category, such as C = 2, only the shadow area is predicted).

[0110] Step B2, obtaining preset shadow label data of the preset image data of the object to be detected, and comparing the preset shadow label data with the predicted probability image data to obtain difference data;

[0111] Step B3, iteratively training and updating the preset prediction model to be trained according to the difference data.

[0112] In this embodiment, the preset shadow label data of the preset image data of the object to be detected is obtained, the preset shadow label data is compared with the predicted probability image data to obtain difference data, and the preset prediction model to be trained is updated through iterative training according to the difference data. Wherein, after obtaining the predicted probability image data and before obtaining the difference data, if the predicted probability image data is in the form of a onehot (unique hot encoding) label map, and the preset shadow label data is in the form of a mask label map, it is necessary to convert the onehot label map form into a mask label map form or convert the mask label map form into a onehot label map form, wherein the conversion of the above two forms can be achieved by stripping or merging pixel point probabilities, for example, if pixel a is a shadow, the mask label map form is expressed by {0.7 (shadow probability), 0.3 (background probability)}, and the onehot (unique hot encoding) label map form is directly expressed by {1 (shadow probability)}, after the unified form, data comparison is performed to obtain difference data, and the preset prediction model to be trained is updated through iterative training according to the difference data.

[0113] This embodiment obtains the preset image data of the object to be detected, inputs the preset image data of the object to be detected into the preset basic model to be trained, and obtains the predicted probability image data, wherein the preset basic model to be trained includes a skip connection layer; obtains the preset shadow label data of the preset image data of the object to be detected, compares the preset shadow label data with the predicted probability image data, and obtains difference data; and iteratively trains and updates the preset prediction model to be trained according to the difference data. In this embodiment, since the preset basic model to be trained includes a skip connection layer, the preset semantic segmentation network model is obtained by training, thereby avoiding the situation where the loss of information during the training process causes inaccurate training, thereby improving the accuracy of the training.

[0114] Further, based on the first embodiment and the second embodiment of the present application, the step of performing noise removal processing on the target initial shadow image to obtain the shadow area of ​​each object to be detected includes:

[0115] Step D1, determining an outer extension rectangular frame of each object to be detected according to the outer extension image of each object to be detected;

[0116] An outer extension rectangular frame of the object to be detected is determined according to the outer extension image of the object to be detected, wherein the outer extension rectangular frame refers to a boundary image or a frame image of the outer extension image.

[0117] Step D2, determining an intersecting image in the initial shadow image that intersects with the outward-extended rectangular frame, and performing a removal process on the intersecting image to obtain a first processed image;

[0118] like Figure 4 As shown, determine the intersecting image in the initial shadow image that intersects with the outer extension rectangular frame (specifically, the shadow image that intersects with the outer extension rectangular frame but does not intersect with the outer frame image, such as Figure 5 The intersecting image is removed to obtain a first processed image. It should be noted that the first processed image may also have other noise shadow areas that do not intersect with the outer extension rectangular frame.

[0119] Step D3, obtaining a first preset shadow area threshold, extracting images whose areas are larger than the preset shadow area threshold from the first processed image, and obtaining the shadow area of ​​each object to be detected.

[0120] A first preset shadow area threshold is obtained, and an image with an area greater than the first preset shadow area threshold is extracted from the first processed image to obtain the shadow area of ​​each object to be detected. That is, in this embodiment, small noise shadows in the image are removed through a preset morphological opening and closing operation. Specifically, the contours of all shadows in the image are determined, and small spot shadows or small contour shadows with an area less than a preset specified threshold (t3) are eliminated, and finally the shadow area of ​​each object to be detected is obtained.

[0121] The step of obtaining a first preset shadow area threshold, extracting images having an area greater than the first preset shadow area threshold from the first processed image, and obtaining the shadow area of ​​each object to be detected comprises:

[0122] Step E1, obtaining a first preset shadow area threshold, extracting an image whose area is larger than the first preset shadow area threshold from the first processed image, and obtaining a second processed image;

[0123] Step E2, obtaining a second preset shadow area threshold, extracting an image whose area is greater than the second preset shadow area threshold from the second processed image to obtain a third processed image, extracting an image that intersects with the bounding box of the bounding box image from the third processed image, and obtaining the shadow area of ​​each object to be detected.

[0124] It should be noted that the first processed image may also have other noise shadow areas that do not intersect with the external image. Therefore, after obtaining the first processed image, the first preset shadow area threshold is obtained, and images with an area larger than the first preset shadow area threshold are extracted from the first processed image to obtain the second processed image. Specifically, through the preset morphological opening and closing operation (only used for small spot processing), images with an area larger than the first preset shadow area threshold are extracted from the first processed image to achieve the removal of small noise shadows or small spots in the image (small noise shadows or small spots refer to those smaller than the first preset shadow area threshold). For example, first, based on the area of ​​all shadow contours in the target shadow image, those smaller than the preset specified threshold, i.e., the first preset shadow area threshold (t3) are eliminated. , and finally obtain the second processed image. After the first denoising process, most of the noise is removed, but there are still other large noises. Therefore, after obtaining the second processed image, the area of ​​a single shadow contour is counted, and the area of ​​the shadow contour whose area is greater than the specified threshold (t4), i.e., the second preset shadow area threshold, is extracted. It should be noted that, in this embodiment, the removal of noise shadows is performed in batches, which can improve the efficiency of noise shadow removal (avoiding the removal jam caused by one-time processing or avoiding the inability to use morphological opening and closing operations, etc.), and obtain the third processed image, and determine the embedded coordinate frame corresponding to the external extension image, i.e., the external frame (such as Figure 5 ), determine the shadow image in the third processed image that intersects with the embedded coordinate frame, and set the shadow image in the third processed image that intersects with the embedded coordinate frame as the absolute shadow image of the object to be detected. That is, in this embodiment, it should be emphasized that the absolute shadow image intersects with the embedded coordinate frame.

[0125] In this embodiment, the outer extension rectangular frame of each object to be detected is determined according to the outer extension image of each object to be detected; the intersecting image in the initial shadow image that intersects with the outer extension rectangular frame is determined, and the intersecting image is removed to obtain a first processed image; a first preset shadow area threshold is obtained, and images with an area greater than the preset shadow area threshold are extracted from the first processed image to obtain the shadow area of ​​each object to be detected. In this embodiment, noise shadows are removed to accurately obtain the shadow area of ​​the object to be detected.

[0126] Reference Figure 3 , Figure 3 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present application.

[0127] like Figure 3As shown, the device for extracting the shadow area of ​​the object to be detected may include: a processor 1001, such as a CPU, a memory 1005, and a communication bus 1002. The communication bus 1002 is used to realize the connection and communication between the processor 1001 and the memory 1005. The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0128] Optionally, the device for extracting the shadow area of ​​the object to be detected may also include a rectangular user interface, a network interface, a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, etc. The rectangular user interface may include a display screen (Display), an input submodule such as a keyboard (Keyboard), and the optional rectangular user interface may also include a standard wired interface and a wireless interface. The network interface may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0129] Those skilled in the art will understand that Figure 3 The structure of the device for extracting the shadow area of ​​the object to be detected shown in the figure does not constitute a limitation on the device for extracting the shadow area of ​​the object to be detected, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0130] like Figure 3 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, and a shadow area extraction program for an object to be detected. The operating system is a program that manages and controls the hardware and software resources of the device for extracting the shadow area of ​​an object to be detected, and supports the operation of the shadow area extraction program for an object to be detected and other software and / or programs. The network communication module is used to realize the communication between the components inside the memory 1005, and the communication with other hardware and software in the shadow area extraction system for an object to be detected.

[0131] exist Figure 3 In the device for extracting the shadow area of ​​an object to be detected shown, the processor 1001 is used to execute the program for extracting the shadow area of ​​an object to be detected stored in the memory 1005 to implement the steps of any of the above-mentioned methods for extracting the shadow area of ​​an object to be detected.

[0132] The specific implementation of the device for extracting the shadow area of ​​the object to be detected in the present application is basically the same as the embodiments of the method for extracting the shadow area of ​​the object to be detected described above, and will not be repeated here.

[0133] The present application also provides a device for extracting a shadow area of ​​an object to be detected, wherein the device for extracting a shadow area of ​​an object to be detected is a virtual device and is applied to a first device or the device for extracting a shadow area of ​​an object to be detected is applied to a device for extracting a shadow area of ​​an object to be detected, and the device for extracting a shadow area of ​​an object to be detected comprises:

[0134] A receiving module, used for receiving a remote sensing image and obtaining an external image of an object to be detected in the remote sensing image;

[0135] A first acquisition module is used to perform preset semantic segmentation processing on the external image to obtain an initial shadow image of the target object to be detected;

[0136] The noise removal module is used to perform noise removal processing on the target initial shadow image to obtain the shadow area of ​​each object to be detected.

[0137] Optionally, the first acquisition module includes:

[0138] A speech segmentation unit, used for inputting the external extension image into a preset semantic segmentation network model to perform preset semantic segmentation processing on the external extension image to obtain a target initial shadow image of the object to be detected;

[0139] The preset semantic segmentation network model is a model of predicted shadow images obtained by training a preset basic model to be trained based on preset image data of an object to be detected with a preset shadow label.

[0140] Optionally, the device for extracting the shadow area of ​​the object to be detected includes:

[0141] A second acquisition module is used to acquire preset image data of the object to be detected, and iteratively train the preset basic model to be trained based on the preset image data of the object to be detected, so as to update the preset prediction model to be trained through iterative training;

[0142] The judgment module is used to judge whether the preset prediction model to be trained after iterative training and updating meets the preset training completion conditions. If the preset training completion conditions are met, the preset semantic segmentation network model is obtained.

[0143] Optionally, the second acquisition module includes:

[0144] A first acquisition unit is used to acquire preset image data of an object to be detected, and input the preset image data of the object to be detected into the preset basic model to be trained to obtain predicted probability image data, wherein the preset basic model to be trained includes a skip connection layer;

[0145] A second acquisition unit is used to acquire preset shadow label data of the preset image data of the object to be detected, and compare the preset shadow label data with the predicted probability image data to obtain difference data;

[0146] A training unit is used to iteratively train and update the preset prediction model to be trained according to the difference data.

[0147] Optionally, the receiving module includes:

[0148] A receiving unit, used to receive a remote sensing image, obtain a circumscribed rectangular frame image of an object to be detected in the remote sensing image, and obtain position information of the circumscribed rectangular frame image;

[0149] The extension unit is used to, based on the position information of the circumscribed rectangular frame image and with the circumscribed rectangular frame image as the center, crop from the remote sensing image to obtain an image including the circumscribed rectangular frame image and having a size that is a preset multiple of the circumscribed rectangular frame image, and set the image including the circumscribed rectangular frame image and having a size that is a preset multiple of the circumscribed rectangular frame image as the extension image.

[0150] Optionally, the noise removal module includes:

[0151] A first determining unit, configured to determine an outward extension rectangular frame of each object to be detected according to the outward extension image of each object to be detected;

[0152] A second determining unit is used to determine an intersecting image in the initial shadow image that intersects with the outer extension rectangular frame, and remove the intersecting image to obtain a first processed image;

[0153] The third acquisition unit is used to acquire a first preset shadow area threshold, extract images with an area larger than the preset shadow area threshold from the first processed image, and obtain the shadow area of ​​each object to be detected.

[0154] Optionally, the third acquiring unit includes:

[0155] A first acquisition subunit is used to acquire a first preset shadow area threshold, extract an image with an area larger than the first preset shadow area threshold from the first processed image, and obtain a second processed image;

[0156] The second acquisition subunit is used to obtain a second preset shadow area threshold, extract an image whose area is greater than the second preset shadow area threshold from the second processed image to obtain a third processed image, extract an image that intersects with the external frame of the external frame image from the third processed image, and obtain the shadow area of ​​each object to be detected.

[0157] Optionally, the first acquisition module includes:

[0158] A preprocessing unit, used for preprocessing the external extension image to obtain a preprocessed image;

[0159] The input unit is used to input the pre-processed image into a preset semantic segmentation network model to perform preset semantic segmentation processing on the external extension image to obtain the target initial shadow image of the object to be detected.

[0160] The specific implementation of the device for extracting the shadow area of ​​the object to be detected in the present application is basically the same as the embodiments of the method for extracting the shadow area of ​​the object to be detected described above, and will not be repeated here.

[0161] An embodiment of the present application provides a medium, and the medium stores one or more programs, and the one or more programs can also be executed by one or more processors to implement the steps of any of the above-mentioned methods for extracting the shadow area of ​​the object to be detected.

[0162] The specific implementation of the medium of the present application is basically the same as the above-mentioned embodiments of the method for extracting the shadow area of ​​the object to be detected, and will not be repeated here.

[0163] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A method for extracting the shadow area of ​​an object to be detected, characterized in that: The method for extracting the shadow area of ​​the object to be detected comprises: Receiving a remote sensing image and obtaining an external image of an object to be detected in the remote sensing image; Performing a preset semantic segmentation process on the external image to obtain an initial shadow image of the target object to be detected; Determine an outer extension rectangular frame of each object to be detected according to the outer extension image of each object to be detected; Determine an intersecting image in the initial shadow image that intersects with the outward-extended rectangular frame, and remove the intersecting image to obtain a first processed image; Obtaining a first preset shadow area threshold, extracting an image whose area is larger than the first preset shadow area threshold from the first processed image, and obtaining a second processed image; A second preset shadow area threshold is obtained, an image whose area is greater than the second preset shadow area threshold is extracted from the second processed image to obtain a third processed image, an image intersecting with the external frame of the external frame image is extracted from the third processed image, and a shadow area of ​​each object to be detected is obtained, wherein the external frame image is an image within an embedded coordinate frame corresponding to the external extension image.

2. The method for extracting the shadow area of ​​an object to be detected as claimed in claim 1, characterized in that: The step of performing preset semantic segmentation processing on the external extension image to obtain the target initial shadow image of the object to be detected includes: Inputting the external extension image into a preset semantic segmentation network model to perform preset semantic segmentation processing on the external extension image to obtain an initial shadow image of the target of the object to be detected; The preset semantic segmentation network model is a model of predicted shadow images obtained by training a preset basic model to be trained based on preset image data of an object to be detected with a preset shadow label.

3. The method for extracting the shadow area of ​​an object to be detected as claimed in claim 2, characterized in that: Before the step of inputting the externalized image into a preset semantic segmentation network model to obtain the target initial shadow image of each object to be detected, the method includes: Acquire preset image data of the object to be detected, and iteratively train the preset basic model to be trained based on the preset image data of the object to be detected, so as to update the preset basic model to be trained through iterative training; It is determined whether the preset basic model to be trained after iterative training and updating satisfies the preset training completion condition. If the preset training completion condition is met, the preset semantic segmentation network model is obtained.

4. The method for extracting the shadow area of ​​an object to be detected as claimed in claim 3, characterized in that: The step of obtaining preset image data of the object to be detected, iteratively training the preset basic model to be trained based on the preset image data of the object to be detected, and updating the preset basic model to be trained by iterative training includes: Acquire preset image data of an object to be detected, and input the preset image data of the object to be detected into the preset basic model to be trained to obtain predicted probability image data, wherein the preset basic model to be trained includes a skip connection layer; Acquire preset shadow label data of the preset image data of the object to be detected, and compare the preset shadow label data with the predicted probability image data to obtain difference data; The preset basic model to be trained is updated by iterative training according to the difference data.

5. The method for extracting the shadow area of ​​an object to be detected according to any one of claims 1 to 4, characterized in that: The step of receiving a remote sensing image and obtaining an external extension image of the object to be detected in the remote sensing image comprises: Receiving a remote sensing image, obtaining an image of a bounding rectangular frame of an object to be detected in the remote sensing image, and obtaining position information of the image of the bounding rectangular frame; According to the position information of the circumscribed rectangular frame image, an image including the circumscribed rectangular frame image and having a size that is a preset multiple of the circumscribed rectangular frame image is cropped from the remote sensing image with the circumscribed rectangular frame image as the center, and the image including the circumscribed rectangular frame image and having a size that is a preset multiple of the circumscribed rectangular frame image is set as the external extension image.

6. The method for extracting the shadow area of ​​an object to be detected as claimed in claim 2, characterized in that: The step of inputting the external extension image into a preset semantic segmentation network model to perform preset semantic segmentation processing on the external extension image to obtain the target initial shadow image of the object to be detected includes: Preprocessing the external extension image to obtain a preprocessed image; The pre-processed image is input into a preset semantic segmentation network model to perform preset semantic segmentation processing on the external extension image to obtain the target initial shadow image of the object to be detected.

7. A device for extracting the shadow area of ​​an object to be detected, characterized in that: The device for extracting the shadow area of ​​the object to be detected comprises: A receiving module, used for receiving a remote sensing image and obtaining an external image of an object to be detected in the remote sensing image; A first acquisition module is used to perform preset semantic segmentation processing on the external image to obtain an initial shadow image of the target object to be detected; A noise removal module, the noise removal module comprising: A first determining unit, configured to determine an outer extension rectangular frame of each object to be detected according to an outer extension image of each object to be detected; A second determining unit is used to determine an intersecting image in the initial shadow image that intersects with the outer extension rectangular frame, and remove the intersecting image to obtain a first processed image; A third acquisition unit, wherein the third acquisition unit comprises: A first acquisition subunit is used to acquire a first preset shadow area threshold, extract an image with an area larger than the first preset shadow area threshold from the first processed image, and obtain a second processed image; The second acquisition subunit is used to obtain a second preset shadow area threshold, extract an image whose area is greater than the second preset shadow area threshold from the second processed image to obtain a third processed image, extract an image that intersects with the external frame of the external frame image from the third processed image, and obtain the shadow area of ​​each object to be detected, wherein the external frame image is an image in the embedded coordinate frame corresponding to the external extension image.

8. A device for extracting the shadow area of ​​an object to be detected, characterized in that: The device for extracting the shadow area of ​​the object to be detected includes: a memory, a processor, and a program stored in the memory for implementing the method for extracting the shadow area of ​​the object to be detected. The memory is used to store a program for implementing a method for extracting the shadow area of ​​an object to be detected; The processor is used to execute a program for implementing the method for extracting the shadow area of ​​the object to be detected, so as to implement the steps of the method for extracting the shadow area of ​​the object to be detected as claimed in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program for implementing a method for extracting a shadow area of ​​an object to be detected, and the program for implementing a method for extracting a shadow area of ​​an object to be detected is executed by a processor to implement the steps of the method for extracting a shadow area of ​​an object to be detected as claimed in any one of claims 1 to 6.

Images