Image sky extraction method, system, device and storage medium

Abstract

The present invention provides a method, system, device and storage medium for extracting the sky from a picture. The method comprises the following steps: obtaining a picture to be replaced; constructing a picture sky extraction model based on a trained picture semantic segmentation model and a trained picture cutout model; training the picture sky extraction model to obtain a target extraction model; inputting the picture to be replaced into the picture semantic segmentation model to obtain a ternary graph; and inputting the picture to be replaced and the ternary graph into the target extraction model to extract a sky area from the picture to be replaced. The present application solves the problem that the extraction of sky areas is inaccurate and that the process requires manual intervention when extracting sky areas of batch images for subsequent transformation, thereby improving the accuracy of extracting sky areas in exterior images.

Description

Technical Field

[0001] The present invention relates to the technical field of picture extraction, and in particular to a picture sky extraction method, system, device and storage medium. Background Art

[0002] With the development of the OTA (Online Travel Agency) industry, people have an increasing demand for personalized outdoor images. As a relatively common category of outdoor images, the sky has a greater demand for quick, effective and automated extraction and subsequent transformation. Existing methods cannot accurately locate the entire sky area in the input image, and cannot distinguish fine sky areas without manual intervention. Summary of the invention

[0003] In view of the problems in the prior art, the object of the present invention is to provide a method, system, device and storage medium for extracting the sky from an image, so as to improve the accuracy of extracting the sky area in an exterior image.

[0004] To achieve the above object, the present invention provides a method for extracting the sky from a picture, the method comprising the following steps:

[0005] S110, obtaining the image to be replaced;

[0006] S120, constructing a picture sky extraction model based on the trained picture semantic segmentation model and the trained picture cutout model;

[0007] S130, training the image sky extraction model to obtain a target extraction model;

[0008] S140, inputting the image to be replaced into the image semantic segmentation model to obtain a ternary graph; and

[0009] S150, inputting the image to be replaced and the ternary map into the target extraction model, and extracting a sky area from the image to be replaced.

[0010] Optionally, before step S120, the method further includes:

[0011] A semantic segmentation dataset and a picture semantic segmentation model are respectively constructed, and a cutout dataset and a picture cutout model are constructed; the sky area segmentation granularity in the cutout dataset is smaller than the sky area segmentation granularity in the semantic segmentation dataset;

[0012] Using the semantic segmentation data set to train the image semantic segmentation model to obtain a trained image semantic segmentation model;

[0013] The image cutout model is trained using the cutout data set to obtain a trained image cutout model.

[0014] Optionally, step S130 includes:

[0015] The image sky extraction model is trained using the cutout data set to obtain a target extraction model.

[0016] Optionally, step S120 includes:

[0017] Based on the ViT model and BEiT model, a picture semantic segmentation model is constructed.

[0018] Optionally, step S120 includes:

[0019] Build an image cutout model based on the Unet network structure.

[0020] Optionally, during the training of the image semantic segmentation model, the loss function is:

[0021] L=L Dice +L BCE

[0022] in,

[0023]

[0024] H represents the length of the input image, and W represents the width of the input image. represents the probability that the i-th pixel in the image belongs to category m, represents the label value corresponding to the i-th pixel in the image belonging to category m, ε is the Laplace smoothing value, L Dice Represents the first loss function value, L BCE Represents the value of the second loss function.

[0025] Optionally, the method further comprises the steps of:

[0026] According to the pictures in the preset picture sky database, the sky area in the picture to be replaced is replaced to generate a target picture.

[0027] Optionally, replacing the sky area in the picture to be replaced based on the picture in the preset picture sky database includes:

[0028] The picture in the preset picture sky database with the smallest difference in color information between the sky area of ​​the picture to be replaced and the picture to be replaced is used as a reference picture;

[0029] The sky area in the picture to be replaced is replaced according to the reference picture to generate a target picture.

[0030] Optionally, the method further comprises the steps of:

[0031] The sky area in the to-be-replaced picture is replaced, and the color and texture of the area other than the sky in the to-be-replaced picture is corrected to generate a target picture.

[0032] Optionally, the method further comprises the steps of:

[0033] Based on the non-sky area in the picture to be replaced, determine the target scenic spot information corresponding to the picture to be replaced, and obtain a picture database associated with the target scenic spot information;

[0034] Screening and obtaining candidate sub-attractions and target weather categories corresponding to the candidate sub-attractions from the image database;

[0035] Based on the candidate sub-attractions and the target weather category, the recommended attraction pictures are screened again from the picture database and displayed on the display page.

[0036] Optionally, the screening from the image database to obtain candidate sub-attractions and target weather categories corresponding to the candidate sub-attractions includes:

[0037] Repeat steps S110 to S150 for the pictures in the picture database to obtain the sky area and the non-sky area corresponding to each picture in the picture database;

[0038] Determine the weather category corresponding to each picture according to the sky area of ​​each picture, and determine the corresponding sub-scenic spot information belonging to the target scenic spot information according to the non-sky area of ​​each picture;

[0039] Acquire the first popularity value corresponding to each sub-attraction information, and use the sub-attraction information with the highest first popularity value as a candidate sub-attraction;

[0040] Acquire the second heat value corresponding to each weather category corresponding to the picture containing the candidate sub-scenic spot in the picture database, and take the weather category with the highest second heat value as the target weather category;

[0041] The step of rescreening the image database to obtain recommended image of the scenic spot based on the candidate sub-attraction and the target weather category, and displaying the recommended image on the display page includes:

[0042] The pictures containing the sky area corresponding to the target weather category and the candidate sub-scenic spots in the picture database are selected as the scenic spot recommendation pictures and displayed on the display page.

[0043] The present invention also provides a picture sky extraction system, which is used to implement the picture sky extraction method, and the system comprises:

[0044] A module for obtaining pictures to be replaced, for obtaining pictures to be replaced;

[0045] The picture sky extraction model construction module builds a picture sky extraction model based on the trained picture semantic segmentation model and the trained picture cutout model;

[0046] A target extraction model acquisition module is used to train the image sky extraction model to obtain a target extraction model;

[0047] A ternary map acquisition module, inputting the image to be replaced into the image semantic segmentation model to obtain a ternary map; and

[0048] The sky area extraction module inputs the image to be replaced and the ternary map into the target extraction model, and extracts the sky area from the image to be replaced.

[0049] The present invention also provides a picture sky extraction device, comprising:

[0050] processor;

[0051] a memory in which an executable program of the processor is stored;

[0052] Wherein, the processor is configured to execute the steps of any one of the above-mentioned image sky extraction methods by executing the executable program.

[0053] The present invention also provides a computer-readable storage medium for storing a program, wherein when the program is executed by a processor, the program implements the steps of any one of the above-mentioned methods for extracting the sky from an image.

[0054] Compared with the prior art, the present invention has the following advantages and outstanding effects:

[0055] The picture sky extraction method, system, device and storage medium provided by the present invention aim to solve the problem of imprecise edges of sky areas extracted by using only a picture semantic segmentation model, and construct a target extraction model by combining a picture semantic segmentation model and a picture cutout model; a ternary graph is extracted by using the picture semantic segmentation model, and the ternary graph is used to represent the respective position information of the sky area, non-sky area and fuzzy area in the picture to be replaced, so as to guide the picture cutout model to extract the semantic fuzzy area in a refined manner from the perspective of graphic morphology, so that the model can extract the sky area of ​​the picture more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Other features, objectives and advantages of the present invention will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following accompanying drawings.

[0057] Figure 1A schematic diagram of a method for extracting the sky from a picture disclosed in an embodiment of the present invention;

[0058] Figure 2 A schematic diagram of a method for extracting the sky from a picture disclosed in another embodiment of the present invention;

[0059] Figure 3 A schematic diagram of a method for extracting the sky from a picture disclosed in another embodiment of the present invention;

[0060] Figure 4 A schematic diagram of the structure of a sky extraction system for an image disclosed in an embodiment of the present invention;

[0061] Figure 5 A schematic diagram of the structure of a sky extraction device for an image disclosed in an embodiment of the present invention;

[0062] Figure 6 A schematic diagram of the structure of a computer-readable storage medium disclosed in one embodiment of the present invention. DETAILED DESCRIPTION

[0063] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention will be comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar structures, and thus their repeated description will be omitted.

[0064] like Figure 1 As shown, an embodiment of the present invention discloses a method for extracting the sky from a picture, which solves the problem that when extracting the sky area of ​​a batch of images for subsequent transformation, the extraction of the sky area is inaccurate, the process requires manual intervention, and is too complicated and cumbersome, so that when the user subsequently replaces the sky area of ​​the picture, the display effect of the replaced picture is better, which is conducive to improving the user experience. The method includes the following steps:

[0065] S110, obtaining the image to be replaced.

[0066] S120, constructing a picture sky extraction model based on the trained picture semantic segmentation model and the trained picture cutout model. Specifically, the trained picture semantic segmentation model and the trained picture cutout model are combined to form an overall model, namely, the picture sky extraction model.

[0067] In this implementation, the ViT model based on self-supervised training of open source data and public data is used as the backbone network and the upsampling head of UperNet is used as the decoder to form the BEiT model, and then the above ViT model and the above BEiT model are combined to form a picture semantic segmentation model. Compared with the prior art that generally uses the FCN model for semantic segmentation, the above model used in this application is conducive to improving the extraction accuracy of the sky area and improving the accuracy of picture pixel extraction, so that the model can extract the sky area more accurately.

[0068] The image semantic segmentation model of the present application uses an adaptive moment estimation optimizer to update the model weights, and the learning rate update strategy uses a preheated cosine annealing strategy, which is conducive to improving the accuracy of the training results.

[0069] In this implementation, a picture matting model is constructed based on the Unet network structure. The encoding network uses Resnet50. Compared with the FBA_Matting model commonly used in the prior art for matting, the above model used in this application is conducive to improving the extraction accuracy of the sky area and improving the accuracy of picture pixel extraction, so that the model can extract the sky area more accurately.

[0070] In this implementation, the sky area segmentation granularity of the cutout dataset used to train the image cutout model is smaller than the sky area segmentation granularity of the semantic segmentation dataset used to train the image semantic segmentation model. That is, the sky area segmentation in the semantic segmentation dataset has edge fuzzy areas, while the sky area segmentation in the cutout dataset does not have edge fuzzy areas and is more refined. In this way, the image semantic segmentation model can be used to obtain a rough mask (mask) of the sky, which has a semantically fuzzy part, and then the image cutout model is used to perform refined recognition of the semantically fuzzy part, and the overall model outputs a refined mask. This is conducive to more refined extraction of the sky area.

[0071] Among them, the sky area and non-sky area are marked in the above semantic segmentation dataset. There is no blurred area in the pictures in this dataset, only the above two areas. The above cutout dataset is generated by image synthesis, that is, the non-sky area of ​​the image is cut out, and then randomly fused with the pure sky image. This can retain the edge information of the image details in the cutout dataset, which is conducive to improving the training accuracy of the model, improving the extraction accuracy of the sky area, and improving the accuracy of image pixel extraction. Adding multi-head attention mechanism and feature pyramid pooling to the above picture semantic segmentation model is conducive to making the model extract the sky area more accurately.

[0072] S130, training the above-mentioned picture sky extraction model to obtain a target extraction model. Specifically, the picture sky extraction model is trained using the above-mentioned cutout dataset to obtain a target extraction model. That is, the present application uses a cutout dataset with a finer granularity of sky area segmentation to train the picture sky extraction model, which is conducive to improving the extraction accuracy of the sky area and improving the accuracy of picture pixel extraction, so that the model can extract the sky area more accurately.

[0073] S140, input the above-mentioned picture to be replaced into the above-mentioned picture semantic segmentation model to obtain a ternary map. The above-mentioned ternary map is used to characterize the position information of the sky area, the non-sky area and the blurred area in the picture to be replaced. The ternary map is used to guide the picture cutout model, select the area that needs to be refined to help the picture cutout model to perform precise recognition, so that the model can extract the sky area more accurately.

[0074] The above ternary image does not contain information such as color texture, semantic information, object distribution, etc. in the input image to be replaced.

[0075] And S150, inputting the to-be-replaced picture and the ternary map into the target extraction model, and extracting the sky area from the to-be-replaced picture.

[0076] The present invention can automatically optimize the output results without adding new links. Since the labels of semantic segmentation output are in the form of probability output, the output of the image semantic segmentation model and the output of the target extraction model are consistent in data form, so the output of the target extraction model can be used to generate the ternary graph required by the image cutout model. Repeating the cycle multiple times can make the final output of the model more accurate.

[0077] In another embodiment of the present application, Figure 2 As shown, another method for extracting the sky from an image is disclosed. Figure 1 On the basis of the corresponding embodiment, between steps S110 and S120, the following is further included:

[0078] S111, constructing a semantic segmentation dataset and a picture semantic segmentation model, and constructing a cutout dataset and a picture cutout model. The sky region segmentation granularity in the cutout dataset is smaller than the sky region segmentation granularity in the semantic segmentation dataset.

[0079] S112, using the semantic segmentation dataset to train the image semantic segmentation model to obtain a trained image semantic segmentation model.

[0080] S113, using the above-mentioned cutout data set to train the above-mentioned image cutout model to obtain a trained image cutout model.

[0081] In this embodiment, a combination of Dice Loss and cross entropy loss functions is used as the loss function of the image semantic segmentation model. In this way, on the basis of training to achieve the classification purpose, some image data with unbalanced categories can be processed more accurately. For example, the proportions of the sky area and the non-sky area in some images are very inconsistent. For example, the sky area may account for 0.9 and the non-sky area may account for 0.1. The present application can accurately train these images, which is conducive to improving the accuracy of the training results of the image semantic segmentation model.

[0082] Specifically, during the training of the above image semantic segmentation model, the loss function is:

[0083] L=L Dice +L BCE

[0084] in,

[0085]

[0086] H represents the length of the input image, and W represents the width of the input image. represents the probability that the i-th pixel in the image belongs to category m, represents the label value corresponding to the i-th pixel in the image belonging to category m, ε is the Laplace smoothing value added to prevent the divisor from being zero, and L Dice Represents the first loss function value, L BCE represents the value of the second loss function. Category m can be a sky area, a non-sky area, or a fuzzy area.

[0087] In another embodiment of the present application, another method for extracting the sky from an image is disclosed. Figure 1 On the basis of the corresponding embodiment, the steps are also included:

[0088] S160, replacing the sky area in the above-mentioned picture to be replaced according to the pictures in the preset picture sky database to generate a target picture. The pictures in the preset picture sky database can be selected by the user or automatically generated. The preset picture sky database contains pure sky pictures.

[0089] In another embodiment of the present application, step S160 includes:

[0090] S161, taking a picture in a preset picture sky database with the smallest color information difference from the sky area of ​​the picture to be replaced as a reference picture. The color information may be the RGB value of the sky area of ​​the picture.

[0091] S162, replacing the sky area in the picture to be replaced according to the reference picture to generate a target picture.

[0092] In another embodiment of the present application, step S160 includes:

[0093] The sky area in the above-mentioned picture to be replaced is replaced, and the color texture of the area other than the sky in the above-mentioned picture to be replaced is corrected to generate a target picture. This can make the display effect of the replaced picture better and more uniform, and improve the user experience.

[0094] It should be noted that all the above embodiments disclosed in this application can be freely combined, and the technical solutions obtained after the combination are also within the protection scope of this application.

[0095] In another embodiment of the present application, Figure 3 As shown, another method for extracting the sky from an image is disclosed. Figure 1 On the basis of the corresponding embodiment, the steps are also included:

[0096] S170, determining target scenic spot information corresponding to the picture to be replaced based on the non-sky area in the picture to be replaced, and acquiring a picture database associated with the target scenic spot information.

[0097] S180: Screening and obtaining candidate sub-attractions and target weather categories corresponding to the candidate sub-attractions from the image database.

[0098] S190, based on the candidate sub-attractions and the target weather category, the recommended attraction pictures are screened again from the picture database and displayed on the display page.

[0099] Specifically, in this embodiment, steps S110 to S150 are repeatedly performed on the pictures in the picture database to obtain the sky area and non-sky area corresponding to each picture in the picture database. The weather category corresponding to each picture is determined based on the sky area of ​​each picture, and the corresponding sub-scenic spot information belonging to the target scenic spot information is determined based on the non-sky area of ​​each picture. The first heat value corresponding to each sub-scenic spot information is obtained, and the sub-scenic spot information with the highest first heat value is used as an alternative sub-scenic spot. The second heat value corresponding to each weather category corresponding to the picture containing the alternative sub-scenic spot in the picture database is obtained, and the weather category with the highest second heat value is used as the target weather category. The pictures in the picture database that contain the sky area corresponding to the target weather category and the alternative sub-scenic spot are selected as recommended scenic spot pictures and displayed on the display page.

[0100] Exemplarily, for example, the input picture to be replaced is a scenic spot with a bamboo sea. The picture to be replaced can be input by a travel platform. In this case, it means that the travel platform needs to promote the scenic spot. Then the non-sky area in the picture to be replaced contains the bamboo sea, and then the application can match and obtain the corresponding target scenic spot information based on picture recognition or based on the geographical location information of the picture. Then a large number of pictures of the target scenic spot can be obtained from the network platform, and the corresponding sub-scenic spots can be identified from the non-sky area matching of these large number of pictures. The heat of each sub-scenic spot is sorted according to the heat value of the picture, and the sub-scenic spot with the highest heat is used as the alternative sub-scenic spot. The above heat value can be calculated based on information such as the number of likes, views and / or reposts of the network platform.

[0101] After sorting the sub-attractions, it is found that the bamboo sea is the most popular. Then, the most popular weather for viewing the bamboo sea is determined among the pictures containing the bamboo sea, such as rainy days. This can also be determined by sorting the heat values. Then, the pictures with the weather category of rainy days and containing the bamboo sea view are used as recommended display pictures, which is conducive to increasing the popularity of the display pictures, that is, to improving the conversion rate of tourism promotion products.

[0102] like Figure 4 As shown, an embodiment of the present invention further discloses a picture sky extraction system 4, which includes:

[0103] The image to be replaced acquisition module 41 acquires the image to be replaced.

[0104] The picture sky extraction model construction module 42 constructs a picture sky extraction model based on the trained picture semantic segmentation model and the trained picture cutout model.

[0105] The target extraction model acquisition module 43 trains the picture sky extraction model to obtain a target extraction model.

[0106] The ternary graph acquisition module 44 inputs the image to be replaced into the image semantic segmentation model to obtain a ternary graph.

[0107] The sky area extraction module 45 inputs the image to be replaced and the ternary map into the target extraction model, and extracts the sky area from the image to be replaced.

[0108] It is understandable that the picture sky extraction system of the present invention also includes other existing functional modules that support the operation of the picture sky extraction system. Figure 4 The picture sky extraction system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0109] The picture sky extraction system in this embodiment is used to implement the above-mentioned picture sky extraction method. Therefore, the specific implementation steps of the picture sky extraction system can refer to the above-mentioned description of the picture sky extraction method, which will not be repeated here.

[0110] An embodiment of the present invention further discloses a picture sky extraction device, comprising a processor and a memory, wherein the memory stores an executable program of the processor; the processor is configured to execute the steps in the above-mentioned picture sky extraction method by executing the executable program. Figure 5 This is a schematic diagram of the structure of the image sky extraction device disclosed in the present invention. Figure 5 The electronic device 600 according to this embodiment of the present invention is described. Figure 5 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0111] like Figure 5 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0112] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps of various exemplary embodiments of the present invention described in the above-mentioned image sky extraction method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0113] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0114] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0115] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0116] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0117] The present invention also discloses a computer-readable storage medium for storing a program, which implements the steps in the above-mentioned picture sky extraction method when executed. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary implementations of the present invention described in the above-mentioned picture sky extraction method of this specification.

[0118] As shown above, when the program of the computer-readable storage medium of this embodiment is executed, in order to solve the problem of unclear edges of the sky area extracted by using only the picture semantic segmentation model, a target extraction model is constructed by combining the picture semantic segmentation model and the picture cutout model; a ternary graph is extracted using the picture semantic segmentation model, and the ternary graph is used to represent the respective position information of the sky area, non-sky area and blurred area in the picture to be replaced, so as to guide the picture cutout model to refine the extraction of the semantically blurred area from the perspective of graphic morphology, so that the model can extract the sky area of ​​the picture more accurately.

[0119] Figure 6 Schematic diagram of the structure of the computer-readable storage medium of the present invention. Figure 6 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0120] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0121] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0122] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0123] The picture sky extraction method, system, device and storage medium provided by the embodiments of the present invention address the problem of imprecise edges of sky areas extracted using only a picture semantic segmentation model, and construct a target extraction model by combining a picture semantic segmentation model and a picture cutout model; a ternary graph is extracted using the picture semantic segmentation model, and the ternary graph is used to represent the position information of the sky area, non-sky area and fuzzy area in the picture to be replaced, thereby guiding the picture cutout model to extract the semantically fuzzy area in a refined manner from a graphic morphology perspective, so that the model can extract the sky area of ​​the picture more accurately.

[0124] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A method for extracting sky from a picture, characterized in that: The following steps are involved: S110, obtaining the image to be replaced; S120, constructing a picture semantic segmentation model based on the ViT model and the BEiT model, constructing a picture cutout model based on the Unet network structure, constructing a picture sky extraction model based on the trained picture semantic segmentation model and the trained picture cutout model, and using a combination of Dice Loss and cross entropy loss functions as the loss function of the picture semantic segmentation model. During the training process of the picture semantic segmentation model, the loss function is: L=L Dice +L BCE in, H represents the length of the input image, and W represents the width of the input image. represents the probability that the i-th pixel in the image belongs to category m, represents the label value corresponding to the i-th pixel in the image belonging to category m, ε is the Laplace smoothing value added to prevent the divisor from being zero, and L Dice Represents the first loss function value, L BCE represents the value of the second loss function, and the category m can be a sky area, a non-sky area, or a fuzzy area; S130, training the image sky extraction model to obtain a target extraction model; S140, inputting the image to be replaced into the image semantic segmentation model to obtain a ternary graph; and S150, inputting the image to be replaced and the ternary map into the target extraction model, and extracting a sky area from the image to be replaced.

2. The method for extracting sky from a picture according to claim 1, wherein: Before step S120, the method further includes: A semantic segmentation dataset and a picture semantic segmentation model are respectively constructed, and a cutout dataset and a picture cutout model are constructed; the sky area segmentation granularity in the cutout dataset is smaller than the sky area segmentation granularity in the semantic segmentation dataset; Using the semantic segmentation data set to train the image semantic segmentation model to obtain a trained image semantic segmentation model; The image cutout model is trained using the cutout data set to obtain a trained image cutout model.

3. The method for extracting sky from a picture according to claim 2, wherein: Step S130 includes: The image sky extraction model is trained using the cutout data set to obtain a target extraction model.

4. The method for extracting sky from a picture according to claim 1, wherein: The method further comprises the steps of: According to the pictures in the preset picture sky database, the sky area in the picture to be replaced is replaced to generate a target picture.

5. The method for extracting sky from a picture according to claim 4, characterized in that: The replacing the sky area in the picture to be replaced according to the picture in the preset picture sky database includes: The picture in the preset picture sky database with the smallest difference in color information between the sky area of ​​the picture to be replaced and the picture to be replaced is used as a reference picture; The sky area in the picture to be replaced is replaced according to the reference picture to generate a target picture.

6. The method for extracting sky from a picture according to claim 4, characterized in that: The method further comprises the steps of: The sky area in the to-be-replaced picture is replaced, and the color and texture of the area other than the sky in the to-be-replaced picture is corrected to generate a target picture.

7. The method for extracting sky from a picture according to claim 1, characterized in that: The method further comprises the steps of: Based on the non-sky area in the picture to be replaced, determine the target scenic spot information corresponding to the picture to be replaced, and obtain a picture database associated with the target scenic spot information; Screening and obtaining candidate sub-attractions and target weather categories corresponding to the candidate sub-attractions from the image database; Based on the candidate sub-attractions and the target weather category, the recommended attraction pictures are screened again from the picture database and displayed on the display page.

8. The method for extracting sky from a picture according to claim 7, wherein: The selecting and obtaining the candidate sub-attractions and the target weather categories corresponding to the candidate sub-attractions from the image database includes: Repeat steps S110 to S150 for the pictures in the picture database to obtain the sky area and the non-sky area corresponding to each picture in the picture database; Determine the weather category corresponding to each picture according to the sky area of ​​each picture, and determine the corresponding sub-scenic spot information belonging to the target scenic spot information according to the non-sky area of ​​each picture; Acquire the first popularity value corresponding to each sub-attraction information, and use the sub-attraction information with the highest first popularity value as a candidate sub-attraction; Acquire the second heat value corresponding to each weather category corresponding to the picture containing the candidate sub-scenic spot in the picture database, and take the weather category with the highest second heat value as the target weather category; The step of rescreening the image database to obtain recommended image of the scenic spot based on the candidate sub-attraction and the target weather category, and displaying the recommended image on the display page includes: The pictures containing the sky area corresponding to the target weather category and the candidate sub-scenic spots in the picture database are selected as the scenic spot recommendation pictures and displayed on the display page.

9. A picture sky extraction system, used to implement the picture sky extraction method according to claim 1, characterized in that: The system comprises: A module for obtaining pictures to be replaced, for obtaining pictures to be replaced; The picture sky extraction model construction module builds a picture semantic segmentation model based on the ViT model and the BEiT model, builds a picture cutout model based on the Unet network structure, and builds a picture sky extraction model based on the trained picture semantic segmentation model and the trained picture cutout model. The Dice Loss and cross entropy loss function combination is used as the loss function of the picture semantic segmentation model. During the training process of the picture semantic segmentation model, the loss function is: L=L Dice +L BCE in, H represents the length of the input image, and W represents the width of the input image. represents the probability that the i-th pixel in the image belongs to category m, represents the label value corresponding to the i-th pixel in the image belonging to category m, ε is the Laplace smoothing value added to prevent the divisor from being zero, and L Dice Represents the first loss function value, L BCE represents the value of the second loss function, and the category m can be a sky area, a non-sky area, or a fuzzy area; A target extraction model acquisition module is used to train the image sky extraction model to obtain a target extraction model; A ternary map acquisition module, inputting the image to be replaced into the image semantic segmentation model to obtain a ternary map; and The sky area extraction module inputs the image to be replaced and the ternary map into the target extraction model, and extracts the sky area from the image to be replaced.

10. A sky extraction device for a picture, characterized in that: include: processor; a memory in which an executable program of the processor is stored; Wherein, the processor is configured to execute the steps of the image sky extraction method described in any one of claims 1 to 8 by executing the executable program.

11. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the image sky extraction method described in any one of claims 1 to 8 are implemented.

Images