Image generation method, image generation device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image generation method, an image generation device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring an original image and an image matting instruction used for instructing to perform image matting processing on the original image, performing image matting processing on the original image according to an automatic image matting instruction or an image matting operation to obtain an image matting sub-image, performing format conversion on the image matting sub-image to obtain an original integer data stream, and sending the original integer data stream to a server; carrying out super-resolution processing on the original integer data stream through a preset super-resolution network to obtain a target integer data stream, carrying out format conversion on the target integer data stream to obtain a reference sub-image, obtaining a layer of the reference sub-image, obtaining a reference layer, obtaining a reference image, and obtaining a reference image according to the reference layer; and carrying out image combination on the reference image and the reference sub-image to obtain the target image, so that the image generation quality can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, is applicable to the field of financial technology, and particularly relates to an image generation method, an image generation device, an electronic device and a storage medium. Background Art

[0002] In the image generation process, it is necessary to obtain the original image from the original material library and use the original image to generate a new image. For example, in the financial technology scenario, the original image is used to generate a promotional poster image so that insurance agents can use the promotional poster image to perform insurance business tasks. However, after the original image is cropped and enlarged, the image clarity will be reduced, resulting in a decrease in the quality of the image generation. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose an image generation method, an image generation device, an electronic device and a storage medium, aiming to improve the quality of image generation.

[0004] To achieve the above object, a first aspect of an embodiment of the present application provides an image generation method, the method comprising:

[0005] Acquire an original image and a cutout instruction for instructing to perform a cutout process on the original image; the cutout instruction includes an automatic cutout instruction or a manual cutout instruction;

[0006] Performing a cutout process on the original image according to the automatic cutout instruction or the cutout operation to obtain a cutout sub-image; wherein the cutout operation is an operation obtained based on the manual cutout instruction;

[0007] Performing format conversion on the cutout sub-image to obtain an original integer data stream;

[0008] Performing super-resolution processing on the original integer data stream through a preset super-resolution network to obtain a target integer data stream;

[0009] Performing format conversion on the target integer data stream to obtain a reference sub-image; the resolution of the reference sub-image is greater than the resolution of the cutout sub-image;

[0010] Acquire the layer of the reference sub-image to obtain a reference layer;

[0011] A reference image is acquired, and the reference image and the reference sub-image are merged according to the reference layer to obtain a target image.

[0012] In some embodiments, performing a cutout process on the original image according to the automatic cutout instruction or the cutout operation to obtain a cutout sub-image includes:

[0013] Performing image segmentation on the original image according to the automatic cutout instruction to obtain an image triplet; the image triplet includes a foreground sub-image, a background sub-image and a boundary sub-image;

[0014] Performing a first mask prediction according to the original image, the foreground sub-image, the background sub-image and the boundary sub-image to obtain a first predicted mask;

[0015] Performing mask prediction according to the original image and the first prediction mask to obtain a second prediction mask;

[0016] performing mask adjustment on the second prediction mask to obtain a target prediction mask;

[0017] The original image is subjected to a matting process according to the target prediction mask to obtain the matting sub-image.

[0018] In some embodiments, performing mask adjustment on the second prediction mask to obtain a target prediction mask includes:

[0019] Performing mask prediction according to the original image and the foreground sub-image to obtain a third predicted mask;

[0020] determining a fourth prediction mask according to the third prediction mask, the boundary sub-image and the foreground sub-image;

[0021] calculating first similarity data between the second prediction mask and the fourth prediction mask;

[0022] The second prediction mask is subjected to mask adjustment according to the first similarity data to obtain the target prediction mask.

[0023] In some embodiments, performing mask adjustment on the second prediction mask according to the first similarity data to obtain the target prediction mask includes:

[0024] If the first similarity data is less than a preset similarity threshold, performing a second mask prediction according to the original image, the foreground sub-image, the background sub-image and the boundary sub-image to obtain a fifth predicted mask;

[0025] calculating second similarity data between the second prediction mask and the fifth prediction mask;

[0026] The second prediction mask is subjected to mask adjustment according to the second similarity data to obtain the target prediction mask.

[0027] In some embodiments, performing a cutout process on the original image according to the automatic cutout instruction or the cutout operation to obtain a cutout sub-image includes:

[0028] Acquiring mask coordinates according to the cutout operation;

[0029] Filtering a foreground area from the original image according to the mask coordinates;

[0030] Extracting color features from the foreground area to obtain a color feature vector;

[0031] Performing Gaussian feature extraction on the foreground area to obtain a Gaussian feature vector;

[0032] Calculating target loss data according to the color feature vector, the Gaussian feature vector and the foreground area;

[0033] Adjusting the foreground area according to the target loss data to obtain a target operation mask;

[0034] The original image is subjected to a cutout process according to the target operation mask to obtain the cutout sub-image.

[0035] In some embodiments, calculating the target loss data according to the color feature vector, the Gaussian feature vector and the foreground area includes:

[0036] Performing Gaussian calculation on the color feature vector and the Gaussian feature vector through a preset mixed Gaussian model to obtain first sub-loss data;

[0037] Performing smoothing calculation on the foreground area through a preset smoothing model to obtain second sub-loss data;

[0038] The first sub-loss data and the second sub-loss data are summed to obtain the target loss data.

[0039] In some embodiments, the super-resolution network includes a first convolutional layer, a dense residual layer, a second convolutional layer, and a third convolutional layer, and the super-resolution processing of the original integer data stream by the preset super-resolution network to obtain the target integer data stream includes:

[0040] Performing convolution processing on the original integer data stream through the first convolution layer to obtain a first convolution data stream;

[0041] Performing residual calculation on the first convolution data stream through the dense residual layer to obtain a residual data stream;

[0042] Performing convolution processing on the residual data stream through the second convolution layer to obtain a second convolution data stream;

[0043] Fusing the first convolution data stream and the second convolution data stream to obtain a fused data stream;

[0044] Downsampling the fused data stream to obtain a downsampled data stream;

[0045] The down-sampled data stream is convolved by the third convolutional layer to obtain the target integer data stream.

[0046] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides an image generating device, the device comprising:

[0047] An instruction acquisition module, used for acquiring an original image and a cutout instruction for instructing to perform a cutout process on the original image; the cutout instruction includes an automatic cutout instruction or a manual cutout instruction;

[0048] A cutout module, used for performing a cutout process on the original image according to the automatic cutout instruction or the cutout operation to obtain a cutout sub-image; wherein the cutout operation is an operation obtained based on the manual cutout instruction;

[0049] An image format conversion module, used for performing format conversion on the cutout sub-image to obtain an original integer data stream;

[0050] A super-resolution module, used for performing super-resolution processing on the original integer data stream through a preset super-resolution network to obtain a target integer data stream;

[0051] A data stream format conversion module, used for performing format conversion on the target integer data stream to obtain a reference sub-image; the resolution of the reference sub-image is greater than the resolution of the cutout sub-image;

[0052] A layer acquisition module, used to acquire the layer of the reference sub-image to obtain a reference layer;

[0053] The image generation module is used to obtain a reference image, and merge the reference image and the reference sub-image according to the reference layer to obtain a target image.

[0054] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0055] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0056] The image generation method, image generation device, electronic device and computer-readable storage medium proposed in the embodiment of the present application obtain an original image and a cutout instruction for indicating the cutout processing of the original image. In order to achieve different cutout requirements, the cutout instruction includes an automatic cutout instruction or a manual cutout instruction. The original image is cutout according to the automatic cutout instruction or the cutout operation to extract the foreground image element of interest from the original image, separate the foreground image element from the background image element, and obtain the cutout sub-image. The cutout sub-image is an enlarged image cut out from the original image, which will have magnification distortion, so that the image clarity of the cutout sub-image is low. In order to improve the quality of image generation, it is necessary to improve the image clarity of the cutout sub-image. Due to the various storage formats of the original image, the cutout sub-image has a variety of data formats, so it is necessary to convert the format of the cutout sub-image, convert the cutout sub-image into a unified data stream, and obtain the original integer data stream. And converting the image into an integer data stream can save storage space, reduce the difficulty of data calculation, and thus improve the efficiency of data processing. In order to improve the image clarity of the cutout sub-image, the original integer data stream is super-resolved through a preset super-resolution network to obtain a target integer data stream, and the target integer data stream is format-converted to obtain clear cutout image elements and obtain a reference sub-image. In order to facilitate image generation using the reference sub-image, the layer of the reference sub-image is obtained to obtain a reference layer. The reference image is obtained, and the reference image and the reference sub-image are merged according to the reference layer, so that the reference sub-image can be distributed in a suitable layer, a new image is generated, and the target image is obtained, thereby improving the quality of image generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a flow chart of an image generation method provided in an embodiment of the present application;

[0058] Figure 2 yes Figure 1 Flow chart of step S120 in FIG.

[0059] Figure 3 yes Figure 2 Flow chart of step S240 in FIG.

[0060] Figure 4 yes Figure 3 Flow chart of step S340 in FIG.

[0061] Figure 5 yes Figure 1 Flow chart of step S120 in FIG.

[0062] Figure 6 yes Figure 5 Flow chart of step S550 in FIG.

[0063] Figure 7 yes Figure 1 Flow chart of step S140 in FIG.

[0064] Figure 8 is a schematic diagram of the structure of an image generating device provided in an embodiment of the present application;

[0065] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0067] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0069] In the image generation process, it is necessary to obtain the original image from the original material library and use the original image to generate a new image. For example, the original image is used to generate a promotional poster image so that insurance agents can use the promotional poster image to perform insurance business tasks. However, after the original image is cropped and enlarged, the image clarity will be reduced, resulting in a decrease in the quality of the image generation.

[0070] Based on this, the embodiments of the present application provide an image generation method, an image generation device, an electronic device and a computer-readable storage medium, aiming to improve the quality of image generation.

[0071] The image generation method, image generation device, electronic device and computer-readable storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the image generation method in the embodiments of the present application is described.

[0072] The image generation method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The image generation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the image generation method, etc., but is not limited to the above forms.

[0073] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0074] Figure 1 It is an optional flowchart of the image generation method provided in an embodiment of the present application. The image generation method may include but is not limited to steps S110 to S170.

[0075] Step S110, acquiring an original image and a cutout instruction for instructing to cutout the original image; the cutout instruction includes an automatic cutout instruction or a manual cutout instruction;

[0076] Step S120, performing a cutout process on the original image according to the automatic cutout instruction or the cutout operation to obtain a cutout sub-image; wherein the cutout operation is an operation obtained based on the manual cutout instruction;

[0077] Step S130, converting the format of the cutout sub-image to obtain an original integer data stream;

[0078] Step S140, super-resolution processing is performed on the original integer data stream through a preset super-resolution network to obtain a target integer data stream;

[0079] Step S150, converting the target integer data stream into a format to obtain a reference sub-image; the resolution of the reference sub-image is greater than the resolution of the cutout sub-image;

[0080] Step S160, obtaining a layer of a reference sub-image to obtain a reference layer;

[0081] Step S170, obtaining a reference image, and merging the reference image and the reference sub-image according to the reference layer to obtain a target image.

[0082] In the steps S110 to S170 shown in the embodiment of the present application, a cutout sub-image is obtained by cutting out the foreground image elements of interest from the original image. In order to prevent the magnification distortion caused by the cutout sub-image from affecting the quality of image generation, the cutout sub-image is super-resolved to obtain a reference sub-image to improve the image clarity of the cutout sub-image. The reference sub-image and the reference image are merged according to the layer to accurately control the distribution of the reference sub-image on the reference sub-image, thereby improving the quality of image generation.

[0083] In step S110 of some embodiments, a data access operation on a terminal device is triggered by a user to obtain a data access mode, which includes a local access mode or an online access mode. If the data access operation is to access the local material library of the terminal device, the data access mode is determined to be a local access mode; if the data access operation is to access the online material library, the data access mode is determined to be an online access mode. According to the data access mode, images of specific business scenarios such as insurance product promotions are obtained to obtain original images. If the data access mode is a local access mode, the terminal device calls up the local system album and obtains the original image from the local system album; if the data access mode is an online access mode, an image link is obtained from the data access mode, and the original image is obtained from the online material library according to the image link.

[0084] There are many materials in the online material library, and frequent downloading will affect the access speed of the materials. In order to improve the access speed of online materials, a three-level cache can be set for the online material library. The three-level cache includes the first cache, the second cache and the third cache. The storage space of the first cache, the second cache and the third cache increases in sequence, and the access speed decreases in sequence. When obtaining the original image from the online material library, the first cache is queried according to the image link. If the original image corresponding to the image link is stored in the first cache, the original image is obtained from the first cache. If the original image corresponding to the image link is not stored in the first cache, the second cache is queried according to the image link. If the original image corresponding to the image link is stored in the second cache, the original image is obtained from the second cache. If the original image corresponding to the image link is not stored in the second cache, the original image is obtained from the second cache. If the second cache does not store the original image corresponding to the image link, the original image is obtained from the third cache.

[0085] After acquiring the original image, the terminal device may pop up a cutout option, which includes an automatic cutout option and a manual cutout option. The automatic cutout option and the manual cutout option are selected to obtain the target cutout option. If the target cutout option is the automatic cutout option, the cutout indication is determined to be an automatic cutout indication; if the target cutout option is the manual cutout option, the cutout indication is determined to be a manual cutout indication. The automatic cutout indication is used to indicate that the original image is cutout by using the automated cutout technology, and the manual cutout indication is used to indicate that the original image is cutout by using the user's manual operation. The user's manual operation can be obtained according to the manual cutout indication to obtain the cutout operation. The cutout operation can be a click operation, a drag operation, a brush tool movement operation, etc. The cutout indication can be used to cut out the required image content from the original image to generate a new image based on the image content.

[0086] See also Figure 2 In some embodiments, step S120 may include but is not limited to steps S210 to S250:

[0087] Step S210, segmenting the original image according to the automatic cutout instruction to obtain an image triplet; the image triplet includes a foreground sub-image, a background sub-image and a boundary sub-image;

[0088] Step S220, performing a first mask prediction based on the original image, the foreground sub-image, the background sub-image, and the boundary sub-image to obtain a first predicted mask;

[0089] Step S230, performing mask prediction according to the original image and the first prediction mask to obtain a second prediction mask;

[0090] Step S240, performing mask adjustment on the second prediction mask to obtain a target prediction mask;

[0091] Step S250, performing a matting process on the original image according to the target prediction mask to obtain a matting sub-image.

[0092] In step S210 of some embodiments, if the cutout indication is an automatic cutout indication, the semantic category information in the original image is learned through the semantic segmentation model, and the original image is semantically segmented to obtain an image mask. The image mask is a three-channel image, and the image mask contains information about the foreground area, the background area, and the uncertain area. A softmax operation is performed on the image mask to segment the three-channel image into three single-channel images to obtain an image triplet. The image triplet includes three images, namely a foreground sub-image, a background sub-image, and a boundary sub-image. The foreground sub-image is a foreground image, which refers to the area in the original image that is determined to belong to the foreground object. The background sub-image is a background image, which refers to the area in the original image that is determined to belong to the background object. The boundary sub-image is an uncertain boundary image, which refers to the area in the original image that neither belongs to a determined foreground object nor a determined background object.

[0093] The semantic segmentation model can be a transformation model (Transformation Network, T-Net), including convolutional layers, pooling layers, and fully connected layers. The convolutional layer has a batch normalization function and an activation function, which is used to extract the image features of the original image. The pooling layer is used to reduce the feature space dimension of the image features and maintain the global information of the image features. The fully connected layer is used to map the pooled image features to a higher-level feature space to obtain the semantic category of each pixel in the original image. According to the semantic category, the pixel can be distinguished as a foreground pixel, a background pixel, or an uncertain pixel.

[0094] In step S220 of some embodiments, the original image, the foreground sub-image, the background sub-image, and the boundary sub-image are stitched to obtain a first stitched image, and the first stitched image is input into a U-Net network for first mask prediction, wherein the U-Net network includes an encoder and a decoder, and a jump connection exists between the encoder and the decoder. The first stitched image is feature encoded by the encoder to obtain image encoding features, and the image encoding features are feature decoded by the decoder and the jump connection to obtain a first prediction mask.

[0095] In step S230 of some embodiments, in order to obtain a more refined mask, the first prediction mask is refined according to the original image to obtain a second prediction mask. Specifically, the pixel values ​​of the pixels of the original image and the pixels of the second preset mask are added in a position-by-position manner to obtain an initial feature map. The initial feature map is feature extracted through a residual network to obtain a residual feature map. The pixel values ​​of the pixels of the second prediction mask and the pixels of the residual feature map are added in a position-by-position manner to obtain a second prediction mask.

[0096] In step S240 of some embodiments, the model performance of the semantic segmentation model, the U-Net network, and the residual network will be affected by the training data. Since the training data contains limited sample types, the model has limited mask prediction capabilities when facing unknown samples such as samples with occluded regions of interest and complex backgrounds, making it difficult for the model to accurately predict the mask. In order to improve the accuracy of mask prediction, the second predicted mask is adjusted to obtain a target predicted mask.

[0097] In step S250 of some embodiments, the target prediction mask is multiplied by the original image, and the image area covered by the target prediction mask is cut out from the original image to obtain a cutout sub-image.

[0098] Through the above steps S210 to S250, the foreground image elements of interest can be automatically and accurately extracted from the original image, so that different foreground image elements can be selected for image generation according to business requirements.

[0099] See also Figure 3 In some embodiments, step S240 may include but is not limited to steps S310 to S340:

[0100] Step S310, performing mask prediction according to the original image and the foreground sub-image to obtain a third predicted mask;

[0101] Step S320, determining a fourth prediction mask according to the third prediction mask, the boundary sub-image and the foreground sub-image;

[0102] Step S330, calculating first similarity data between the second prediction mask and the fourth prediction mask;

[0103] Step S340: performing mask adjustment on the second prediction mask according to the first similarity data to obtain a target prediction mask.

[0104] In step S310 of some embodiments, the original image provides global visual information, and the foreground sub-image contains detail information such as edges and textures. In order to accurately identify the foreground area in the original image, the original image and the foreground sub-image are spliced ​​to fully utilize global information and local information to obtain a second spliced ​​image. A network with an encoding-decoding structure is used to perform mask prediction on the second spliced ​​image, so that the model can focus on the detail extraction of the foreground area to obtain a third predicted mask.

[0105] In step S320 of some embodiments, in an image with a complex background and an obstructed foreground area, uncertainty will be generated when the model recognizes the foreground and background due to unclear or missing image features, thereby generating an uncertain area. In order to reduce the impact of the uncertain area on the mask prediction, the third predicted mask and the boundary sub-image are multiplied to obtain a mask feature map. In order to obtain detailed information of the foreground area, such as detailed introduction of insurance products, the pixel values ​​of the pixels in the mask feature map and the pixel values ​​of the pixels in the foreground sub-image are added in a symmetrical manner to combine the rough semantic information with the fine detail information, thereby achieving accurate mask prediction and obtaining a fourth predicted mask.

[0106] In step S330 of some embodiments, in order to determine the accuracy of the second prediction mask, the similarity between the second prediction mask and the fourth prediction mask may be calculated using an index such as intersection-over-union ratio and Dice coefficient to obtain first similarity data. The larger the first similarity data, the more similar the second prediction mask and the fourth prediction mask are.

[0107] In step S340 of some embodiments, if the first similarity is greater than or equal to a preset similarity threshold, it means that the second prediction mask is accurate, and then any one of the second prediction mask or the fourth prediction mask is used as the target prediction mask, or the second prediction mask and the fourth prediction mask are masked and added to obtain the target prediction mask. If the first similarity is less than the preset similarity threshold, it means that the second prediction mask may be inaccurate, and mask prediction needs to be performed again to obtain the target prediction mask.

[0108] Through the above steps S310 to S340, the accuracy of the second prediction mask can be determined, so that the second prediction mask can be adjusted when it is inaccurate, thereby improving the accuracy of mask prediction, thereby improving the quality of matting, and further improving the quality of image generation.

[0109] See also Figure 4 In some embodiments, step S340 may include but is not limited to steps S410 to S430:

[0110] Step S410, if the first similarity data is less than a preset similarity threshold, a second mask prediction is performed according to the original image, the foreground sub-image, the background sub-image and the boundary sub-image to obtain a fifth predicted mask;

[0111] Step S420, calculating second similarity data between the second prediction mask and the fifth prediction mask;

[0112] Step S430: performing mask adjustment on the second prediction mask according to the second similarity data to obtain a target prediction mask.

[0113] In step S410 of some embodiments, if the first similarity data is less than a preset similarity threshold, it means that the second predicted mask may be inaccurate and mask prediction needs to be re-performed, then referring to steps S220 to S230, the second mask prediction is performed based on the original image, foreground sub-image, background sub-image and boundary sub-image to obtain a fifth predicted mask.

[0114] In step S420 of some embodiments, due to the randomness of the model training process such as dropout technology and model parameter initialization, the same input may produce different mask prediction results. In order to reduce the impact of randomness on the mask prediction results and make the difference between two adjacent mask prediction results larger, the similarity between the second prediction mask and the fifth prediction mask is calculated using indicators such as intersection-over-union ratio and Dice coefficient to obtain second similarity data. The greater the second similarity, the more similar the second prediction mask and the fifth prediction mask are.

[0115] In step S430 of some embodiments, if the second similarity data is greater than or equal to the preset similarity threshold, it indicates that the mask prediction performance of the model is stable. In order to improve the accuracy and comprehensiveness of the mask prediction and avoid missing foreground area information, the second prediction mask, the fourth prediction mask and the fifth prediction mask are masked and added to obtain a target prediction mask. If the second similarity data is less than the preset similarity threshold, it indicates that the mask prediction performance of the model is unstable, and the third similarity data of the fourth prediction mask and the fifth prediction mask is calculated. If the third similarity data is greater than or equal to the preset similarity threshold, any one of the fourth prediction mask or the fifth prediction mask is used as the target prediction mask. If the third similarity data is less than the preset similarity threshold, the second prediction mask, the fourth prediction mask and the fifth prediction mask are averaged to obtain the target prediction mask. That is, if the second prediction mask is represented by x2, the fourth prediction mask is represented by x4, and the fifth prediction mask is represented by x5, the target prediction mask is represented by (x2+x4+x5) / 3. It should be noted that the preset similarity thresholds used in step S430 and step S340 can be set according to actual conditions, and the two can be the same or different.

[0116] Through the above steps S410 to S430, the erroneous prediction mask can be updated in real time, achieving a balance between mask prediction accuracy and mask prediction efficiency, thereby improving the cutout quality.

[0117] See also Figure 5 In some embodiments, step S120 may include but is not limited to steps S510 to S570:

[0118] Step S510, obtaining mask coordinates according to the cutout operation;

[0119] Step S520, filtering out a foreground area from the original image according to the mask coordinates;

[0120] Step S530, extracting color features from the foreground area to obtain a color feature vector;

[0121] Step S540, performing Gaussian feature extraction on the foreground area to obtain a Gaussian feature vector;

[0122] Step S550, calculating target loss data according to the color feature vector, the Gaussian feature vector and the foreground area;

[0123] Step S560, adjusting the foreground area according to the target loss data to obtain a target operation mask;

[0124] Step S570, performing a cutout process on the original image according to the target operation mask to obtain a cutout sub-image.

[0125] In step S510 of some embodiments, mask coordinates are acquired according to a cutout operation, where the mask coordinates are coordinates of the pixel points marked by the cutout operation in the original image.

[0126] In step S520 of some embodiments, if the number of mask coordinates is less than 2, that is, the number of marked pixel points is less than 2, the cutout operation is performed again. If the number of mask coordinates is equal to 2, it is determined whether the line between the two marked pixel points is a horizontal line or a vertical line. If the line is a horizontal line or a vertical line, the cutout operation is performed again; if the line is not a horizontal line or a vertical line, a closed contour is generated based on the line, and the closed contour is a rectangle. If the number of mask coordinates is greater than 2, the marked pixel points represented by the mask coordinates are connected to form a closed contour. The area formed by the pixels of the original image located within the closed contour and on the closed contour is used as the foreground area.

[0127] In step S530 of some embodiments, in order to improve the accuracy of manually extracting the mask, if the foreground area is a color image, the pixel values ​​of the pixel points in the foreground area in the red channel, the green channel, and the blue channel are obtained to obtain the color feature vector of the pixel point. If the foreground area is a grayscale image, the pixel values ​​of the pixel points in the foreground area are obtained to obtain the color feature vector of the pixel point.

[0128] In step S540 of some embodiments, the cluster center is randomly initialized, the distance between the coordinates of the pixel points in the foreground area and the coordinates of the cluster center is calculated, and the pixel points are divided into the cluster center with the smallest distance to obtain a pixel cluster. The color feature vectors of all the pixels in the pixel cluster are averaged to obtain a mean. The variance is calculated based on the color feature vectors and the mean of each pixel in the pixel cluster to obtain the variance of each pixel, the variance matrix is ​​obtained based on the variance of all the pixels in the pixel cluster, and the Gaussian feature vector is obtained based on the mean and the variance matrix.

[0129] In step S550 of some embodiments, in order to optimize the foreground area, loss calculation is performed based on the color feature vector, the Gaussian feature vector and the foreground area to obtain target loss data.

[0130] In step S560 of some embodiments, the closed contour is adjusted by minimizing the target loss data, the closed contour and the area inside the closed contour are used as the foreground area, and the area outside the closed contour is used as the background area, so as to obtain a target operation mask.

[0131] In step S570 of some embodiments, the target operation mask is multiplied by the original image to extract a cutout sub-image from the original image.

[0132] Through the above steps S510 to S570, the manual mask obtained by the cutout operation can be adjusted, and the user only needs to input at least two pixel coordinates to obtain a complete mask, making manual cutout simpler and more convenient, and improving the accuracy and efficiency of manual cutout.

[0133] See also Figure 6 In some embodiments, step S550 may include but is not limited to steps S610 to S630:

[0134] Step S610, performing Gaussian calculation on the color feature vector and the Gaussian feature vector through a preset mixed Gaussian model to obtain first sub-loss data;

[0135] Step S620, performing smoothing calculation on the foreground area using a preset smoothing model to obtain second sub-loss data;

[0136] Step S630: sum the first sub-loss data and the second sub-loss data to obtain target loss data.

[0137] In step S610 of some embodiments, the Gaussian feature vector includes a mean and a variance matrix. In order to measure the degree of matching between the pixel and the Gaussian feature vector, the color feature vector, mean and variance matrix of each pixel in the pixel cluster are input into the mixed Gaussian model for Gaussian calculation, and the output value of the mixed Gaussian model is used as the first sub-loss data. The smaller the first sub-loss data, the higher the degree of matching between the pixel and the Gaussian feature vector, and the higher the probability that the pixel belongs to the foreground area. The definition of Gaussian calculation is as follows:

[0138]

[0139] Where N represents a mixed Gaussian model that obeys a Gaussian distribution; K is the number of pixel clusters; z n Represents the color feature vector of the pixel in the kth pixel cluster; μ k is the mean of the kth pixel cluster; σ k is the variance matrix of the kth pixel cluster.

[0140] In step S620 of some embodiments, for each pixel in the foreground area, the pixel points in the four neighborhoods or eight neighborhoods of the pixel point are obtained from the original image to obtain the neighborhood pixel points. Based on the smoothing model, the pixel point is compared with each neighborhood pixel point respectively. If the pixel point and the neighborhood pixel point both belong to the foreground area, the label value of the pixel point is 0, otherwise the label value of the pixel point is 1. The label values ​​of all pixels in the foreground area are added to obtain the second sub-loss data. The second sub-loss data is used to measure the similarity between neighborhood pixels. The more similar the neighborhood pixels are, the smaller the second sub-loss data is. The smoothing model is defined as:

[0141] ∑ p∈F Σ q∈L(p) δ(y p ,y q ),

[0142] Where F is the foreground area; p is a pixel in the foreground area; L represents the four-neighborhood or eight-neighborhood of p; q is a neighboring pixel; y p and q are the pixel categories of pixel p and pixel q respectively; δ is the indicator function. If the pixel category of pixel p and the pixel category of pixel q are both foreground pixels, the output value of the indicator function is 0, otherwise it is 1.

[0143] In step S630 of some embodiments, the first sub-loss data and the second sub-loss data are added to obtain the target loss data.

[0144] Through the above steps S610 to S630, target loss data can be obtained, so as to adjust the foreground area based on the target loss data, thereby improving the accuracy of the manual mask.

[0145] In step S130 of some embodiments, the clarity of the cutout sub-image may be reduced due to factors such as distortion amplification and low clarity of the original image, resulting in low quality of image generation. In order to improve the quality of image generation, it is necessary to improve the image clarity of the cutout sub-image. There are multiple storage formats of the original image, such as PNG, JPG, JPEG, WebP, etc., so there are also multiple storage formats for the cutout sub-image, so it is necessary to convert the cutout sub-image into a unified data stream. In order to ensure the security of image generation, the image generation method is executed using a terminal device to avoid interaction between the terminal device and the background server. The storage space of the terminal device is limited. In order to save storage space and improve data processing efficiency, the cutout sub-image is imported into the memory space of the terminal device to obtain a bitmap object. The bitmap object is a two-dimensional array. The bitmap object is tiled into a one-dimensional integer array data stream to obtain the original integer data stream.

[0146] Initialize the tensorflow lite data package on the terminal device, and use the tensorflow lite data package to deploy the super-resolution network to the terminal device. The super-resolution network performs super-resolution processing on the original integer data stream to obtain the target integer data stream. The super-resolution network can be an ESRGAN model, and the target integer data stream is a data stream of integer arrays.

[0147] See also Figure 7 In some embodiments, the super-resolution network includes a first convolutional layer, a dense residual layer, a second convolutional layer, and a third convolutional layer that are cascaded in sequence, and step S140 may include but is not limited to steps S710 to S760:

[0148] Step S710, performing convolution processing on the original integer data stream through a first convolution layer to obtain a first convolution data stream;

[0149] Step S720, performing residual calculation on the first convolution data stream through a dense residual layer to obtain a residual data stream;

[0150] Step S730, performing convolution processing on the residual data stream through a second convolution layer to obtain a second convolution data stream;

[0151] Step S740, fusing the first convolution data stream and the second convolution data stream to obtain a fused data stream;

[0152] Step S750, downsampling the fused data stream to obtain a downsampled data stream;

[0153] Step S760, performing convolution processing on the downsampled data stream through the third convolution layer to obtain a target integer data stream.

[0154] In step S710 of some embodiments, a convolution calculation is performed on the original integer data stream through a first convolution layer to extract image data features of the cutout sub-image to obtain a first convolution data stream.

[0155] In step S720 of some embodiments, the dense residual layer uses a plurality of dense residual blocks cascaded in sequence, and performs residual processing on the first convolution data stream through the first dense residual block to obtain the residual features output by the first dense residual block. The residual features output by the current dense residual block are input to the next dense residual block for residual processing until the residual features output by the last dense residual block are obtained, and the residual features output by the last dense residual block are used as the residual data stream, so that the super-resolution network can learn richer feature information, improve the expression ability of the super-resolution network, and thus provide clearer edges and better visual effects.

[0156] In step S730 of some embodiments, in order to deeply understand the deep-level features of the image data, a convolution calculation is performed on the residual data stream through a second convolution layer to obtain a second convolution data stream.

[0157] In step S740 of some embodiments, in order to ensure the comprehensiveness and sufficiency of feature expression, the first convolution data stream and the second convolution data stream are added to obtain a fused data stream to avoid poor processing results of the super-resolution network due to missing features.

[0158] In step S750 of some embodiments, in order to expand the receptive field and improve the ability to extract global features, the fused data stream is downsampled to obtain a downsampled data stream.

[0159] In step S760 of some embodiments, a convolution calculation is performed on the downsampled data stream through a third convolution layer to obtain a target integer data stream.

[0160] In the above steps S710 to S760, integer data streams are used instead of two-dimensional images as network input, which can improve the processing efficiency of the super-resolution network and improve the clarity of the cutout sub-image.

[0161] In step S150 of some embodiments, in order to obtain a clear cutout image, the target integer data stream is format-converted, and the target integer data stream is read as an image into a bitmap object in the memory of the terminal device to obtain a reference sub-image. The resolution of the reference sub-image is greater than the resolution of the cutout sub-image, that is, the image clarity of the reference sub-image is higher than the image clarity of the cutout sub-image.

[0162] In step S160 of some embodiments, in order to facilitate subsequent image generation operations, a layer of a reference sub-image is obtained to obtain a reference layer, so as to use the reference layer to control the distribution of the reference sub-image on the base sub-image and improve the quality of image generation.

[0163] In step S170 of some embodiments, a reference image is obtained from a local material library or an online material library, and the reference image is another image different from the original image. According to the reference layer, the reference sub-image is placed on the reference image, and the position of the reference sub-image on the reference image is adjusted to obtain the target image, thereby avoiding the unclear target image caused by the unclear original image or the cutout sub-image, and improving the quality of image generation. For example, in the insurance industry, different image elements can be stacked on the reference image to generate a new poster image for the business operation of insurance products.

[0164] See also Figure 8 The present application also provides an image generating device that can implement the above-mentioned image generating method. The image generating device includes:

[0165] An instruction acquisition module 810 is used to acquire an original image and a cutout instruction for instructing to perform a cutout process on the original image; the cutout instruction includes an automatic cutout instruction or a manual cutout instruction;

[0166] A cutout module 820, configured to cutout the original image according to an automatic cutout instruction or a cutout operation to obtain a cutout sub-image; wherein the cutout operation is an operation obtained based on a manual cutout instruction;

[0167] An image format conversion module 830 is used to convert the format of the cutout sub-image to obtain an original integer data stream;

[0168] A super-resolution module 840 is used to perform super-resolution processing on the original integer data stream through a preset super-resolution network to obtain a target integer data stream;

[0169] The data stream format conversion module 850 is used to convert the target integer data stream into a format to obtain a reference sub-image; the resolution of the reference sub-image is greater than the resolution of the cutout sub-image;

[0170] The layer acquisition module 860 is used to acquire the layer of the reference sub-image to obtain the reference layer;

[0171] The image generation module 870 is used to obtain a reference image, and merge the reference image and the reference sub-image according to the reference layer to obtain a target image.

[0172] The specific implementation of the image generating device is substantially the same as the specific implementation of the above-mentioned image generating method, and will not be described in detail here.

[0173] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned image generation method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0174] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0175] The processor 910 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0176] The memory 920 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 920, and the processor 910 calls and executes the image generation method of the embodiment of this application;

[0177] Input / output interface 930, used to implement information input and output;

[0178] Communication interface 940, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0179] bus 950 , which transmits information between the various components of the device (e.g., processor 910 , memory 920 , input / output interface 930 , and communication interface 940 );

[0180] The processor 910 , the memory 920 , the input / output interface 930 , and the communication interface 940 are connected to each other in communication within the device via a bus 950 .

[0181] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned image generation method is implemented.

[0182] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0183] The image generation method, image generation device, electronic device and computer storage medium provided in the embodiments of the present application obtain a cutout sub-image by cutting out the foreground image element of interest from the original image. In order to prevent the magnification distortion caused by the cutout sub-image from affecting the quality of image generation, the cutout sub-image is super-resolution processed to obtain a reference sub-image to improve the image clarity of the cutout sub-image. The reference sub-image and the reference image are merged according to the layer to accurately control the distribution of the reference sub-image on the reference sub-image, thereby improving the quality of image generation.

[0184] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0185] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0186] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0187] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0188] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0189] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0190] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0191] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0192] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0193] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0194] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. An image generation method, characterized in that: The method comprises: Acquire an original image and a cutout instruction for instructing to perform a cutout process on the original image; the cutout instruction includes an automatic cutout instruction or a manual cutout instruction; Performing a cutout process on the original image according to the automatic cutout instruction or the cutout operation to obtain a cutout sub-image; wherein the cutout operation is an operation obtained based on the manual cutout instruction; Performing format conversion on the cutout sub-image to obtain an original integer data stream; Performing super-resolution processing on the original integer data stream through a preset super-resolution network to obtain a target integer data stream; Performing format conversion on the target integer data stream to obtain a reference sub-image; the resolution of the reference sub-image is greater than the resolution of the cutout sub-image; Acquire the layer of the reference sub-image to obtain a reference layer; A reference image is acquired, and the reference image and the reference sub-image are merged according to the reference layer to obtain a target image.

2. The method according to claim 1, characterized in that The performing a cutout process on the original image according to the automatic cutout instruction or the cutout operation to obtain a cutout sub-image includes: Performing image segmentation on the original image according to the automatic cutout instruction to obtain an image triplet; the image triplet includes a foreground sub-image, a background sub-image and a boundary sub-image; Performing a first mask prediction according to the original image, the foreground sub-image, the background sub-image and the boundary sub-image to obtain a first predicted mask; Performing mask prediction according to the original image and the first prediction mask to obtain a second prediction mask; performing mask adjustment on the second prediction mask to obtain a target prediction mask; The original image is subjected to a matting process according to the target prediction mask to obtain the matting sub-image.

3. The method according to claim 2, characterized in that The step of performing mask adjustment on the second prediction mask to obtain a target prediction mask includes: Performing mask prediction according to the original image and the foreground sub-image to obtain a third predicted mask; determining a fourth prediction mask according to the third prediction mask, the boundary sub-image and the foreground sub-image; calculating first similarity data between the second prediction mask and the fourth prediction mask; The second prediction mask is subjected to mask adjustment according to the first similarity data to obtain the target prediction mask.

4. The method according to claim 3, characterized in that The step of performing mask adjustment on the second prediction mask according to the first similarity data to obtain the target prediction mask includes: If the first similarity data is less than a preset similarity threshold, performing a second mask prediction according to the original image, the foreground sub-image, the background sub-image and the boundary sub-image to obtain a fifth predicted mask; calculating second similarity data between the second prediction mask and the fifth prediction mask; The second prediction mask is subjected to mask adjustment according to the second similarity data to obtain the target prediction mask.

5. The method according to any one of claims 1 to 4, characterized in that: The performing a cutout process on the original image according to the automatic cutout instruction or the cutout operation to obtain a cutout sub-image includes: Acquiring mask coordinates according to the cutout operation; Filtering a foreground area from the original image according to the mask coordinates; Extracting color features from the foreground area to obtain a color feature vector; Performing Gaussian feature extraction on the foreground area to obtain a Gaussian feature vector; Calculating target loss data according to the color feature vector, the Gaussian feature vector and the foreground area; Adjusting the foreground area according to the target loss data to obtain a target operation mask; The original image is subjected to a cutout process according to the target operation mask to obtain the cutout sub-image.

6. The method according to claim 5, characterized in that The calculating target loss data according to the color feature vector, the Gaussian feature vector and the foreground area comprises: Performing Gaussian calculation on the color feature vector and the Gaussian feature vector through a preset mixed Gaussian model to obtain first sub-loss data; Performing smoothing calculation on the foreground area through a preset smoothing model to obtain second sub-loss data; The first sub-loss data and the second sub-loss data are summed to obtain the target loss data.

7. The method according to any one of claims 1 to 4, characterized in that: The super-resolution network includes a first convolutional layer, a dense residual layer, a second convolutional layer and a third convolutional layer. The original integer data stream is subjected to super-resolution processing by a preset super-resolution network to obtain a target integer data stream, including: Performing convolution processing on the original integer data stream through the first convolution layer to obtain a first convolution data stream; Performing residual calculation on the first convolution data stream through the dense residual layer to obtain a residual data stream; Performing convolution processing on the residual data stream through the second convolution layer to obtain a second convolution data stream; Fusing the first convolution data stream and the second convolution data stream to obtain a fused data stream; Downsampling the fused data stream to obtain a downsampled data stream; The down-sampled data stream is convolved by the third convolutional layer to obtain the target integer data stream.

8. An image generating device, characterized in that: The device comprises: An instruction acquisition module, used for acquiring an original image and a cutout instruction for instructing to perform a cutout process on the original image; the cutout instruction includes an automatic cutout instruction or a manual cutout instruction; A cutout module, configured to perform a cutout process on the original image according to the automatic cutout instruction or the cutout operation to obtain a cutout sub-image; wherein the cutout operation is an operation obtained based on the manual cutout instruction; An image format conversion module, used for performing format conversion on the cutout sub-image to obtain an original integer data stream; A super-resolution module, used for performing super-resolution processing on the original integer data stream through a preset super-resolution network to obtain a target integer data stream; A data stream format conversion module, used for performing format conversion on the target integer data stream to obtain a reference sub-image; the resolution of the reference sub-image is greater than the resolution of the cutout sub-image; A layer acquisition module, used to acquire the layer of the reference sub-image to obtain a reference layer; The image generation module is used to obtain a reference image, and merge the reference image and the reference sub-image according to the reference layer to obtain a target image.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images