Image data generation method and device, electronic equipment and storage medium

Abstract

The invention discloses an image data generation method, belongs to the field of data processing, and is used for increasing the sense of reality of a generated image. The method comprises the following steps: acquiring a background picture to be subjected to data fusion, and performing mask segmentation on the background picture to obtain a region to be fused in the background picture; a template text is obtained, a template picture fused with the background picture is generated based on the template text, and the template text is a text which is generated randomly and used for describing the content of the template picture; obtaining a preset to-be-migrated style picture, and inputting the to-be-migrated style picture and the template picture into a pre-trained style migration model to obtain a foreground picture; and fusing the foreground picture into the to-be-fused region in the background picture to obtain target image data.

Description

Technical Field

[0001] The present application belongs to the field of data processing, and specifically relates to an image data generation method, device, electronic device and storage medium. Background Art

[0002] The existing image generation methods are based on rules and foreground-background fusion, which mainly rely on predefined rules and geometric models. By separating and fusing the foreground and background, images with specific structures and contents are generated. For example, in virtual scene construction, geometric shapes and color filling are used to construct the foreground of an object, which is then synthesized with a designed background. The advantage of this method is the precise control of the generated content, which is suitable for scenes that require clear geometric relationships and content structures. However, due to the lack of processing capabilities for complex details and natural changes, the generated images often appear more rigid and lack a sense of reality.

[0003] Therefore, there is a need for a method of generating image data that can increase the realism of the generated image. Summary of the invention

[0004] The embodiment of the present application provides a method for generating image data, which can increase the realism of generated image data.

[0005] In the first aspect, an embodiment of the present application provides an image data generation method, which includes: obtaining a background image to be fused, and performing mask segmentation on the background image to obtain an area to be fused in the background image; obtaining a template text, and generating a template image fused with the background image based on the template text, wherein the template text is a randomly generated text describing the content of the template image; obtaining a preset style image to be transferred, and inputting the style image to be transferred and the template image into a pre-trained style transfer model to obtain a foreground image; fusing the foreground image into the area to be fused in the background image, thereby obtaining target image data.

[0006] In the second aspect, an embodiment of the present application provides an image data generating device, which includes: a first segmentation module, used to obtain a background image to be fused, and perform mask segmentation on the background image to obtain an area to be fused in the background image; a first generation module, used to obtain a template text, and generate a template image fused with the background image based on the template text, wherein the template text is a randomly generated text describing the content of the template image; a first migration module, used to obtain a preset style image to be migrated, and input the style image to be migrated and the template image into a pre-trained style migration model to obtain a foreground image; a first fusion module, used to fuse the foreground image into the area to be fused in the background image, thereby obtaining target image data.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0010] In an embodiment of the present application, a background image to be fused is obtained, and mask segmentation is performed on the background image to obtain an area to be fused in the background image; a template text is obtained, and a template image fused with the background image is generated based on the template text, wherein the template text is a randomly generated text describing the content of the template image; a preset style image to be transferred is obtained, and the style image to be transferred and the template image are input into a pre-trained style transfer model to obtain a foreground image; the foreground image is fused into the area to be fused in the background image to obtain target image data, thereby increasing the authenticity of the generated image data. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 It is a flowchart of a method for generating image data provided by an embodiment of the present application; Figure 2 is a flow chart of a second method for generating image data provided in an embodiment of the present application; Figure 3 is a flowchart of a third method for generating image data provided in an embodiment of the present application; Figure 4 is a flowchart of a fourth method for generating image data provided in an embodiment of the present application; Figure 5 is a structural schematic diagram of an image data generating device provided in an embodiment of the present application; Figure 6 It is a structural schematic diagram of an image data generating device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0012] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0013] The following is a detailed description of the image data generation method provided by the embodiment of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0014] Figure 1 An image data generation method provided by an embodiment of the present invention is shown. The method can be executed by an electronic device, and the electronic device may include: a server and / or a terminal device, wherein the terminal device may be, for example, a vehicle-mounted terminal or a mobile phone terminal. In other words, the method can be executed by software or hardware installed in the image data generation device, and the method includes the following steps: Step 102: obtaining a background image to be subjected to data fusion, and performing mask segmentation on the background image to obtain a region to be fused in the background image; The image data generation system obtains the background image to be fused and segments the background image to obtain the area to be fused in the background image. That is, the background image to be fused includes the area to be fused and the non-fused area. The image data generation system determines the area to be fused and the non-fused area in the background image by performing mask segmentation on the background image.

[0015] Specifically, when the image data generation system performs mask segmentation on the background image, it can be determined by a pre-trained segmentation model or a preset algorithm. It performs convolution, pooling and other operations on the input image through a neural network, gradually extracts the feature information of the image, and accurately describes the positions and boundaries of different objects in the image. That is, by performing mask segmentation on the background image, the objects in the image can be accurately separated from the background.

[0016] More specifically, the background image to be fused may be a water area background image, such as a picture of a mid-high point monitoring scene with a water area, and the water area background image includes a water area and a non-water area. The image data generation system performs mask segmentation on the water area background image to obtain the area to be fused (i.e., the water area) and the non-fused area (i.e., the non-water area) in the image.

[0017] Step 104: Acquire a template text, and generate a template image fused with the background image based on the template text.

[0018] The template text is randomly generated text that describes the content of the template image.

[0019] The image data generation system obtains a template text, which is a randomly generated text describing the content of the template image. After obtaining the template text, the image data generation system generates images and texts according to the description content of the template text, thereby determining the template image that needs to be merged with the background image.

[0020] Specifically, the template text obtained by the image data generation system is generated according to a preset rule, and the preset rule is determined based on the feature information of the template image. The preset rule includes multiple features and / or multiple rules of the template image, so the image data generation system can randomly generate the template text according to certain features and / or rules. For example, when the background image to be fused is a water background image, the template image fused with the background image can be a water level gauge image, and the content of the water level gauge is composed of a group of continuous numbers (from large to small, the maximum is 99, the minimum is 0, the length is 2 to 9, and 0 may also represent 10) and the letter E. There is also an inverted E on the right side of each number. Therefore, when synthesizing the water gauge, a group of continuous numbers from large to small with a length between 2-9 is randomly generated according to the characteristics or rules, and then E is inserted between the numbers. The starting position of the number also chooses whether to add E, and finally generates a random number that meets the rules, such as E9E8E7E6.

[0021] After the template text is generated, the image data generation system generates images and texts according to the template text, thereby determining the template image corresponding to the template text. For example, when generating a template image corresponding to the water level gauge content, in addition to synthesizing numbers and the letter E, it is also necessary to synthesize the mirror image of E. First, use Pillow to write the characters from top to bottom, with a white background, black font color, and the font type being the standard Microsoft YaHei font, and record the position of each number. At the same time, generate a white background image of the same size as the image, and then write E in the position of the number in the white background image. After writing E, flip the image horizontally 180 degrees, and then splice it from the right to the image with the numbers and E, thereby obtaining the template image.

[0022] Step 106: Obtain a preset style image to be transferred, and input the style image to be transferred and the template image into a pre-trained style transfer model to obtain a foreground image.

[0023] After generating the template image, the image data generation system obtains the preset style image to be transferred, and inputs the style image to be transferred and the template image into the pre-trained style transfer model to obtain the foreground image. In other words, after the image data generation system generates the template image, it does not directly merge the template image with the background image, but first obtains the style image to be transferred, and transfers the style in the style image to be transferred to the template image through the pre-trained style transfer model to obtain the foreground image.

[0024] Specifically, the image data generation system pre-stores a plurality of style images to be transferred. After each template image is generated, the image data generation system randomly selects a style image to be transferred from the plurality of style images to be transferred, and determines the style in the randomly selected style image as the style to be transferred to the template image. When performing style transfer, the image data generation system can transfer the style through StyleGAN, transfer the style in the style image to be transferred to the template image, and thus obtain the template image after style transfer, that is, the foreground image.

[0025] Step 108: Fusing the foreground image into the to-be-fused region in the background image to obtain target image data.

[0026] After obtaining the foreground image, the image data generation system fuses the foreground image into the area to be fused in the background image, thereby obtaining target image data. That is, the background image includes the area to be fused and the non-fused area, and the image data generation system fuses the foreground image into the area to be fused in the background image, thereby obtaining target image data.

[0027] Specifically, when fusing the foreground image with the area to be fused, the foreground image can be fused to any position in the area to be fused, or the fusion point of the foreground image and the area to be fused can be determined according to a preset rule, and then the foreground image and the background image are fused at the fusion point.

[0028] More specifically, the image data generation system uses the ControlCom solution (improved based on Stable diffusion) to fuse the foreground image into the background image, and in order to reduce computing power overhead, ControlCom is selected to generate fusion in a 512*512 image. Therefore, a local background image of a 512*512 area is selected from the background image based on the coordinates of the foreground to be fused in the background image, and then the foreground is fused into the local background image, and finally the local background image is restored to the original background image. The image data generation system combines the Transformer-based foreground encoder with the diffusion U-Net model, which enhances the details and controllability of the generated image. The foreground encoder extracts global and local features from the foreground image, and then fuses them with the background image in the controllable generator. The whole process uses the attention mechanism and diffusion technology to ensure high-quality and consistent integration between the foreground and the background.

[0029] The image data generation method provided by the embodiment of the present invention obtains a background image to be fused and performs mask segmentation on the background image to obtain a region to be fused in the background image; obtains a template text and generates a template image fused with the background image based on the template text, wherein the template text is a randomly generated text describing the content of the template image; obtains a preset style image to be transferred, and inputs the style image to be transferred and the template image into a pre-trained style transfer model to obtain a foreground image; and fuses the foreground image into the region to be fused in the background image to obtain target image data, thereby increasing the realism of the generated image data.

[0030] In one implementation, the background image to be fused is obtained, and the background image is segmented by mask to obtain the area to be fused in the background image (step 102), and steps A1-A4 may be performed: Step A1: Obtain a first image, and perform mask segmentation on the first image to obtain a first region and a second region.

[0031] The first area is a to-be-fused area in the first picture, and the second area is a non-fused area in the first picture.

[0032] The image data generation system obtains a first image and performs mask segmentation on the first image to obtain an area to be fused (i.e., the first area) and a non-fused area (i.e., the second area) in the first image. The first image is a background image including the area to be fused and the non-fused area, and the first image can be obtained from the network or from a preset database.

[0033] Step A2: obtaining a first length corresponding to each preset reference object in the second area; After determining the first area and the second area, the image data generation system determines the first length corresponding to each reference object in the second area (i.e., the non-fused area). In other words, the non-fused area of ​​the background image also includes multiple reference objects. After determining the non-fused area, the image data generation system first detects each reference object in the non-fused area, and then determines the first length of each reference object.

[0034] Specifically, when the background image is a water background image, the first area is the water area, and the second area is the non-water area. The image data generation system determines each reference object in the non-water area and the first length of each reference object. The reference object can be used to constrain the size of the fused water level gauge, and can also be used to detect whether the first image is suitable as a background image. The reference objects mainly include 6 categories: people, cars, houses, trees, telephone poles, and bridge piers. Taking people as an example, people are generally about 1.7 meters tall, and the height of the water level gauge is generally about 1 meter. In the same picture, the height ratio of people and water level gauges with similar distances should be controlled between 1.5 and 1.9. The target detection model of the reference object in this algorithm adopts yolov8s, and the size of the picture is 960*960 when training the model.

[0035] Step A3: Determine whether the first image meets a preset feasibility condition based on the first area and the second area.

[0036] The feasibility condition includes a first condition and / or a second condition, the first condition is that the area ratio of the first region to the first image is greater than a first threshold, and the second condition is that there is any first length greater than a second threshold.

[0037] After determining the first area and the second area, the image data generation system determines whether the first picture meets a preset feasibility condition based on the first area and the second area, and the feasibility condition includes a first condition and / or a second condition, the first condition being that the area of ​​the first area accounts for a proportion of the total area of ​​the first picture that is greater than a preset first threshold, and the second condition being that there is any reference object whose length is greater than a preset second threshold, wherein the first threshold and the second threshold are values ​​set in advance based on actual conditions.

[0038] Specifically, the first condition is used to determine whether the area ratio of water in the background image is greater than a threshold, and the second condition is used to determine whether the longest side dimension of the reference object is greater than a preset threshold. For example, the second condition is used to determine whether there is a reference object whose longest side dimension is greater than 100 pixels, and the first condition is used to determine whether the water area is greater than 0.25 of the total image area.

[0039] Step A4: If yes, determine that the first picture is the background picture.

[0040] The image data generation system determines that the first image is a background image when the first image meets a preset feasibility condition, and determines that the first image is a non-background image when it is determined that the first image does not meet the preset feasibility condition.

[0041] Specifically, the image data generation system can determine that the first image is suitable as a background image when it is determined that the first image satisfies the first condition and the second condition, and when the first image does not satisfy any of the first condition and the second condition, it is determined that it is not suitable as a background image. For example, when the first image is a water background image, when its longest side dimension is less than 100 pixels, the size of the fused water level gauge will be too small, and the reading of the water level gauge will also be unrecognizable. Therefore, when the longest side dimension of the reference object is less than 100 pixels, it is determined that it is not suitable as a background image. For example, when the area of ​​the water area is too small (set to 0.25 of the area of ​​the whole image based on experience), when the water area in the water background image is lower than this threshold, the generated water level gauge image will still look unreal. Therefore, when the area ratio of the water area is less than 0.25, it is determined that it is not suitable as a background image.

[0042] In one implementation, the background image to be fused is obtained, and the background image is segmented to obtain the area to be fused in the background image (step 102), and steps B1-B4 may be performed: Step B1: Obtain a third image and annotation data of the third image.

[0043] The annotation data is data determined by annotating a first area and a second area in the third image, the first area is an area to be fused in the third image, and the second area is an unfused area in the third image.

[0044] The image data generation system obtains a third image and the annotation data of the third image, wherein the third image is a background image including the area to be fused and the non-fused area, and the annotation data is data determined by annotating the area to be fused and the non-fused area in the third image. In other words, the image data generation system obtains the background image annotated with the area to be fused and the non-fused area.

[0045] Step B2: input the third picture into the image segmentation model to be trained, and obtain a segmentation result.

[0046] After acquiring the third picture, the image data generation system inputs the third picture into the image segmentation model to be trained and obtains a segmentation result. That is, the image data generation system inputs the third picture into the image segmentation model to be trained so that the model segments the third picture and determines the first region and the second region obtained by the model segmentation.

[0047] Step B3: iteratively updating the image segmentation model to be trained based on the segmentation result and the annotation data to obtain a pre-trained image segmentation model.

[0048] After obtaining the segmentation result (i.e., the segmentation result of the image segmentation model to be trained for the first area and the second area in the third picture), the image data generation system compares the segmentation result with the annotation data, and iteratively updates the image segmentation model to be trained based on the comparison result to obtain a pre-trained image segmentation model.

[0049] Specifically, the water segmentation of background images adopts the fine-tuned SAM model (image encoder microvit-b). During fine-tuning, a water segmentation dataset of 200 labeled mid-high point monitoring scenes is used. The model input size is changed from the original 512*512 to 960*960. During fine-tuning, about 4 rounds of iterations are required to achieve the expected results.

[0050] Step B4: input the background image into the pre-trained image segmentation model to obtain the area to be fused in the background image.

[0051] After obtaining the pre-trained image segmentation model, the image data generation system inputs the background image into the pre-trained image segmentation model, thereby obtaining a segmentation result obtained by the pre-trained image segmentation model for segmenting the background image, that is, determining the area to be fused and the non-fused area in the background image.

[0052] Figure 2 is a flow chart of a second method for generating image data provided in an embodiment of this specification, such as Figure 2 As shown, the schematic diagram includes: Step 202: Obtain a third image and annotation data of the third image.

[0053] The annotation data is data determined by annotating a first area and a second area in the third image, the first area is an area to be fused in the third image, and the second area is an unfused area in the third image.

[0054] Step 204: input the third image into the image segmentation model to be trained, and obtain a segmentation result.

[0055] Step 206: iteratively update the image segmentation model to be trained based on the segmentation result and the annotation data to obtain a pre-trained image segmentation model.

[0056] Step 208: Obtain a first image, and input the first image into the pre-trained image segmentation model to obtain a first region and a second region.

[0057] The first area is a to-be-fused area in the first picture, and the second area is a non-fused area in the first picture.

[0058] Step 210: Obtain a first length corresponding to each preset reference object in the second area.

[0059] Step 212: Determine whether the first image meets a preset feasibility condition based on the first area and the second area.

[0060] The feasibility condition includes a first condition and / or a second condition, the first condition is that the area ratio of the first region to the first image is greater than a first threshold, and the second condition is that there is any first length greater than a second threshold.

[0061] Step 214: If yes, determine that the first picture is the background picture; Step 216: Acquire a template text, and generate a template image fused with the background image based on the template text.

[0062] The template text is randomly generated text that describes the content of the template image.

[0063] Step 218: obtaining a preset style image to be transferred, and inputting the style image to be transferred and the template image into a pre-trained style transfer model to obtain a foreground image; Step 220: Fuse the foreground image into the to-be-fused region in the background image, thereby obtaining target image data.

[0064] This embodiment segments the first image through a pre-trained model to obtain the area to be fused and the non-fused area, and determines whether the area to be fused and the non-fused area meet the feasibility conditions. Finally, when the feasibility conditions are met, the first image is determined as the background image. This not only improves the accuracy of segmenting the first image, but also ensures that the background image is more realistic after fusion.

[0065] In one implementation, before obtaining the style image to be transferred and inputting the style image to be transferred and the template image into the pre-trained style transfer model to obtain the foreground image (step 106), steps C1-C6 may be further performed: Step C1: Get a sample foreground image.

[0066] The sample foreground picture is segmented from a picture including the sample foreground picture.

[0067] The image data generation system obtains a sample foreground image, which is obtained by segmenting a picture including the sample foreground image. That is, the image data generation system first obtains an overall picture including the foreground picture, and then segments the sample foreground image from the overall picture, wherein the sample foreground image is a foreground image in the overall picture.

[0068] Specifically, the image data generation system can first obtain a water area picture including a water level gauge (i.e., the whole picture) from the template library, and then obtain a water level gauge picture (i.e., the sample foreground picture) by cutting out the water level gauge picture from the picture. After obtaining the water level gauge picture, the image data generation system can first obtain the four vertices of the water level gauge picture, rotate the four vertices, and then perform tilt and distortion correction processing on the rotated picture, and finally determine the obtained picture as the sample foreground picture.

[0069] Step C2: performing distortion and cropping processing on the sample foreground image to obtain a sample style image.

[0070] After determining the sample foreground image, the image data generation system distorts and crops the acquired foreground image to obtain a sample style image. Specifically, after acquiring the sample foreground image, the image data generation system can randomly crop the sample foreground image, and then randomly rotate and distort the remaining image to obtain the sample style image.

[0071] Step C3: Perform text recognition on the sample foreground image, and generate text and images based on the recognized text to obtain a sample template image.

[0072] The image data generation system performs text recognition on the sample foreground image to obtain the recognized text and generates text and images based on the recognized text to obtain the sample template image. Specifically, the image data generation system can recognize the content in the water level gauge through OCR (CRNN), and then synthesize the sample template image through the text content.

[0073] Step C4: input the sample template image and the sample style image into the style transfer model to be trained to obtain a sample prediction image.

[0074] After determining the sample template image and the sample style image, the sample template image is used as the template image for style transfer, and the sample style image is used as the reference image for style transfer. Then, the sample template image and the sample style image are input into StyleGAN for prediction, and a sample prediction image is obtained.

[0075] Step C5: input the sample prediction picture and the sample foreground picture into the discriminator to obtain a discrimination result.

[0076] The discriminator is used to determine whether the style and / or content of the sample prediction picture is the same as that of the sample foreground picture.

[0077] After the image data generation system generates a sample prediction picture, it can identify the sample prediction picture and the sample foreground picture through a preset discriminator, judge the style and / or content of both the sample prediction picture and the sample foreground picture, and determine whether the style and / or content of the two are the same.

[0078] Step C6: updating the parameters of the style transfer model to be trained based on the discrimination result to obtain the pre-trained style transfer model.

[0079] After determining the discrimination result, the image data generation system iteratively updates the parameters in the style transfer model to be trained according to the discrimination result, thereby obtaining a trained style transfer model.

[0080] Figure 3 is a flow chart of a second method for generating image data provided in an embodiment of this specification, such as Figure 3 As shown, the schematic diagram includes: Step 302: Obtain a background image to be subjected to data fusion, and perform mask segmentation on the background image to obtain a region to be fused in the background image.

[0081] Step 304: Acquire a template text, and generate a template image fused with the background image based on the template text.

[0082] The template text is randomly generated text that describes the content of the template image.

[0083] Step 306: Obtain a sample foreground image.

[0084] The sample foreground picture is segmented from a picture including the sample foreground picture.

[0085] Step 308: Perform distortion and cropping processing on the sample foreground image to obtain a sample style image.

[0086] Step 310: Perform text recognition on the sample foreground image, and generate text and images based on the recognized text to obtain a sample template image.

[0087] Step 312: input the sample template image and the sample style image into the style transfer model to be trained to obtain a sample prediction image.

[0088] Step 314: input the sample prediction picture and the sample foreground picture into a discriminator to obtain a discrimination result.

[0089] The discriminator is used to determine whether the style and / or content of the sample prediction picture is the same as that of the sample foreground picture.

[0090] Step 316: updating the parameters of the style transfer model to be trained based on the discrimination result to obtain a pre-trained style transfer model.

[0091] Step 318: Obtain a preset style image to be transferred, and input the style image to be transferred and the template image into the pre-trained style transfer model to obtain a foreground image.

[0092] Step 320: Fuse the foreground image into the to-be-fused area in the background image, thereby obtaining target image data.

[0093] This embodiment trains the style transfer model to be trained, and performs style transfer on the randomly generated template image based on the pre-trained style transfer image, and then fuses the foreground image obtained by style transfer into the background image. This can quickly generate a foreground image for fusion, thereby improving the efficiency of image data generation.

[0094] In one implementation, the foreground image is merged into the to-be-merged region in the background image to obtain target image data (step 108), and steps D1-D2 may be performed: Step D1: determining a fusion position in the area to be fused based on a preset first algorithm.

[0095] The first algorithm is used to determine the position of the foreground image in the background image when the foreground image and the background image are merged.

[0096] Before the image data generation system integrates the foreground image into the background image, the image data generation system needs to determine the coordinates of the foreground image in the background image, that is, the fusion position. The image data generation system can determine the fusion position of the foreground image in the to-be-fused area of ​​the background image according to a preset first algorithm, which is preset by the user and is used to determine the position of the foreground image in the background image when the foreground image and the background image are fused.

[0097] Specifically, when determining the fusion position in the area to be fused based on the first algorithm, the center point of the area to be fused can be determined by the first algorithm, and the center point can be determined as the fusion position. A point can also be determined by the first algorithm among multiple points adjacent to the area to be fused and the non-fused area as the fusion position. The point in the algorithm of the area to be fused that is closer to the non-fused area can also be determined by the first algorithm as the fusion position.

[0098] Step D2: Based on the determined fusion position, the foreground image is fused into the background image to obtain the target icon data.

[0099] After determining the fusion position, the image data generation system fuses the foreground image to the fusion position of the area to be fused based on the determined fusion position, thereby obtaining target image data.

[0100] In one implementation, the step of determining the fusion position in the to-be-fused area based on the preset first algorithm (step D1) may include executing steps E1-E3: Step E1: Acquire the first center point in the area to be fused and the second center point in the second area.

[0101] The first center point is the center point of the inscribed rectangle in the area to be fused, the second center point is the center point of a preset reference object in the second area, and the second area is a non-fused area in the background image.

[0102] The image data generation system obtains a first center point in the area to be fused and a second center point in the second area, wherein the first center point is the geometric center point of the inscribed rectangle of the area to be fused, and the second center point is the geometric center point of the reference object in the non-fused area (second area).

[0103] Specifically, the image data generation system first determines the inscribed rectangle of the area to be fused, then determines the coordinates of the center point of the inscribed rectangle, then determines the reference object area in the non-fused area, and finally determines the center point of the reference.

[0104] More specifically, when there are multiple reference objects in the non-fusion area, the image data generation system can determine the target reference object from the multiple reference objects according to a preset priority determination algorithm (such as selecting the reference object with the highest priority among the multiple reference objects as the target reference object), and determine the center point of the target reference object, wherein the priority determination algorithm is determined according to the reference object recognition confidence and the basic priority, the reference object recognition confidence is the confidence determined when the image data generation system recognizes the reference object in the non-fusion area, and the basic priority is the different priorities pre-set according to the different types of reference objects. Further, when the image data generation system determines the priority according to the reference object recognition confidence and the basic priority, it can comprehensively sort the priorities of multiple reference objects based on the reference object recognition confidence and the basic priority (such as sorting based on the product of the reference object recognition confidence and the reference object priority coefficient), and determine the target reference object according to the sorting result.

[0105] Step E2: Connect the first center point and the second center point to obtain a first line segment, and connect the second center point and the third center point to obtain a second line segment.

[0106] The third center point is a point whose straight-line distance from the second center point meets a preset condition, and the preset condition is determined based on the length value and the width value of the inscribed rectangle.

[0107] After determining the first center point and the second center point, the image data generation system can not only connect the first center point and the second center point to obtain a first line segment, but also determine an area where the distance from the second center point meets preset conditions, and obtain the third center point from the area, and finally connect the third center point and the second center point to obtain a second line segment.

[0108] Specifically, when determining the third center point, the image data generation system can first determine the length and width of the inscribed rectangle in the area to be fused, then determine an area whose distance from the second center point is greater than 0.5 times the width and less than 0.5 times the length, and select any point in the area as the third center point.

[0109] Step E3: When the angle between the first line segment and the second line segment is smaller than a preset threshold, determining the third center point as the fusion position.

[0110] After determining the first line segment and the second line segment, the image data generation system determines whether the angle between the first line segment and the second line segment is less than a preset threshold (for example, 30°). If so, the third center point is determined as the fusion position. If not, the third center point is reselected and determined.

[0111] That is to say, when determining the fusion position, the image data generation system first calculates the center point of the inscribed rectangle of the non-fused area, then connects the center point and the center point of the reference object into a line segment, and then selects a third center point. The angle between the point and the line segment is between -30° and +30°, and the radius is between the small circle and the large circle. Finally, the third center point is determined as the fusion position, that is, the position when the water level scale of the foreground image is fused with the area to be fused.

[0112] In one implementation, the foreground image is merged into the to-be-merged region in the background image to obtain target image data (step 108), and steps F1-F4 may be performed: Step F1: performing mask segmentation on the background image to obtain a second area in the background image.

[0113] The second area is a non-fused area in the background image.

[0114] The image data generation system performs mask segmentation on the background image, and can segment the to-be-fused area and the non-fused area in the background image, thereby determining the first area (the to-be-fused area) and the second area (the non-fused area).

[0115] Step F2: Obtain a first length of a preset reference object in the second area.

[0116] After determining the second area (non-fused area), the image data generation system detects a preset reference object in the second area and determines a first length of the preset reference object, wherein the preset reference object may be a number, a person, or the like.

[0117] Step F3: Determine the second length of the foreground image based on the first length and a preset ratio information table.

[0118] The ratio information table pre-stores the length ratio between the preset reference object and the foreground image.

[0119] After determining the first length of the preset reference object, the image data generation system obtains the foreground image ratio corresponding to the first length in the preset ratio information table, and then determines the length of the foreground image according to the first length and the foreground image ratio. In other words, the image data generation system can determine the length of the foreground image according to the length of the reference object in the background image and the preset ratio relationship between the reference object and the foreground image.

[0120] Step F4: setting the length of the foreground image to the second length, and integrating the set foreground image into the to-be-integrated area in the background image, thereby obtaining target image data.

[0121] After determining the length of the foreground image, the image data generation system sets the length of the foreground image to the second length, and then determines the foreground image that needs to be merged into the background image, and merges the foreground image into the area to be merged to obtain target image data.

[0122] Specifically, when the image data generation system determines the foreground image to be merged into the background image, it can first determine the second length of the foreground image, then determine the width of the foreground image according to a preset foreground image aspect ratio, and then determine the foreground image that needs to be merged into the background image.

[0123] Figure 4 is a flow chart of a second method for generating image data provided in an embodiment of this specification, such as Figure 4 As shown, the schematic diagram includes: Step 402: Obtain a background image to be subjected to data fusion, and perform mask segmentation on the background image to obtain a region to be fused in the background image.

[0124] Step 404: Acquire a template text, and generate a template image fused with the background image based on the template text.

[0125] The template text is randomly generated text that describes the content of the template image.

[0126] Step 406: Obtain a preset style image to be transferred, and input the style image to be transferred and the template image into a pre-trained style transfer model to obtain a foreground image.

[0127] Step 408: Acquire the first center point in the area to be fused and the second center point in the second area.

[0128] The first center point is the center point of the inscribed rectangle in the area to be fused, the second center point is the center point of a preset reference object in the second area, and the second area is a non-fused area in the background image.

[0129] Step 410: Connect the first center point and the second center point to obtain a first line segment, and connect the second center point and the third center point to obtain a second line segment.

[0130] The third center point is a point whose straight-line distance from the second center point meets a preset condition, and the preset condition is determined based on the length value and the width value of the inscribed rectangle.

[0131] Step 412: When the angle between the first line segment and the second line segment is smaller than a preset threshold, determine the third center point as the fusion position.

[0132] Step 414: Obtain a first length of a preset reference object in the second area.

[0133] Step 416: Determine a second length of the foreground image based on the first length and a preset ratio information table.

[0134] The ratio information table pre-stores the length ratio between the preset reference object and the foreground image.

[0135] Step 418: Set the length of the foreground image to the second length.

[0136] Step 420: Based on the determined fusion position, the set foreground image is fused into the background image to obtain the target icon data.

[0137] The embodiment of this specification determines the fusion position when the foreground image is merged into the background image through a preset first algorithm, and merges the foreground image and the background image at the fusion position. In this way, the foreground image and the background image can be merged at a position suitable for fusion, thereby improving the authenticity of image data generation.

[0138] It should be noted that the image data generation method provided in the embodiment of the present application can be executed by an image data generation device or a control module in the image data generation device for executing the image data generation method. In the embodiment of the present application, the image data generation device provided in the embodiment of the present application is described by taking the image data generation method executed by the image data generation device as an example.

[0139] Figure 5 is a schematic diagram of the structure of an image data generating device according to an embodiment of the present invention. Figure 5 As shown, the image data generating device includes: a first segmentation module 502 , a first generation module 504 , a first migration module 506 , and a first fusion module 508 .

[0140] The first segmentation module 502 is used to obtain a background image to be subjected to data fusion, and perform mask segmentation on the background image to obtain a region to be fused in the background image.

[0141] The first generating module 504 is used to obtain a template text, and generate a template image fused with the background image based on the template text, wherein the template text is a randomly generated text describing the content of the template image.

[0142] The first transfer module 506 is used to obtain a preset style image to be transferred, and input the style image to be transferred and the template image into a pre-trained style transfer model to obtain a foreground image.

[0143] The first fusion module 508 is used to fuse the foreground image into the to-be-fused area in the background image, so as to obtain target image data.

[0144] The image data generating device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0145] The image data generating device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0146] The image data generating device provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented in the method embodiment are not described here.

[0147] Based on the same technical concept, an embodiment of the present application further provides an electronic device, which is used to execute the above-mentioned image data generating method. Figure 6 A schematic diagram of the structure of an electronic device for implementing various embodiments of the present application. The electronic device may have relatively large differences due to different configurations or performances, and may include a processor (processor) 602, a communication interface (Communications Interface) 604, a memory (memory) 606, and a communication bus 608, wherein the processor 602, the communication interface 604, and the memory 606 communicate with each other through the communication bus 608. The processor 602 may call a computer program stored in the memory 606 and executable on the processor 602 to perform the following steps: Obtaining a background image to be fused, and performing mask segmentation on the background image to obtain a region to be fused in the background image; Acquire a template text, and generate a template image fused with the background image based on the template text, wherein the template text is a randomly generated text describing the content of the template image; Obtain a preset style image to be transferred, and input the style image to be transferred and the template image into a pre-trained style transfer model to obtain a foreground image; The foreground image is fused into the area to be fused in the background image, thereby obtaining target image data.

[0148] In one implementation, the step of obtaining a background image to be subjected to data fusion and performing mask segmentation on the background image to obtain a region to be fused in the background image includes: Acquire a first picture, and perform mask segmentation on the first picture to obtain a first region and a second region, wherein the first region is a region to be fused in the first picture, and the second region is a non-fused region in the first picture; Obtaining a first length corresponding to each preset reference object in the second area; Determine whether the first image satisfies a preset feasibility condition based on the first area and the second area, where the feasibility condition includes a first condition and / or a second condition, the first condition being that the area ratio of the first area to the first image is greater than a first threshold, and the second condition being that any of the first lengths exists and is greater than a second threshold; If so, it is determined that the first picture is the background picture.

[0149] In one implementation, the step of obtaining a background image to be subjected to data fusion and segmenting the background image to obtain a region to be fused in the background image includes: Acquire a third image and annotation data of the third image, where the annotation data is data determined by annotating a first region and a second region in the third image, where the first region is a region to be fused in the third image, and the second region is a non-fused region in the third image; Inputting the third picture into the image segmentation model to be trained, and obtaining a segmentation result; Iteratively updating the image segmentation model to be trained based on the segmentation result and the labeled data to obtain a pre-trained image segmentation model; The background image is input into the pre-trained image segmentation model to obtain the area to be fused in the background image.

[0150] In one implementation, before obtaining the style image to be transferred and inputting the style image to be transferred and the template image into a pre-trained style transfer model to obtain a foreground image, the method further includes: Acquire a sample foreground picture, where the sample foreground picture is segmented from a picture including the sample foreground picture; Performing distortion and cropping processing on the sample foreground image to obtain a sample style image; Performing text recognition on the sample foreground image, and generating images and texts based on the recognized text to obtain a sample template image; Inputting the sample template image and the sample style image into the style transfer model to be trained to obtain a sample prediction image; Inputting the sample prediction picture and the sample foreground picture into a discriminator to obtain a discrimination result, wherein the discriminator is used to determine whether the style and / or content of the sample prediction picture and the sample foreground picture are the same; Based on the discrimination result, parameters of the style transfer model to be trained are updated to obtain the pre-trained style transfer model.

[0151] In one implementation, the step of fusing the foreground image into the to-be-fused region in the background image to obtain target image data includes: Determine a fusion position in the to-be-fused area based on a preset first algorithm, where the first algorithm is used to determine a position of the foreground image in the background image when the foreground image is fused with the background image; The foreground image is fused into the background image based on the determined fusion position to obtain the target icon data.

[0152] In one implementation, the determining the fusion position in the to-be-fused area based on a preset first algorithm includes: Acquire a first center point in the area to be fused and a second center point in the second area, wherein the first center point is the center point of the inscribed rectangle in the area to be fused, the second center point is the center point of a preset reference object in the second area, and the second area is a non-fused area in the background image; Connecting the first center point and the second center point to obtain a first line segment, and connecting the second center point and the third center point to obtain a second line segment, wherein the third center point is a point whose straight-line distance from the second center point meets a preset condition, and the preset condition is determined based on the length value and the width value of the inscribed rectangle; When the angle between the first line segment and the second line segment is smaller than a preset threshold, the third center point is determined as the fusion position.

[0153] In one implementation, fusing the foreground image into the to-be-fused region in the background image to obtain target image data includes: Performing mask segmentation on the background image to obtain a second area in the background image, where the second area is a non-fused area in the background image; Acquire a first length of a preset reference object in the second area; determining a second length of the foreground image based on the first length and a preset ratio information table, wherein the ratio information table pre-stores a length ratio between the preset reference object and the foreground image; The length of the foreground image is set to the second length, and the set foreground image is merged into the to-be-merged area in the background image, thereby obtaining target image data.

[0154] The specific execution steps can refer to the various steps of the above-mentioned image data generation method embodiment, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0155] It should be noted that the electronic devices in the embodiments of the present application include: servers, terminals, or other devices except terminals.

[0156] The above electronic device structure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured with a display panel in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit includes a touch panel and at least one of other input devices. The touch panel is also called a touch screen. Other input devices may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0157] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory may include a volatile memory or a non-volatile memory, or the memory may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM).

[0158] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor.

[0159] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image data generation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0160] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0161] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image data generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0162] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0163] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0164] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0165] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for generating image data, comprising: Obtaining a background image to be fused, and performing mask segmentation on the background image to obtain a region to be fused in the background image; Acquire a template text, and generate a template image fused with the background image based on the template text, wherein the template text is a randomly generated text describing the content of the template image; Obtain a preset style image to be transferred, and input the style image to be transferred and the template image into a pre-trained style transfer model to obtain a foreground image; The foreground image is fused into the area to be fused in the background image, thereby obtaining target image data.

2. According to the method of claim 1, the step of obtaining a background image to be subjected to data fusion and performing mask segmentation on the background image to obtain a region to be fused in the background image comprises: Acquire a first picture, and perform mask segmentation on the first picture to obtain a first region and a second region, wherein the first region is a region to be fused in the first picture, and the second region is a non-fused region in the first picture; Obtaining a first length corresponding to each preset reference object in the second area; Determine whether the first image satisfies a preset feasibility condition based on the first area and the second area, where the feasibility condition includes a first condition and / or a second condition, the first condition being that the area ratio of the first area to the first image is greater than a first threshold, and the second condition being that any of the first lengths is greater than a second threshold; If so, it is determined that the first picture is the background picture.

3. According to the method of claim 1, the step of obtaining a background image to be subjected to data fusion and segmenting the background image to obtain a region to be fused in the background image comprises: Acquire a third image and annotation data of the third image, where the annotation data is data determined by annotating a first region and a second region in the third image, where the first region is a region to be fused in the third image, and the second region is a non-fused region in the third image; Inputting the third picture into the image segmentation model to be trained, and obtaining a segmentation result; Iteratively updating the image segmentation model to be trained based on the segmentation result and the labeled data to obtain a pre-trained image segmentation model; The background image is input into the pre-trained image segmentation model to obtain the area to be fused in the background image.

4. The method according to claim 1, before obtaining the style image to be transferred and inputting the style image to be transferred and the template image into a pre-trained style transfer model to obtain a foreground image, the method further comprises: Acquire a sample foreground picture, where the sample foreground picture is segmented from a picture including the sample foreground picture; Performing distortion and cropping processing on the sample foreground image to obtain a sample style image; Performing text recognition on the sample foreground image, and generating images and texts based on the recognized text to obtain a sample template image; Inputting the sample template image and the sample style image into the style transfer model to be trained to obtain a sample prediction image; Inputting the sample prediction picture and the sample foreground picture into a discriminator to obtain a discrimination result, wherein the discriminator is used to determine whether the style and / or content of the sample prediction picture and the sample foreground picture are the same; Based on the discrimination result, parameters of the style transfer model to be trained are updated to obtain the pre-trained style transfer model.

5. The method according to claim 1, wherein the step of fusing the foreground image into the to-be-fused area in the background image to obtain target image data comprises: Determine a fusion position in the to-be-fused area based on a preset first algorithm, where the first algorithm is used to determine a position of the foreground image in the background image when the foreground image is fused with the background image; The foreground image is fused into the background image based on the determined fusion position to obtain the target icon data.

6. The method according to claim 5, wherein determining the fusion position in the area to be fused based on a preset first algorithm comprises: Acquire a first center point in the area to be fused and a second center point in the second area, wherein the first center point is the center point of the inscribed rectangle in the area to be fused, the second center point is the center point of a preset reference object in the second area, and the second area is a non-fused area in the background image; Connecting the first center point and the second center point to obtain a first line segment, and connecting the second center point and the third center point to obtain a second line segment, wherein the third center point is a point whose straight-line distance from the second center point meets a preset condition, and the preset condition is determined based on the length value and the width value of the inscribed rectangle; When the angle between the first line segment and the second line segment is smaller than a preset threshold, the third center point is determined as the fusion position.

7. The method according to claim 1, wherein the step of fusing the foreground image into the to-be-fused region in the background image to obtain target image data comprises: Performing mask segmentation on the background image to obtain a second area in the background image, where the second area is a non-fused area in the background image; Acquire a first length of a preset reference object in the second area; determining a second length of the foreground image based on the first length and a preset ratio information table, wherein the ratio information table pre-stores a length ratio between the preset reference object and the foreground image; The length of the foreground image is set to the second length, and the set foreground image is merged into the to-be-merged area in the background image, thereby obtaining target image data.

8. An image data generating device, comprising: A first segmentation module is used to obtain a background image to be subjected to data fusion, and perform mask segmentation on the background image to obtain a region to be fused in the background image; A first generating module is used to obtain a template text, and generate a template image fused with the background image based on the template text, wherein the template text is a randomly generated text describing the content of the template image; A first migration module is used to obtain a preset style image to be migrated, and input the style image to be migrated and the template image into a pre-trained style migration model to obtain a foreground image; The first fusion module is used to fuse the foreground image into the to-be-fused area in the background image, thereby obtaining target image data.

9. A computer device, characterized in that: The device comprises: Processor; and A memory arranged to store computer executable instructions, the executable instructions being configured to be executed by the processor, the executable instructions comprising instructions for performing the steps in the method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is used to store computer-executable instructions, and the executable instructions enable a computer to execute the method according to any one of claims 1 to 7.

Images