Image generation method, device and electronic device

Abstract

The present disclosure provides an image generation method, device and electronic device. The image generation method includes: obtaining the coordinates, foreground attribute values ​​and foreground description information of one or more preset foreground controls in a target area; selecting a foreground material in a material library according to the foreground description information, adjusting the foreground material according to the foreground attribute value, and setting the adjusted foreground material in the target area according to the coordinates; obtaining background description information of a preset background control in the target area, selecting a background material in the material library according to the background description information; generating a target image according to the background material and the adjusted foreground material, and adding a label including the foreground description information to the target image. The embodiment of the present disclosure can automatically generate a large number of different labeled training images in batches.

Description

Background Art

[0002] With the rapid development of image processing technology, how to enhance sample image data is very important in solving the problem of uneven distribution of sample images in actual industrial projects. For example, when performing computer image detection and classification, if the number of sample images of a certain type is small, the image detection and classification model will be trained less on sample images of this type, which will lead to incorrect detection and classification results when performing detection and classification of a related image, that is, reduce the classification accuracy of the model.

[0003] In order to accurately train the image detection classification model, a large number of labeled images are needed as training images, and obtaining sufficient training images is a time-consuming and labor-intensive task. Manually labeling training data is a common means of obtaining labeled data images, but the labor and time costs of this method are huge. In addition, there are technologies in the related art that add noise, distortion, and other processing to labeled images to generate new labeled images, but the new training images generated in this way are prone to overfitting risks, resulting in a large gap between the target objects in the generated images and the real objects corresponding to the labels, and the model cannot be trained to recognize real objects in various normal forms.

[0004] Therefore, how to generate enough real labeled training images in large quantities becomes an important research topic.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The purpose of the present disclosure is to provide an image generation method, device and electronic device, which are used to overcome, at least to a certain extent, the problems of high time cost, high labor cost and poor training image quality in training image enhancement technology caused by the limitations and defects of related technologies.

[0007] According to a first aspect of an embodiment of the present disclosure, there is provided an image generating method, including: obtaining coordinates, foreground attribute values ​​and foreground description information of one or more preset foreground controls in a target area; selecting a foreground material in a material library according to the foreground description information, adjusting the foreground material according to the foreground attribute values, and setting the adjusted foreground material in the target area according to the coordinates; obtaining background description information of preset background controls in the target area, selecting a background material in the material library according to the background description information; generating a target image according to the background material and the adjusted foreground material, and adding a label including the foreground description information to the target image.

[0008] In an exemplary embodiment of the present disclosure, at least one of the preset foreground controls is automatically generated in response to an image generation instruction and / or is manually added, and the preset background controls are automatically generated in response to an image generation instruction or are manually added.

[0009] In an exemplary embodiment of the present disclosure, it also includes: responding to the image generation instruction, determining the foreground type, background type, and foreground quantity range of the target image; generating a first number of the preset foreground controls according to the foreground quantity range, and generating the foreground attribute value, the coordinates, and the foreground description information of each preset foreground control according to the foreground type; generating the background description information of the preset background control according to the background type.

[0010] In an exemplary embodiment of the present disclosure, the foreground attribute value includes at least the size of the preset foreground control, and the generating the foreground attribute value, the coordinates and the foreground description information of each preset foreground control according to the foreground type includes: determining a target foreground type of each preset foreground control according to the foreground type; when the first number is greater than one, determining a size ratio relationship of the multiple preset foreground controls according to the target foreground type of each preset foreground control; determining the sizes and the coordinates of the multiple preset foreground controls according to the size ratio relationship and the size of the target area; generating the foreground description information of each preset foreground control according to the target foreground type of each preset foreground control, generating the foreground attribute value of each preset foreground control according to the size of each preset foreground control, and recording the coordinates of each preset foreground control.

[0011] In an exemplary embodiment of the present disclosure, the image generation instruction includes foreground overlap information, and determining the sizes and coordinates of the multiple preset foreground controls according to the size ratio relationship and the size of the target area includes: when the foreground overlap information shows that none of the preset foreground controls overlap, determining the size of each of the preset foreground controls when a first number of the preset foreground controls are arranged in the target area without overlapping according to the size ratio relationship and the size of the target area; when the foreground overlap information shows that at least two preset foreground controls overlap, determining multiple target preset foreground controls that have an overlapping relationship, and determining the overlap ratio and occlusion relationship between each two of the target preset foreground controls according to the foreground overlap information; and determining the size of each of the preset foreground controls according to the overlap ratio, the occlusion relationship, the size ratio relationship, and the size of the target area.

[0012] In an exemplary embodiment of the present disclosure, the selecting foreground material in the material library according to the foreground description information includes: determining a target foreground type for each of the preset foreground controls according to the foreground description information; determining a target material of the target foreground type or a subtype of the target foreground type in the material library; and randomly selecting the foreground material from the target material.

[0013] In an exemplary embodiment of the present disclosure, generating a target image based on the background material and the adjusted foreground material includes: acquiring image parameters of the background material and one or more foreground materials, the image parameters including at least color information and brightness information; generating target image parameters of the target image based on the image parameters of the background material and one or more foreground materials; setting the one or more foreground materials in front of the background material, and adjusting the image parameters of the background material and the foreground material based on the target image parameters; generating the target image based on the background material and the foreground material after adjusting the image parameters.

[0014] According to a second aspect of an embodiment of the present disclosure, an image generating device is provided, including: a foreground information acquisition module, configured to acquire coordinates, foreground attribute values ​​and foreground description information of one or more preset foreground controls in a target area; a foreground generation module, configured to select a foreground material in a material library according to the foreground description information, adjust the foreground material according to the foreground attribute values, and set the adjusted foreground material in the target area according to the coordinates; a background generation module, configured to acquire background description information of preset background controls in the target area, select background material in the material library according to the background description information; and an image fusion module, configured to generate a target image according to the background material and the adjusted foreground material, and add a label including the foreground description information to the target image.

[0015] According to a third aspect of the present disclosure, there is provided an electronic device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute any one of the methods described above based on instructions stored in the memory.

[0016] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a program is stored, and when the program is executed by a processor, the image generating method as described in any one of the above items is implemented.

[0017] The disclosed embodiment automatically identifies information of preset foreground controls and information of preset background controls to automatically generate a target image, and can quickly batch generate labeled model training images based on a large number of automatically or manually generated preset foreground controls and preset background controls, thereby improving the generation efficiency of training images. In addition, since the target image is generated based on the foreground material and the background material, the problem in the related art of enhancing the training samples by deforming and distorting the existing images, which results in a large difference between the training images and the images commonly seen in the actual application of the model, can be overcome, and the recognition accuracy of the model can be more effectively improved.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0020] Figure 1 is a flowchart of an image generating method in an exemplary embodiment of the present disclosure.

[0021] Figure 2 It is a flowchart of automatically generating a preset foreground control in one embodiment of the present disclosure.

[0022] Figure 3 It is a sub-flowchart of step S22 in one embodiment of the present disclosure.

[0023] Figure 4 It is a sub-flowchart of step S223 in one embodiment of the present disclosure.

[0024] Figure 5 It is a schematic diagram of manually set preset foreground controls and preset background controls in one embodiment of the present disclosure.

[0025] Figure 6 is a flowchart of step S12 in one embodiment of the present disclosure.

[0026] Figure 7 is a flowchart of step S14 in one embodiment of the present disclosure.

[0027] Figure 8 It is a schematic diagram of a process of processing manually generated preset foreground controls and preset background controls in one embodiment of the present disclosure.

[0028] Fig. 9is a block diagram of an image generating apparatus in an exemplary embodiment of the present disclosure.

[0029] Fig.10 is a block diagram of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0031] In addition, the accompanying drawings are only schematic diagrams of the present disclosure, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0032] The exemplary embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0033] Figure 1 is a flowchart of an image generating method in an exemplary embodiment of the present disclosure.

[0034] refer to Figure 1 , the image generation method 100 may include:

[0035] Step S11, obtaining coordinates, foreground attribute values ​​and foreground description information of one or more preset foreground controls in the target area;

[0036] Step S12, selecting a foreground material in a material library according to the foreground description information, adjusting the foreground material according to the foreground attribute value, and setting the adjusted foreground material in the target area according to the coordinates;

[0037] Step S13, obtaining background description information of a preset background control in the target area, and selecting a background material in the material library according to the background description information;

[0038] Step S14, generating a target image according to the background material and the adjusted foreground material, and adding a label including the foreground description information to the target image.

[0039] The disclosed embodiment automatically identifies information of preset foreground controls and information of preset background controls to automatically generate a target image, and can quickly batch generate labeled model training images based on a large number of automatically or manually generated preset foreground controls and preset background controls, thereby improving the generation efficiency of training images. In addition, since the target image is generated based on the foreground material and the background material, the problem in the related art of enhancing the training samples by deforming and distorting the existing images, which results in a large difference between the training images and the images commonly seen in the actual application of the model, can be overcome, and the recognition accuracy of the model can be more effectively improved.

[0040] Next, each step of the image generating method 100 is described in detail.

[0041] In step S11, the coordinates, foreground attribute values ​​and foreground description information of one or more preset foreground controls in the target area are obtained.

[0042] In the disclosed embodiment, the preset foreground control may be automatically randomly generated or manually added.

[0043] Figure 2 It is a flowchart of automatically generating a preset foreground control in one embodiment of the present disclosure.

[0044] refer to Figure 2 In one embodiment, the method 200 for automatically generating a preset foreground control may include:

[0045] Step S21, responding to the image generation instruction, determining the foreground type, background type, and foreground quantity range of the target image;

[0046] Step S22, generating a first number of the preset foreground controls according to the foreground quantity range, and generating the foreground attribute value, the coordinates and the foreground description information of each of the preset foreground controls according to the foreground type;

[0047] Step S23: generating the background description information of the preset background control according to the background type.

[0048] In the embodiment of the present disclosure, the number of target images corresponding to an image generation instruction can be one or more, or multiple target images of one type, or multiple target images of multiple types. Figure 2 In the illustrated embodiment, a target image is taken as an example for description.

[0049] In step S21, the foreground category of the target image can be a single category or multiple categories. For example, if you want to generate a target image for training a data classification model of cats, you can set all the foreground categories of all target images to cats. Alternatively, you can add other foreground categories as interference items to train the data classification model's ability to recognize cats among multiple objects. For example, set the foreground categories of the target image to four categories: cats, fruits, canines, and reptiles, and then generate a target image containing interference items.

[0050] The foreground quantity range of each target image can be an integer greater than or equal to 1. For example, the foreground quantity range can be set to 1, and the corresponding number of foreground types can be 1; or, the foreground quantity range can be set to 5 to 15, and the foreground types can be set to five types: A, B, C, D, and E. Among them, different foreground types can be subordinate categories of the same major category, or different major categories. For example, the five types A, B, C, D, and E can be set to five dog types such as Husky, Alaska, Labrador, Border Collie, and Schnauzer, or two canines, three felines, and the like. The foreground type and foreground quantity range of each target image can be set by a technician in this field according to the training purpose, and the present disclosure does not impose any special restrictions on this.

[0051] In order to improve the practical applicability of model training, the background types may include scenes and backgrounds that match the foreground types. For example, if the foreground type is determined to be a dog according to the training objective, the background may be a home, a shopping mall, a subway, a street, a park, and other scenes where the foreground type may appear. If certain foreground types exist as interference items, the background type may be set according to the main training objective. For example, if the training objective is a domestic cat and the interference item is a large cat, the background type may still be set to a scene where a domestic cat appears, rather than a grassland or forest where large cats are located. In some embodiments, for other training purposes, the background type may also be set to a solid color, a geometric pattern, and the like, and the present disclosure does not impose any particular restrictions on this.

[0052] The benchmarks for determining the foreground type, background type, and foreground quantity range of the target image can be generated based on manually input setting information, or can be randomly determined based on a preset correspondence between the training target and the foreground type or background type. For example, cats can be preset to automatically correspond to several subclasses, and each subclass corresponds to several subclasses. No matter what level the training target is set to, the corresponding type of the same level or the corresponding subclass of the sublevel can be found, thereby achieving random matching of foreground types of the same level or sublevel.

[0053] In step S22, a first number may be randomly generated according to the foreground number range as the number of preset foreground controls. The foreground attribute may at least include the size of the foreground material. The size is, for example, a rectangular box corresponding to the maximum boundary of the foreground material, or the maximum height and maximum width of the foreground material.

[0054] Figure 3 It is a sub-flowchart of step S22 in one embodiment of the present disclosure.

[0055] refer to Figure 3 In one embodiment, the foreground attribute value at least includes the size of the preset foreground control, and step S22 may include:

[0056] Step S221, determining a target foreground type of each of the preset foreground controls according to the foreground type;

[0057] Step S222, when the first number is greater than one, determining the size ratio relationship of the plurality of preset foreground controls according to the target foreground type of each of the preset foreground controls;

[0058] Step S223, determining the sizes and coordinates of the plurality of preset foreground controls according to the size ratio relationship and the size of the target area;

[0059] Step S224, generating the foreground description information of each preset foreground control according to the target foreground type of each preset foreground control, generating the foreground attribute value of each preset foreground control according to the size of each preset foreground control, and recording the coordinates of each preset foreground control.

[0060] exist Figure 3In the illustrated embodiment, the target foreground category can be either the foreground category itself or a subcategory of the foreground category selected according to the image generation instruction. For example, when the foreground category is a cat and the first number of preset foreground controls is 3, three target foreground categories of cat, lion, and tiger can be set for the three preset foreground controls. Then, the size ratio of each preset foreground control is determined according to the preset size ratio relationship of each cat. For example, the conventional size ratio of a cat, a lion, and a tiger is 1:20:23 (the numbers are only examples), and the conventional size ratio can be directly applied to limit the size ratio of the preset foreground control to increase authenticity, or the size ratio of the three preset foreground controls can be set to 1:1:1 for interference purposes.

[0061] Then, according to the size of the target area, the size of each preset foreground control is determined when the first number of preset foreground controls can be completely accommodated in the target area. In one embodiment, the preset foreground controls may have overlapping relationships, and the overlapping relationship may exist between any two or more preset foreground controls. The overlapping relationship and the overlapping ratio may be determined by the image generation instruction.

[0062] Figure 4 It is a sub-flowchart of step S223 in one embodiment of the present disclosure.

[0063] refer to Figure 4 In one embodiment, the image generation instruction includes foreground overlap information, and step S223 includes:

[0064] Step S2231, when the foreground overlap information indicates that none of the preset foreground controls overlap, determining, according to the size ratio relationship and the size of the target area, the size of each of the preset foreground controls when a first number of the preset foreground controls are arranged in the target area without overlapping;

[0065] Step S2232, when the foreground overlap information indicates that at least two preset foreground controls overlap, multiple target preset foreground controls having an overlapping relationship are determined, and the overlap ratio and occlusion relationship between every two of the target preset foreground controls are determined according to the foreground overlap information;

[0066] Step S2233, determining the size of each of the preset foreground controls according to the overlap ratio, the occlusion relationship, the size ratio relationship and the size of the target area.

[0067] exist Figure 4In the illustrated embodiment, if none of them overlap, the preset foreground control with the largest size ratio among the preset foreground controls is first set as the default size, and the sizes of other preset foreground controls are determined according to the size ratio relationship. Then the coordinates of the largest preset foreground control are set, and the coordinates of other preset foreground controls are set in sequence, so that each preset foreground control does not overlap in the target area. If this cannot be achieved, the size of the largest preset foreground control is reduced according to the set ratio or set value, the sizes of other preset foreground controls are re-determined according to the size ratio relationship, and the coordinates of all preset foreground controls are re-determined until all preset foreground controls can be accommodated in the target area without overlapping.

[0068] If the foreground overlap information shows that at least two preset foreground controls overlap, first randomly determine the target preset foreground controls that have an overlapping relationship, and then determine the overlap ratio and occlusion relationship between each two target preset foreground controls among these target preset foreground controls that have an overlapping relationship. If the foreground overlap information specifies the overlap ratio, it can be set according to the overlap ratio. If the foreground overlap information does not specify the overlap ratio, the overlap ratio is randomly determined within a preset range, and the preset range is, for example, 30% to 70%. The overlap ratio can be set, for example, as the ratio of the occluded area of ​​the occluded target preset foreground control to the total size area of ​​the target preset foreground control (i.e., width*height).

[0069] The occlusion relationship can be determined randomly or according to the setting of the image generation instruction. For example, when there are three target preset foreground controls A, B, and C, and the foreground overlap information is an overlap of 40%, A and B can be randomly set to overlap 40%, and A occludes 40% of the area within the size of B, or A and C can be set to overlap at the same time, and A occludes 40% of the area within the size of C. The overlapping relationship can also be generated in three or more preset foreground controls, and the specific setting instructions and overlapping ratio relationship can be set by technicians in this field according to the training objectives.

[0070] After determining the overlap ratio and occlusion relationship of the preset foreground controls, the preset foreground control with the largest size ratio among the preset foreground controls can be set as the default size, and the sizes of other preset foreground controls can be determined according to the size ratio relationship. Then, all preset foreground controls are arranged according to the overlap ratio and occlusion relationship, so that each preset foreground control can be generated in the target area while satisfying the above overlap ratio and occlusion relationship. The process of determining the preset foreground controls is, for example, a random process. If it is ultimately impossible to achieve through calculation, the size of the largest preset foreground control is reduced according to the set ratio or set value, the sizes of other preset foreground controls are re-determined according to the size ratio relationship, and the coordinates of all preset foreground controls are re-determined until all preset foreground controls can be accommodated in the target area according to the above overlap ratio and occlusion relationship.

[0071] In addition to the mutual occlusion between the preset foreground controls, other information such as only a portion of the preset foreground controls can also be set to be exposed in the target area. In the embodiment of the present disclosure, the image generation instruction can also add other restrictions on the preset foreground controls. In this case, the method 200 can generate the preset foreground controls based on these settings. The present disclosure does not impose any special restrictions on the restrictions on the preset foreground controls.

[0072] In step S224, foreground description information of each preset foreground control is generated according to the target foreground type of each preset foreground control, foreground attribute values ​​of each preset foreground control are generated according to the size of each preset foreground control, and the coordinates of each preset foreground control are recorded. The foreground description information may also record other restriction conditions from the image generation instruction, and the foreground attribute values ​​may also include other information of the preset foreground control, but the present disclosure is not limited thereto.

[0073] After the preset foreground control is generated, a preset background control can be generated in step S23, and the background description information of the preset background control can be recorded according to the background type. In some embodiments, a preset background control has only one background type. In other embodiments, a preset background control may have multiple background types. For example, in a Christmas scene, there may be four background types: snow, night, stars, and moon. The relative position relationship of these four background types can be recorded in the background description information. The relative position relationship can be determined according to the preset corresponding relationship (for example, snow is below the picture, blue sky, white clouds and night are above the picture, and white clouds can only appear in the area where the blue sky is located), or it can be determined randomly (for example, white clouds are set to appear only in the blue sky, but the specific position of white clouds can be randomly determined). There can be multiple automatic setting schemes for the number of background types and the relative position relationship, and the present disclosure does not impose any special restrictions on this.

[0074] In other embodiments of the present disclosure, preset foreground controls and preset background controls may also be manually set in the target area. For example, a rectangular frame of a preset foreground control is manually drawn in the target area by dragging, inputting a size, etc., and foreground description information such as the target foreground type, overlapping relationship, and occlusion relationship of the preset foreground control is set and adjusted. Then, a preset background control is manually set, and background description information of the preset background control is input.

[0075] In one embodiment, the process of manually generating preset foreground controls and preset background controls can, for example, be for the annotator to lay out on the drawing board image, draw a rectangular label of any size at the corresponding position, write the foreground material name and foreground attribute value (such as rotation angle, etc.) in the rectangular label, and write the name and attributes of the background or scene at any position without a label.

[0076] Figure 5It is a schematic diagram of manually set preset foreground controls and preset background controls in one embodiment of the present disclosure.

[0077] refer to Figure 5 In the target area 500, there are multiple preset foreground controls 51 and preset background controls 52. The multiple preset foreground controls 51 are embodied as rectangular boxes, the size of the rectangular boxes is the size of the preset foreground controls 51, and the coordinates of the center point of the rectangular boxes are the coordinates of the preset foreground controls 51. Figure 5 In the illustrated embodiment, multiple preset foreground controls 51 do not overlap, and text information such as foreground type and foreground description information is recorded thereon. In the preset background control 52, background description information such as background type and attributes is recorded in text form, wherein the attributes can be used to record material restriction information such as resolution of the background description information and material selection restrictions that cannot be reflected by the background type.

[0078] After automatically or manually generating the preset foreground control and the preset background control, the corresponding relationship between the target image and the preset foreground control and the preset background control can be saved, and the preset foreground control and the preset background control can be automatically parsed by a subsequent target image generation method to generate a target image.

[0079] In step S12, a foreground material is selected from a material library according to the foreground description information, the foreground material is adjusted according to the foreground attribute value, and the adjusted foreground material is set in the target area according to the coordinates.

[0080] The material library can include pictures corresponding to multiple foreground and background types. For example, in the subdirectory of Cats in the material library, pictures of cats in various shapes and colors can be stored as picture materials; in the white snow directory of the material library, pictures of snow accumulation and falling snow in various thicknesses, colors, and shapes can be stored as picture materials, etc. Pictures in the material library can be used as both foreground and background materials, and the material library itself does not restrict the use scenarios of the materials.

[0081] Figure 6 is a flowchart of step S12 in one embodiment of the present disclosure.

[0082] refer to Figure 6 In one embodiment, step S12 may include:

[0083] Step S121, determining a target foreground type of each of the preset foreground controls according to the foreground description information;

[0084] Step S122, determining a target material of the target foreground category or a subcategory of the target foreground category in the material library;

[0085] Step S123: randomly selecting the foreground material from the target material.

[0086] exist Figure 6 In the illustrated embodiment, whether the preset foreground control is automatically generated or manually generated, it is saved in the same format. The target foreground type of the preset foreground control can be parsed by text recognition or the like, and it may be necessary to parse the constraints of other foreground description information at the same time. Then, according to the parsed foreground description information, a material picture is randomly selected from the corresponding directory of the material library. For example, when the target foreground type is a dog, it can be determined that there are a variety of dog sub-types under the "dog" directory in the material library, such as golden retrievers, Labrador dogs, etc., and there may also be a variety of labels such as "running" and "eating". All pictures under the "dog" directory can be set as target materials, pictures of sub-types of golden retrievers and Labrador dogs can be set as target materials, and material pictures under the "dog-running" label can also be used as target materials. Which target material is specifically selected can be determined by the foreground description information of the preset foreground control, or it can be randomly determined when selecting the material. After determining the target material, a foreground material can be randomly selected for the current preset foreground control in the target material.

[0087] Then, the foreground material is adjusted according to the foreground attribute value of the preset foreground control. The above steps are repeated until all preset foreground controls are set. During this period, the level of the foreground material can be adjusted according to the occlusion relationship in the foreground description information in the preset foreground control.

[0088] In step S13, background description information of a preset background control in the target area is obtained, and background material is selected from the material library according to the background description information.

[0089] The steps of selecting background material are the same as those of selecting foreground material. The background type is determined according to the background description information, and the background material is selected from the material library according to the background type. When there are multiple background types, multiple background materials are arranged in the size area of ​​the preset background control according to the relative position relationship or coordinates of the background materials corresponding to each background type recorded in the background description information. In general, the size of the preset background control can be equal to the size of the target area, and there is no need to adjust the size of the preset background control. However, in some cases, such as when the preset background control only includes one background type, that is, only corresponds to one background material, when the default size of the background material is larger or smaller than the target area, the size of the background material can be adjusted so that the maximum size of the largest background material or the single background material can be equal to the width or height of the target area, that is, try to ensure that the background material can cover or fill the target area or the background material combination can cover or fill the target area. It should be noted that the level of the background material needs to be set below the foreground material.

[0090] In step S14, a target image is generated according to the background material and the adjusted foreground material, and a label including the foreground description information is added to the target image.

[0091] In order to prevent the fusion of the background material and the foreground material from being abrupt, in the embodiment of the present disclosure, it is necessary to centrally adjust the background material and the foreground material.

[0092] Figure 7 is a flowchart of step S14 in one embodiment of the present disclosure.

[0093] refer to Figure 7 In one embodiment, step S14 may include:

[0094] Step S141, obtaining image parameters of the background material and one or more foreground materials, wherein the image parameters at least include color information and brightness information;

[0095] Step S142, generating target image parameters of the target image according to the image parameters of the background material and one or more foreground materials;

[0096] Step S143, setting the one or more foreground materials in front of the background material, and adjusting the image parameters of the background material and the foreground material according to the target image parameters;

[0097] Step S144, generating the target image according to the background material and the foreground material after adjusting the image parameters.

[0098] exist Figure 7 In the illustrated embodiment, the process of generating the target image parameters of the target image according to the image parameters of the background material and the foreground material can, for example, extract the brightness information of each material image. Y represents the brightness of the material image, and R / G / B represents the pixel values ​​of the three channels of the material image. First, the brightness of each material image is calculated according to formula (1):

[0099] Y=(0.299*R)+(0.587*G)+(0.114*B) (1)

[0100] Then, the brightness of the target image is obtained according to the brightness of each material image (whether it is a foreground material or a background material), and Y (after fusion) represents the brightness of the fused target image, and N represents the number of preset foreground controls, that is, the first number, which is:

[0101]

[0102] And adjust the color of each material image according to the brightness of the fused target image:

[0103]

[0104] Then, a target image is generated according to each foreground material and background material, and a label of the target image is generated according to the foreground description information of the foreground material. It should be noted that the foreground description information of the preset foreground control can record which preset foreground control (i.e., the training target) whose label is set as the label of the target image, and it is not necessary to set the labels of all preset foreground controls as the label of the target image.

[0105] When the preset foreground image and the preset background image are both manually generated, the method of the embodiment of the present disclosure can be briefly summarized as follows:

[0106] 1. The image drawn by the annotator is first processed by image processing technology (such as OPENCV technology) to find the position of the rectangle, save the rectangular image information (rectangular coordinates, rectangular image data, etc.), and clear the rectangular image on the drawing board;

[0107] 2. Use image text recognition technology (such as OCR technology) to identify the classification text (material type) and attribute text from the image without the rectangular label and the saved rectangular label image;

[0108] 3. According to the classification text (material type), randomly select a material image from the corresponding category in the material library. If the attribute has classification requirements, continue to search for more detailed categories from the image library and randomly select a material image from them;

[0109] 4. Use image processing technology (such as OPENCV technology) to process the material images according to the requirements of the attributes;

[0110] 5. Using image fusion technology, the background material and the rectangular label image (foreground material) are fused according to the layout embodied by the rectangular frame;

[0111] 6. Randomly select other pictures and repeat steps 1 to 5 above to generate a large number of new labeled training samples.

[0112] Figure 8 It is a schematic diagram of a process of processing manually generated preset foreground controls and preset background controls in one embodiment of the present disclosure.

[0113] refer to Figure 8In step S81, the smart tags are manually deployed, and multiple preset foreground controls 51 and preset background controls 52 are added. In step S82, the preset foreground controls 51 and preset background controls 52 are segmented, and the foreground material and background material are determined by text recognition and querying the material library. In step S83, the foreground material 81 and the background material 82 are adjusted. In step S84, the images are fused, and the color and brightness of each foreground material 81 and each background material 82 are adjusted, and the color and brightness of the entire target image are adjusted to generate the target image.

[0114] By using image text recognition technology to identify the text in the label, automatically finding the image dataset corresponding to the text, and then randomly selecting images and inserting them into the location of the label, the diversity of image data can be enhanced. This is particularly suitable for scenarios where datasets are difficult to obtain or collect, and can improve the accuracy of model training such as image recognition.

[0115] Through the method of the embodiment of the present disclosure, target images carrying labels of training targets and images of training targets can be automatically generated in large quantities. Compared with the inefficiency of using manual annotation to produce training images in the prior art, and the method of using geometric operations such as flipping, rotation, etc. and color transformations such as noise and blur to perform single sample image data enhancement to generate training images, the embodiment of the present disclosure can greatly improve the generation speed and number of training images while effectively improving the quality of training images, avoiding unqualified training images from being unable to accurately train the model to recognize normal images. In addition, by generating training images through rich material pictures, it is also possible to avoid the problems of too few material types and single training content in related image enhancement technologies, and improve the problem of uneven distribution of sample images.

[0116] Corresponding to the above method embodiments, the present disclosure also provides an image generating device, which can be used to execute the above method embodiments.

[0117] Fig. 9 is a block diagram of an image generating apparatus in an exemplary embodiment of the present disclosure.

[0118] refer to Fig. 9 , the image generating device 900 may include:

[0119] A foreground information acquisition module 91, configured to acquire coordinates, foreground attribute values ​​and foreground description information of one or more preset foreground controls in a target area;

[0120] A foreground generation module 92, configured to select a foreground material from a material library according to the foreground description information, adjust the foreground material according to the foreground attribute value, and set the adjusted foreground material in the target area according to the coordinates;

[0121] A background generation module 93, configured to obtain background description information of a preset background control in the target area, and select a background material in the material library according to the background description information;

[0122] The image fusion module 94 is configured to generate a target image according to the background material and the adjusted foreground material, and to add a label including the foreground description information to the target image.

[0123] In an exemplary embodiment of the present disclosure, at least one of the preset foreground controls is automatically generated in response to an image generation instruction and / or is manually added, and the preset background controls are automatically generated in response to an image generation instruction or are manually added.

[0124] In an exemplary embodiment of the present disclosure, it also includes a control generation module 95, and the control generation module 95 is configured to: respond to the image generation instruction, determine the foreground type, background type, and foreground quantity range of the target image; generate a first number of the preset foreground controls according to the foreground quantity range, and generate the foreground attribute value, the coordinates, and the foreground description information of each preset foreground control according to the foreground type; generate the background description information of the preset background control according to the background type.

[0125] In an exemplary embodiment of the present disclosure, the foreground attribute value includes at least the size of the preset foreground control, and the control generation module 95 is configured to: determine the target foreground type of each of the preset foreground controls according to the foreground type; when the first number is greater than one, determine the size ratio relationship of the multiple preset foreground controls according to the target foreground type of each of the preset foreground controls; determine the sizes and coordinates of the multiple preset foreground controls according to the size ratio relationship and the size of the target area; generate the foreground description information of each of the preset foreground controls according to the target foreground type of each of the preset foreground controls, generate the foreground attribute value of each of the preset foreground controls according to the size of each of the preset foreground controls, and record the coordinates of each of the preset foreground controls.

[0126] In an exemplary embodiment of the present disclosure, the image generation instruction includes foreground overlap information, and the control generation module 95 is configured to: when the foreground overlap information indicates that none of the preset foreground controls overlap, determine the size of each of the preset foreground controls when a first number of the preset foreground controls are arranged in the target area without overlapping according to the size ratio relationship and the size of the target area; when the foreground overlap information indicates that at least two preset foreground controls overlap, determine a plurality of target preset foreground controls that have an overlapping relationship, and determine the overlap ratio and occlusion relationship between each two of the target preset foreground controls according to the foreground overlap information; and determine the size of each of the preset foreground controls according to the overlap ratio, the occlusion relationship, the size ratio relationship, and the size of the target area.

[0127] In an exemplary embodiment of the present disclosure, the foreground generation module 92 is configured to: determine the target foreground type of each of the preset foreground controls according to the foreground description information; determine the target material of the target foreground type or a subtype of the target foreground type in the material library; and randomly select the foreground material from the target material.

[0128] In an exemplary embodiment of the present disclosure, the image fusion module 94 is configured to: obtain image parameters of the background material and one or more foreground materials, the image parameters including at least color information and brightness information; generate target image parameters of the target image according to the image parameters of the background material and one or more foreground materials; set the one or more foreground materials in front of the background material, and adjust the image parameters of the background material and the foreground material according to the target image parameters; generate the target image according to the background material and the foreground material after adjusting the image parameters.

[0129] Since the functions of the device 900 have been described in detail in the corresponding method embodiments, the present disclosure will not elaborate on them here.

[0130] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0131] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0132] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0133] Refer to the following Fig.10 The electronic device 1000 according to this embodiment of the present invention will be described. Fig.10 The electronic device 1000 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0134] like Fig.10 As shown, the electronic device 1000 is in the form of a general computing device. The components of the electronic device 1000 may include but are not limited to: the at least one processing unit 1010, the at least one storage unit 1020, and a bus 1030 connecting different system components (including the storage unit 1020 and the processing unit 1010).

[0135] The storage unit stores program codes, which can be executed by the processing unit 1010, so that the processing unit 1010 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 1010 can perform the method shown in the embodiment of the present disclosure.

[0136] The storage unit 1020 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 10201 and / or a cache storage unit 10202 , and may further include a read-only storage unit (ROM) 10203 .

[0137] The storage unit 1020 may also include a program / utility 10204 having a set (at least one) of program modules 10205, such program modules 10205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0138] Bus 1030 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0139] The electronic device 1000 may also communicate with one or more external devices 1100 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1000, and / or may communicate with any device that enables the electronic device 1000 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1050. Furthermore, the electronic device 1000 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1060. As shown, the network adapter 1060 communicates with other modules of the electronic device 1000 via a bus 1030. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0140] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0141] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0142] The program product for implementing the above method according to an embodiment of the present invention can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0143] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0144] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0145] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0146] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0147] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0148] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are to be considered exemplary only, and the true scope and concept of the present disclosure are indicated by the claims.

Claims

1. An image generation method, characterized in that: include: Acquire coordinates, foreground attribute values, and foreground description information of one or more preset foreground controls in the target area, wherein the foreground attribute values ​​at least include the size of the preset foreground controls; selecting a foreground material in a material library according to the foreground description information, adjusting the foreground material according to the foreground attribute value, and setting the adjusted foreground material in the target area according to the coordinates; Acquire background description information of a preset background control in the target area, and select a background material from the material library according to the background description information; Generate a target image according to the background material and the adjusted foreground material, and add a label including the foreground description information to the target image; Wherein, at least one of the preset foreground controls is automatically generated in response to an image generation instruction, and automatically generating the preset foreground control includes: In response to the image generation instruction, determining the foreground type and foreground quantity range of the target image; Generate a first number of the preset foreground controls according to the foreground quantity range, and generate the foreground attribute value, the coordinates and the foreground description information of each of the preset foreground controls according to the foreground type; Wherein, generating the foreground attribute value, the coordinates and the foreground description information of each preset foreground control according to the foreground type includes: Determining a target foreground type for each of the preset foreground controls according to the foreground type; When the first number is greater than one, determining the size ratio relationship of the plurality of preset foreground controls according to the target foreground type of each of the preset foreground controls; Determining the sizes and coordinates of the plurality of preset foreground controls according to the size ratio relationship and the size of the target area; The foreground description information of each preset foreground control is generated according to the target foreground type of each preset foreground control, the foreground attribute value of each preset foreground control is generated according to the size of each preset foreground control, and the coordinates of each preset foreground control are recorded.

2. The image generation method according to claim 1, characterized in that: At least one of the preset foreground controls is manually added, and the preset background controls are automatically generated in response to an image generation instruction or are manually added.

3. The image generation method according to claim 2, characterized in that: Also includes: In response to the image generation instruction, determining a background type of a target image; The background description information of the preset background control is generated according to the background type.

4. The image generation method according to claim 1, wherein: The image generation instruction includes foreground overlap information, and determining the sizes and coordinates of the plurality of preset foreground controls according to the size ratio relationship and the size of the target area includes: When the foreground overlap information indicates that none of the preset foreground controls overlap, determining, according to the size ratio relationship and the size of the target area, the size of each of the preset foreground controls when the first number of the preset foreground controls are arranged in the target area without overlapping; When the foreground overlap information indicates that at least two preset foreground controls overlap, a plurality of target preset foreground controls having an overlapping relationship are determined, and an overlap ratio and an occlusion relationship between every two of the target preset foreground controls are determined according to the foreground overlap information; The size of each of the preset foreground controls is determined according to the overlap ratio, the occlusion relationship, the size ratio relationship, and the size of the target area.

5. The image generation method according to claim 1, characterized in that: The selecting a foreground material from a material library according to the foreground description information comprises: Determine the target foreground type of each of the preset foreground controls according to the foreground description information; Determine, in the material library, a target material of the target foreground category or a subcategory of the target foreground category; The foreground material is randomly selected from the target material.

6. The image generation method according to claim 1, wherein: Generating a target image according to the background material and the adjusted foreground material comprises: Acquire image parameters of the background material and one or more foreground materials, wherein the image parameters at least include color information and brightness information; Generate target image parameters of the target image according to the image parameters of the background material and one or more of the foreground materials; The one or more foreground materials are arranged in front of the background material, and image parameters of the background material and the foreground material are adjusted according to the target image parameters; The target image is generated according to the background material and the foreground material after adjusting the image parameters.

7. An image generating device, characterized in that: include: A foreground information acquisition module, configured to acquire coordinates, foreground attribute values ​​and foreground description information of one or more preset foreground controls in the target area, wherein the foreground attribute values ​​at least include the size of the preset foreground controls; A foreground generation module, configured to select a foreground material in a material library according to the foreground description information, adjust the foreground material according to the foreground attribute value, and set the adjusted foreground material in the target area according to the coordinates; A background generation module, configured to obtain background description information of a preset background control in the target area, and select a background material in the material library according to the background description information; An image fusion module, configured to generate a target image according to the background material and the adjusted foreground material, and add a label including the foreground description information to the target image; Wherein, at least one of the preset foreground controls is automatically generated in response to an image generation instruction, and automatically generating the preset foreground control includes: determining the foreground type and foreground quantity range of the target image in response to the image generation instruction; generating a first number of the preset foreground controls according to the foreground quantity range, and generating the foreground attribute value, the coordinates and the foreground description information of each preset foreground control according to the foreground type; Wherein, generating the foreground attribute value, the coordinates and the foreground description information of each preset foreground control according to the foreground type includes: Determining a target foreground type for each of the preset foreground controls according to the foreground type; When the first number is greater than one, determining the size ratio relationship of the plurality of preset foreground controls according to the target foreground type of each of the preset foreground controls; Determining the sizes and coordinates of the plurality of preset foreground controls according to the size ratio relationship and the size of the target area; The foreground description information of each preset foreground control is generated according to the target foreground type of each preset foreground control, the foreground attribute value of each preset foreground control is generated according to the size of each preset foreground control, and the coordinates of each preset foreground control are recorded.

8. An electronic device, characterized in that: include: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the image generating method according to any one of claims 1 to 6 based on instructions stored in the memory.

9. A computer-readable storage medium having a program stored thereon, wherein when the program is executed by a processor, the image generating method according to any one of claims 1 to 6 is implemented.

Images