Pattern generation method, pattern generation system and object pattern training sample generation method

Abstract

The invention discloses a pattern generation method, a pattern generation system and a generation method of an object pattern training sample. A processing unit loads a program code to execute the pattern generation method. The pattern generation method includes the following steps. An object identification program is carried out on a picture through an object identification model, and then an original object pattern of at least one object on the picture is obtained. The original object pattern is cut along the edge of the object. The original object pattern is stored in a storage unit. Receiving at least one pattern conversion instruction; according to the pattern transformation instruction, a pattern generation model is used for carrying out a pattern transformation program on the original object pattern, and then a transformed object pattern is obtained;

Description

Technical Field

[0001] The present disclosure relates to an image processing method and an image processing system, and more particularly to a pattern generation method, a pattern generation system and a method for generating object pattern training samples. Background Art

[0002] As electronic documents and reports become more and more popular, images will also be used in the content of documents. When users search for images on the Internet, they often cannot find images that fully meet their requirements, so they have to spend a lot of time searching for images, which compresses the precious time of users to write content.

[0003] If users need to spend time editing the searched pictures, it will cause inconvenience to users. Therefore, researchers are working on developing a pattern generation method, hoping to intelligently generate the picture content required by users. Summary of the invention

[0004] The present disclosure is related to a pattern generation method, a pattern generation system and a method for generating object pattern training samples, which utilizes an object recognition model to accurately obtain a certain object pattern close to the user's needs, and utilizes a pattern generation model to perform a pattern transformation process on the object pattern to generate the object pattern that the user really needs. Therefore, the user no longer needs to spend a lot of time searching to obtain the required object pattern.

[0005] According to one aspect of the present disclosure, a pattern generation method is provided. A processing unit loads a program code to execute the pattern generation method. The pattern generation method includes the following steps. An object recognition process is performed on a screen using an object recognition model, thereby obtaining an original object pattern of at least one object on the screen. The original object pattern is cut along the edge of the object. The original object pattern is stored in a storage unit. At least one pattern transformation instruction is received. According to the pattern transformation instruction, a pattern transformation process is performed on the original object pattern using a pattern generation model, thereby obtaining a transformed object pattern.

[0006] According to another aspect of the present disclosure, a pattern generation system is proposed. The pattern generation system includes a picture capture unit, an object recognition model, an input unit, a pattern generation model, a storage unit and a processing unit. The picture capture unit is used to capture a picture. The object recognition model is used to perform an object recognition procedure on the picture, thereby obtaining an original object pattern of at least one object on the picture. The original object pattern is cut along the edge of the object. The input unit is used to receive at least one pattern transformation instruction. The pattern generation model performs a pattern transformation procedure on the original object pattern according to the pattern transformation instruction, thereby obtaining a transformed object pattern. The storage unit is used to store the original object pattern and the transformed object pattern. The processing unit is used to execute the object recognition model and the pattern generation model.

[0007] According to another aspect of the present disclosure, a method for generating object pattern training samples is provided. The method for generating object pattern training samples includes the following steps: Obtain a plurality of screen training samples. Perform an object recognition procedure on each screen training sample using an object recognition model to obtain an object pattern training sample of at least one object on each screen training sample and attribute information of each object pattern training sample. Use the object pattern training samples and the corresponding attribute information to train a pattern generation model, thereby obtaining a trained pattern generation model.

[0008] In order to better understand the above and other aspects of the present disclosure, the following embodiments are specifically described in detail with reference to the accompanying drawings as follows: BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 An example is given to illustrate a pattern conversion process according to an embodiment.

[0010] Figure 2 A block diagram of a pattern generating system according to an embodiment is shown.

[0011] Figure 3 A flow chart of a pattern generating method according to an embodiment is shown.

[0012] Figure 4 Example Figure 3 Each step.

[0013] Figure 5 FIG. 4 is a flow chart of a method for generating object pattern training samples according to an embodiment of the present invention.

[0014] Figure 6 Example Figure 5 Each step.

[0015] Figure 7 The following example illustrates a pattern conversion process according to another embodiment.

[0016] Figure 8 The following example illustrates a pattern conversion process according to another embodiment.

[0017] in,

[0018] 100: pattern generation system;

[0019] 110: Display;

[0020] 120: picture capture unit;

[0021] 130: Object recognition model;

[0022] 140: input unit;

[0023] 150: Pattern generation model;

[0024] 160: storage unit;

[0025] 170: processing unit;

[0026] ATjk, ATmn: attribute information;

[0027] CMt: pattern transformation instruction;

[0028] FMi: picture;

[0029] FMm: picture training sample;

[0030] OBj,OBmn: objects;

[0031] PTj, PT7, PT8: original object pattern;

[0032] PTj', PT7', PT8': transform object pattern;

[0033] PTmn: object pattern training sample;

[0034] S310, S320, S330, S510, S520: steps. DETAILED DESCRIPTION

[0035] Please refer to Figure 1 , which illustrates an example of a pattern conversion procedure according to an embodiment. When a frame FMi contains multiple objects, the object required by the user may be blocked by other objects. Figure 1 As shown, the object OBj required by the user is a table, but part of it is blocked by a cup, so that the user cannot directly capture the complete table image.

[0036] In this embodiment, the required object OBj can be identified and cut along the edge to obtain an original object pattern PTj with defects. Then, the defects of the original object pattern PTj are filled to obtain a complete transformed object pattern PTj'. In some embodiments, the material, color, etc. of the original object pattern PTj can even be changed according to the needs of the user. Therefore, the user no longer needs to spend a lot of time searching to obtain the required object pattern. The object pattern can also be used more efficiently in document files.

[0037] Please refer to Figure 2, which depicts a block diagram of a pattern generation system 100 according to an embodiment. The pattern generation system 100 includes a display 110, a screen capture unit 120, an object recognition model 130, an input unit 140, a pattern generation model 150, a storage unit 160 and a processing unit 170. The screen capture unit 120 is used to perform a screen capture procedure. The object recognition model 130 is used to perform an object recognition procedure. The input unit 140 is used to receive instructions. The pattern generation model 150 is used to perform a pattern conversion procedure. The storage unit 160 is used to store data. The processing unit 170 is used to execute various processing procedures and control procedures. The screen capture unit 120, the object recognition model 130 and / or the pattern generation model 150 are, for example, a circuit, a chip, a circuit board, a storage device storing program codes. The input unit 140 is, for example, a keyboard, a mouse, a touch panel or a similar input / output device.

[0038] The processing unit 170 is, for example, a central processing unit (CPU), or other programmable general-purpose or special-purpose micro control unit (MCU), microprocessor, digital signal processor (DSP), programmable controller, application specific integrated circuit (ASIC), graphics processing unit (GPU), image signal processor (ISP), image processing unit (IPU), arithmetic logic unit (ALU), complex programmable logic device (CPLD), field programmable gate array (FPGA), or other similar components or combinations of the above components.

[0039] The storage unit 160 is, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid state drive (SSD) or similar element or a combination of the above elements, and is used to store multiple modules or various application programs that can be executed by the processing unit 170.

[0040] In this embodiment, the object pattern required by the user can be accurately generated through the object recognition process of the object recognition model 130 and the pattern conversion process of the pattern generation model 150. The operation of the above-mentioned components is described in detail below with a flow chart.

[0041] Please refer to Figure 3 and Figure 4 , Figure 3 A flow chart of a pattern generation method according to an embodiment is shown. Figure 4 Example Figure 3 The pattern generation method includes steps S310 to S330. In step S310, Figure 4 As shown, an object recognition process is performed on the image FMi using the object recognition model 130, thereby obtaining the original object pattern PTj of the object OBj on the image FMi. The object recognition model 130 is, for example, a convolutional neural network model (CNN), a regional convolutional neural network model (R-CNN), or a fully convolutional neural network model (RCN). The original object pattern PTj is cut along the edge of the object OBj. Figure 4 As shown, the original object pattern PTj may be blocked by other objects and thus may have defects. Alternatively, the material or color of the original object pattern PTj may not be what the user needs.

[0042] Next, in step S320, Figure 4 As shown, the input unit 140 receives at least one pattern transformation instruction CMt and provides it to the pattern generation model 150. The pattern transformation instruction CMt is, for example, a color transformation instruction, a material transformation instruction, a defect filling instruction, or a light source direction change instruction. In one embodiment, a graphic user interface can be provided for the user to select and adjust various pattern transformation instructions CMt.

[0043] Then, in step S330, Figure 4As shown, according to at least one pattern transformation instruction CMt, the pattern generation model 150 performs a pattern transformation procedure on the original object pattern PTj, thereby obtaining a transformed object pattern PTj'. The pattern generation model 150 is, for example, a DALL·E neural network or a generative adversarial network (GAN). Figure 4 In the example, the pattern transformation instruction CMt includes a defect filling instruction. In the pattern transformation process, the defect in the upper right corner of the original object pattern PTj is filled. The pattern transformation instruction CMt can also be a color transformation instruction. In the pattern transformation process, the color of the original object pattern PTj is changed.

[0044] Through the above pattern generation method, the object recognition program of the object recognition model 130 can extract the original object pattern PTj. Even if the original object pattern PTj does not completely meet the user's needs, the pattern generation model 150 can accurately generate the transformed object pattern PTj' required by the user through the pattern transformation program.

[0045] The present embodiment must execute the pattern conversion procedure through the pattern generation model 150. However, the training of the pattern generation model 150 must rely on a large number of object pattern training samples, and the color, material and other attribute information corresponding to these object pattern training samples must be recorded before the pattern generation model 150 can be trained. Generally speaking, the object patterns that can be obtained do not additionally record attribute information such as color and material, which makes the training of the pattern generation model 150 difficult. The present embodiment further proposes a method for generating pattern training samples.

[0046] Please refer to Figure 5 and Figure 6 , Figure 5 FIG. 1 is a flow chart showing a method for generating object pattern training samples PTmn according to an embodiment of the present invention. Figure 6 Example Figure 5 The method for generating object pattern training samples PTmn includes steps S510 to S530.

[0047] In step S510, Figure 6 As shown, a plurality of picture training samples FMm are obtained. The picture training samples FMm include one or more objects OBmn. These picture training samples FMm may include the same or different objects OBmn.

[0048] Next, in step S520, Figure 6As shown, the object recognition model 130 is used to perform an object recognition procedure on each image training sample FMm to obtain an object pattern training sample PTmn of the object OBmn on each image training sample FMm and an attribute information ATmn of each object pattern training sample PTmn. The attribute information ATmn can be an object type, a color, a material, and a light source direction. In this embodiment, the user does not need to mark the object type, color, material, and light source direction information for these object pattern training samples PTmn. The attribute information ATmn can be directly marked for each object pattern training sample PTmn through the object recognition model 130. These object pattern training samples PTmn and their corresponding attribute information ATmn will be stored in the storage unit 160.

[0049] Then, in step S530, after having a large number of object pattern training samples PTmn and their corresponding attribute information ATmn, the object pattern training samples PTmn and the attribute information ATmn can be used to train the pattern generation model 150, thereby obtaining the trained pattern generation model 150. Since the pattern generation model 150 is trained using the object pattern training samples PTmn and their corresponding attribute information ATmn, the pattern generation model 150 can change or adjust the attribute information ATmn of the original object pattern PTj to generate the transformed object pattern PTj' required by the user.

[0050] In addition to the above Figure 1 In addition to the exemplary image transformation process, the technology disclosed herein can also implement various other image transformation processes.

[0051] Please refer to Figure 7 , which illustrates an example of a pattern transformation procedure according to another embodiment. The original object pattern PT7 is a pattern with a shadow. The pattern transformation instruction CMt received by the pattern generation model 150 is a light source transformation instruction. The pattern generation model 150 of this embodiment can perform a pattern transformation procedure on the original object pattern PT7, so that the light source direction of the original object pattern PT7 is changed, and then a transformed object pattern PT7' is obtained. The shadow of the transformed object pattern PT7' is different from the shadow of the original object pattern PT7.

[0052] Please refer to Figure 8 , which illustrates an example of a pattern transformation procedure according to another embodiment. The material of the original object pattern PT8 is plastic. The pattern transformation instruction CMt received by the pattern generation model 150 is a material transformation instruction. The pattern generation model 150 can perform a pattern transformation procedure on the original object pattern PT7, so that the material of the original object pattern PTj is changed, thereby obtaining a transformed object pattern PT8'. The material of the transformed object pattern PT8' is different from the material of the original object pattern PT8.

[0053] According to the above embodiment, the object recognition process of the object recognition model 130 can extract the original object pattern PTj. Even if the original object pattern PTj does not completely meet the user's needs, the pattern conversion process of the pattern generation model 150 can accurately generate the transformed object pattern PTj' required by the user.

[0054] In summary, although the present disclosure has been disclosed as above by way of embodiments, it is not intended to limit the present disclosure. A person skilled in the art of the present disclosure may make various changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of the present disclosure shall be determined by the definition of the claims.

Claims

1. A pattern generation method, a processing unit loads a program code to execute the pattern generation method, the pattern generation method comprising: Performing an object recognition process on a picture with an object recognition model, thereby obtaining an original object pattern of at least one object on the picture, wherein the original object pattern is cut along the edge of the object, and the original object pattern is stored in a storage unit; receiving at least one pattern transformation instruction; as well as According to the at least one pattern transformation instruction, a pattern generation model is used to perform a pattern transformation procedure on the original object pattern, thereby obtaining a transformed object pattern.

2. The pattern generating method according to claim 1, characterized in that: The at least one pattern transformation instruction includes a defect filling instruction. In the pattern transformation process, at least one defect of the original object pattern is filled.

3. The pattern generating method according to claim 1, characterized in that: The pattern conversion instruction includes a color conversion instruction. In the pattern conversion process, the color of the original object pattern is changed.

4. The pattern generating method according to claim 1, characterized in that: The pattern transformation instruction includes a material transformation instruction. In the pattern transformation process, the material of the original object pattern is changed.

5. The pattern generating method according to claim 1, characterized in that: The pattern transformation instruction includes a light source direction transformation instruction. In the pattern transformation process, the light source direction of the original object pattern is changed.

6. The pattern generating method according to claim 1, characterized in that: After the object recognition model performs the object recognition process on the image, it also obtains a plurality of attribute information of the original object pattern.

7. The pattern generating method according to claim 6, characterized in that: The attribute information includes an object type, a color, a material, and a light source direction.

8. A pattern generation system, comprising: a picture capturing unit, used for capturing a picture; An object recognition model is used to perform an object recognition process on the image to obtain an original object pattern of at least one object on the image, wherein the original object pattern is cut along the edge of the object; An input unit for receiving at least one pattern transformation instruction; a pattern generation model, which performs a pattern transformation process on the original object pattern according to the at least one pattern transformation instruction, thereby obtaining a transformed object pattern; a storage unit for storing the original object pattern and the transformed object pattern; as well as A processing unit is used to execute the object recognition model and the pattern generation model.

9. The pattern generation system according to claim 8, characterized in that: The at least one pattern transformation instruction includes a defect filling instruction. In the pattern transformation process, at least one defect of the original object pattern is filled.

10. The pattern generation system according to claim 8, characterized in that: The pattern conversion instruction includes a color conversion instruction. In the pattern conversion process, the color of the original object pattern is changed.

11. The pattern generation system according to claim 8, characterized in that: The pattern transformation instruction includes a material transformation instruction. In the pattern transformation process, the material of the original object pattern is changed.

12. The pattern generation system according to claim 8, characterized in that: The pattern transformation instruction includes a light source direction transformation instruction. In the pattern transformation process, the light source direction of the original object pattern is changed.

13. The pattern generation system according to claim 8, characterized in that: After the object recognition model performs the object recognition process on the image, it also obtains a plurality of attribute information of the original object pattern.

14. The pattern generation system according to claim 13, characterized in that: The attribute information includes an object type, a color, a material, and a light source direction.

15. A method for generating object pattern training samples, comprising: Obtain a plurality of picture training samples; Using an object recognition model to perform an object recognition process on each of the frame training samples to obtain an object pattern training sample of at least one object on each of the frame training samples and attribute information of each of the object pattern training samples; and The object pattern training samples and the corresponding attribute information are used to train a pattern generation model, thereby obtaining the trained pattern generation model.

16. The method for generating object pattern training samples as claimed in claim 15, wherein: The attribute information includes an object type, a color, a material, and a light source direction.

Images