Recognition system and method for augmenting and training of its data

Abstract

A recognition system and a method for augmenting and training of a dataset thereof. The method for augmenting and training of a dataset of the recognition system comprises the following steps. A plurality of dataset images are obtained. Each dataset image comprises an object image. A plurality of environment images are obtained. The object image is segmented from each dataset image. A plurality of dataset parameters are set. Based on the object image and the environment image, augmentation of the plurality of dataset images is performed according to the dataset parameters to increase the number of the plurality of dataset images. A recognition model is trained using the plurality of dataset images.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a recognition system and a method for augmenting and training of dataset. BACKGROUND

[0002] With the development of artificial intelligence, various objects can be recognized by a recognition system so that the objects can be introduced into, for example, unmanned vending fields. However, there are many types of objects, and when training a recognition model, a large amount of manpower and time is needed to obtain sufficient training datasets and perform object labeling. In this way, the recognition system is difficult to apply immediately.

[0003] In addition, the training datasets obtained by shooting can only be limited to certain specific background environments. In the actual application of the recognition system, the difference in background environment will cause the recognition accuracy to decrease. SUMMARY

[0004] The present disclosure relates to a recognition system and a method for augmenting and training of dataset.

[0005] According to an embodiment of the present disclosure, a method for augmenting and training of dataset of a recognition system is provided. The method for augmenting and training of dataset of the recognition system includes the following steps. A plurality of dataset pictures are obtained, each dataset picture including an object image. A plurality of environment images are obtained. The object image is segmented from each dataset picture. A plurality of dataset parameters are set. Based on the object image and the environment image, augmentation of the dataset pictures is performed according to the dataset parameters to increase the number of the dataset pictures. A recognition model is trained using the dataset pictures.

[0006] According to another embodiment of the present disclosure, a recognition system is provided. The recognition system includes an image processing device and a model establishing device. The image processing device includes an image capturing unit, a segmentation unit, and a parameter setting unit. The image capturing unit is used to obtain a plurality of dataset pictures and a plurality of environment images. Each dataset picture includes an object image. The segmentation unit is used to segment the object image from each dataset picture. The parameter setting unit is used to set a plurality of dataset parameters. The model establishing device includes a dataset augmentation unit and a training unit. The dataset augmentation unit is used to perform augmentation of the dataset pictures based on the object image and the environment image according to the dataset parameters to increase the number of the dataset pictures. The training unit is used to train a recognition model according to the dataset pictures.

[0007] For a better understanding of the above and other aspects of the present disclosure, embodiments are hereinafter described in detail with reference to the following drawings: BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 A block diagram of a recognition system according to an embodiment is shown.

[0009] Figure 2 A flowchart of a method of augmenting and training of a recognition system according to an embodiment is shown.

[0010] Figure 3 A plurality of images of the physical object in a standing position are taken from different angles by the image capturing unit.

[0011] Figure 4 A plurality of images of the physical object in a lying position are taken from different angles by the image capturing unit.

[0012] Figure 5 An image of the vending machine is taken by the image capturing unit.

[0013] Figure 6 An image of the display platform is taken by the image capturing unit.

[0014] Figure 7 An action diagram of separating the object image from the image is shown.

[0015] Figure 8 Examples of various object rotation angles are shown.

[0016] Figure 9 Examples of object images after rotation are shown.

[0017] Figure 10 Examples of various object positions are shown.

[0018] Figure 11 Examples of various object magnifications are shown.

[0019] Figure 12 Examples of various object overlay ratios are shown.

[0020] Figure 13 Examples of various object size relationships are shown.

[0021] Figure 14 Examples of various environmental background colors are shown.

[0022] Figure 15 An example of a material image is shown.

[0023] Figure 16 An example of an application field is shown.

[0024] Figure 17 An example of allowing augmentation of incomplete objects is shown.

[0025] Figure 18 A detailed flowchart of step S220 according to an embodiment is shown.

[0026] Figure 19 A detailed flowchart of step S310 according to an embodiment is shown.

[0027] Symbol Description

[0028] 1000: recognition system

[0029] 100: image processing device

[0030] 110: image capturing unit

[0031] 130: segmentation unit

[0032] 140: parameter setting unit

[0033] 200: model establishing device

[0034] 210: map augmentation unit

[0035] 220: distribution adjustment unit

[0036] 230: training unit

[0037] 240: database

[0038] 300: verification device

[0039] 310: training decision unit

[0040] 320: application unit

[0041] 330: performance decision unit

[0042] EG: edge

[0043] EP: environmental image

[0044] FM: map image

[0045] L1: target loss function value

[0046] m1: target detection average precision

[0047] MD: recognition model

[0048] MK1, MK2: mask

[0049] OB: physical object

[0050] OP: object image

[0051] PD1: image capturing and processing program

[0052] PD2: map augmentation and model training program

[0053] PD3: training result and recognition performance verification program

[0054] PR: map parameter

[0055] RG: Range of the edge

[0056] S110, S120, S130, S140, S210, S220, S221, S222, S23, S224, S230, S310, S311, S312, S313, S314, S315, S320, S330, S340, S350, S360: Step

[0057] TA: Target area DETAILED DESCRIPTION

[0058] Please refer to Figure 1 , which illustrates a block diagram of a recognition system 1000 according to an embodiment. The recognition system 1000 includes an image processing device 100, a model establishing device 200, and a verification device 300. The image processing device 100 is used to perform the capturing and processing of the image data frames FM. The image data frames FM include the object images OP and the environment images EP. The model establishing device 200 is used to perform the augmentation of the image data frames FM and the training of the recognition model MD. The verification device 300 is used to perform the verification of the training results and the recognition effectiveness. The present embodiment increases the number of the image data frames FM through the adaptive image data augmentation technology, and various different settings are made to the object images OP and the environment images EP during the image data augmentation process, which enriches the variability of the image data frames FM. Therefore, a recognition model MD with high accuracy can be trained through a small amount of image data frames FM.

[0059] The image processing device 100, the model establishing device 200, and the verification device 300 are, for example, a camera, a display device, a circuit, a chip, a circuit board, a computer, or a storage device storing program codes, etc. The image processing device 100, the model establishing device 200, and the verification device 300 can be three different devices; or, the image processing device 100, the model establishing device 200, and the verification device 300 can be integrated into two or one device.

[0060] The image processing device 100 includes an image capturing unit 110, a segmentation unit 130, and a parameter setting unit 140. The model establishing device 200 includes a map augmenting unit 210, a distribution adjusting unit 220, a training unit 230, and a database 240. The verification device 300 includes a training determining unit 310, an application unit 320, and an effectiveness determining unit 330. The image capturing unit 110 is, for example, a camera, a data input device, a circuit, a chip, a circuit board, a computer, or a storage device storing program codes. The segmentation unit 130, the parameter setting unit 140, the map augmenting unit 210, the distribution adjusting unit 220, the training unit 230, the training determining unit 310, the application unit 320, and the effectiveness determining unit 330 are, for example, a circuit, a chip, a circuit board, a computer, or a storage device storing program codes. The database 240 is, for example, a memory, a hard disk, or a cloud storage center. The operation of each component is described in detail below in combination with a flowchart.

[0061] Please refer to Figure 2 , which shows a flowchart of a map augmenting and training method of the recognition system 1000 according to an embodiment. Steps S110-S140 are an image capturing and processing procedure PD1. In step S110, the image capturing unit 110 obtains a plurality of map pictures FM. Each map picture FM includes at least one object image OP and an environment image EP. In this step, the image capturing unit 110 performs shooting according to different postures and / or different angles to obtain these map pictures FM. For example, please refer to Figure 3 , which shows that the image capturing unit 110 performs shooting on the physical object OB in 8 angles in a standing posture. Please refer to Figure 4 , which shows that the image capturing unit 110 performs shooting on the physical object OB in 8 angles in a lying posture. When the angle is changed, the physical object OB can be rotated, or the image capturing unit 110 can revolve around the physical object OB. The image capturing unit 110 can shoot various object images OP of the physical object OB in various orientations according to different postures and / or different angles. In fact, the object to be recognized in the picture can be presented in different postures and / or different angles, so these object images OP in different orientations can improve the accuracy of the recognition model MD.

[0062] Then, in step S120, the image capturing unit 110 obtains a plurality of environment images EP. The environment images EP can not contain object images OP. In this step, the image capturing unit 110 can shoot environment images EP on a physical environment, or obtain environment images EP in a drawing manner. For example, please refer to Figure 5 , which shows that the image capturing unit 110 performs shooting on a vending machine. Please refer to Figure 6The image capturing unit 110 captures the display platform. The object to be recognized can appear in various environments, and thus the various environment images EP can improve the accuracy of the recognition model MD.

[0063] The order of the steps S110 and S120 can be changed, or they can be performed simultaneously.

[0064] The object image OP is included in the image FM obtained in the step S110. In the step S130, the segmentation unit 130 segments the object image OP from each image FM. For example, refer to Figure 7 The leftmost diagram shows that the image FM includes the object image OP and the environment image EP. The segmentation unit 130 finds the edge EG of the object image OP by using an edge detection technique, and obtains a mask MK1. In the mask MK1, the weight inside the edge EG is 100%, and the weight outside the edge EG is 0%. The segmentation unit 130 can further perform a smoothing process on the edge EG to obtain a mask MK2. The mask MK2 has an edge range RG. The edge range RG of the mask MK2 is a strip range that is expanded by a certain multiple of the edge EG of the mask MK1. In the edge range RG, the weight decreases from 100% to 0% outwardly. Then, the segmentation unit 130 superimposes the mask MK2 on the image FM to retain the object image OP. The gradual weight of the edge range RG can smooth the edge of the object image OP, and avoid the edge of the object image OP being too sharp to affect the model accuracy.

[0065] Next, in the step S140, the parameter setting unit 140 sets a plurality of image parameters PR. The image parameters PR are, for example, an object rotation angle, an object position, an object magnification, an object superimposition ratio, an object size relationship, an environment background color, a material image, an application field, or an incomplete object augmentation. Refer to Figure 8 Various object rotation angles are shown. The object image OP can be rotated by a certain angle clockwise or counterclockwise along the X-axis, the Y-axis, or the Z-axis. Refer to Figure 9 The object image OP after rotation is shown. The object image OP is rotated in this step. The rotation can be directly deformed on the object image OP to simulate the rotation, or a three-dimensional model can be obtained, the three-dimensional model is rotated, and then projected into a two-dimensional object image OP. For example, Figure 9 For example, the diagram in the upper left corner shows that when the object image OP is rotated 30 degrees clockwise along the X-axis, the head is reduced and the feet are enlarged, so that the adjusted object image OP simulates the appearance of the physical object OB (shown in Figure 3 ) after being rotated 30 degrees clockwise along the X-axis. For example, Figure 9For example, in the lower right corner, the object image OP is rotated 30 degrees counterclockwise along the Z axis, and the size is not changed, only the image is rotated, so that the object image OP simulates the appearance of the physical object OB (shown in Figure 3 ) after being rotated 30 degrees counterclockwise along the Z axis.

[0066] Please refer to Figure 10 , which illustrates various object positions. For example, the object image OP can be moved to the coordinate point (45, 55), or to the coordinate point (15, 75), or to the coordinate point (80, 35). In Figure 10 , the object image OP is rotated along the Z axis. The various parameters PR of the embodiment can be used in combination.

[0067] Please refer to Figure 11 , which illustrates various object magnifications. For example, the object image OP can be reduced by a factor of 0.5, or enlarged by a factor of 1.3. The reduction or enlargement of the magnification is equivalent to simulating the distance of the physical object OB (shown in Figure 3 ).

[0068] Please refer to Figure 12 , which illustrates various object overlap ratios. For example, the overlap ratio of the object image OP is, for example, 0, 0.3, or 0.7. The change in the overlap ratio is equivalent to simulating the overlap of the physical objects OB (shown in Figure 3 ).

[0069] Please refer to Figure 13 , which illustrates various object size relationships. For example, when multiple object images OP are obtained, the size relationship is preferably matched, for example, a bottle should be larger than a can at the same distance. Each object image OP in the positioning area TA can be adjusted to the appropriate size relationship in the area according to the parameters of the parameter setting unit 140, such as object position, object magnification, object overlap ratio, etc.

[0070] Please refer to Figure 14 , which illustrates various environmental background colors. For example, as shown in the left side of the figure, the parameter setting unit 140 can set the environmental background color to be a single color (such as yellow (R: 238, G: 245, B: 149)). Or, as shown in the right side of the figure, the parameter setting unit 140 can set the environmental color to be a multi-color level (for example, a multi-color level of colors or a multi-color level of single colors).

[0071] Please refer to Figure 15An example of a material image is shown. The parameter setting unit 140 can use a real image, a three-primary color mixed image, a drawing, or an image obtained from a network as the material image. The material image is, for example, a cartoon pattern, a color block combination, or a real image. A change in the material image corresponds to a simulation of the physical object OB (shown in Figure 3 ) in a different environmental scene.

[0072] Referring to Figure 16 , an example of an application field is shown. The application field is, for example, a display stand or a desktop. The parameter setting unit 140 can change the environmental image EP to be the application field. A change in the application field corresponds to a simulation of the physical object OB (shown in Figure 3 ) in a different environment.

[0073] Referring to Figure 17 , an example of allowing augmentation of an incomplete object is shown. As shown in the left drawing, the parameter setting unit 140 can define the object image OP to be located entirely within the positioning area TA. Alternatively, as shown in the right drawing, the parameter setting unit 140 can define the object image OP to have a portion located outside the positioning area TA. A change in the degree of the object image OP extending out of the frame can simulate a situation in which only a portion of the physical object OB (shown in Figure 3 ) is captured.

[0074] Next, steps S210 to S230 are entered. Steps S210 to S230 are the image augmentation and model training program PD2. In step S210, the image augmentation unit 210 augments the image frames FM based on the object images OP and the environmental images EP according to the image parameters PR to increase the number of the image frames FM. The object images OP in various poses and / or various angles can be combined with various environmental images EP to form new image frames FM. Alternatively, the object images OP or the environmental images EP can be adjusted according to the image parameters PR to form new image frames FM. The augmented images can simulate various possible situations and reduce the manual shooting action. In addition, in this step, the image augmentation unit 210 knows the positions and ranges of the object images OP in the process of augmenting the images, so the object images OP can be automatically labeled in the image frames FM without manual labeling.

[0075] Next, in step S220, the distribution adjustment unit 220 adjusts the distribution of the image frames FM to make the distribution of the image frames FM uniform. Referring to Figure 18, which illustrates a detailed flowchart of step S220 according to an embodiment. In step S221, the distribution adjustment unit 220 judges whether the types of the image asset pictures FM are balanced. The types are defined according to various combinations of different object images OP, different environment images EP, different image asset parameters PR, etc. The so-called balance means whether the quantities of the various types are close. If the types of the image asset pictures FM are balanced, step S224 is entered; if the types of the image asset pictures FM are not balanced, step S222 is entered.

[0076] In step S222, the distribution adjustment unit 220 expands the image asset pictures FM for the type with a smaller quantity of image assets. The distribution adjustment unit 220 can expand the image asset pictures FM for only one type at a time. Alternatively, the distribution adjustment unit 220 can expand the image asset pictures FM for all types with a quantity of image assets lower than the highest quantity of image assets at a time. The distribution adjustment unit 220 makes the quantities of the various types close through this step. For example, 3 objects, 3 poses, and 3 environments can form 9 types. If the quantity of image assets of a type is 950 while the quantity of image assets of most types is 1050, the image asset pictures FM of this type need to be expanded so that the quantity of image assets reaches 1050.

[0077] In step S224, the distribution adjustment unit 220 expands the image asset pictures FM for the types lower than a predetermined threshold value. The distribution adjustment unit 220 makes the quantities of the various types higher than or equal to the predetermined threshold value through this step.

[0078] Next, please refer to Figure 1 and Figure 2 In step S230, the training unit 230 trains the recognition model MD according to the image asset pictures FM. The recognition model MD can be stored in the database 240.

[0079] Then, steps S310-S330 are entered. Steps S310-S330 are the verification procedure PD3 of the training result and the recognition effectiveness. In step S310, the training judgment unit 310 judges whether the recognition model MD is trained according to a training output data of the recognition model MD. The training output data is, for example, a target detection mean average precision (mAP) m1 or a target loss function value (Loss error) L1, etc. Please refer to Figure 19, which illustrates a detailed flowchart of step S310 according to an embodiment. In step S311, the training decision unit 310 determines whether the target detection average precision m1 is greater than a predetermined percentage (e.g., 75%, 80%, 85%, 90%, etc.). If the target detection average precision m1 is greater than the predetermined percentage, step S312 is entered; if the target detection average precision m1 is not greater than the predetermined percentage, the training of step S230 is continued.

[0080] In step S312, the training decision unit 310 determines whether the target loss function value L1 is less than a predetermined loss value (e.g., 0.8, 0.9, 1, 1.1, 1.2, etc.). If the target loss function value L1 is not less than the predetermined loss value, the training of step S230 is continued; if the target loss function value L1 is less than the predetermined loss value, step S313 is optionally entered or step S320 is directly entered.

[0081] In step S313, the iteration action is repeated a predetermined number of times (e.g., 1000 times, which can be adjusted as needed).

[0082] In step S314, the training decision unit 310 determines whether the change in the target loss function value L1 is less than a predetermined multiple (e.g., 0.7 times, 0.8 times, 0.9 times, etc.). If the target loss function value L1 continues to converge and the change is not less than the predetermined multiple, step S315 is entered and the training is terminated.

[0083] Next, please refer to Figure 1 and Figure 2 In step S320, the training effectiveness of the recognition model MD is confirmed.

[0084] Next, in step S330, the effectiveness decision unit 330 determines whether the recognition model MD is accurate. If the recognition model MD is not accurate, step S220 is returned to; if the recognition model MD is accurate, step S350 is entered.

[0085] In addition, step S340 can also enter step S350. In step S340, an object import program is performed.

[0086] In step S350, the application unit 320 applies the recognition model MD to perform an identification program. For example, the application unit 320 can obtain an actual captured object image, and the application unit 320 applies the recognition model MD to identify the probability of containing one or more objects in the object image and the position and range of the object.

[0087] Next, in step S360, the effectiveness determination unit 330 determines whether there are new objects to be imported. If there are new objects to be imported, return to step S110, and the corresponding entity object OB is extracted from the image picture FM, and the subsequent steps are executed in sequence; if there are no new objects to be imported, return to step S230, and the recognition model MD is trained.

[0088] That is, once it is found that the recognition model MD is inaccurate, return to step S220, and the distribution adjustment unit 220 is used to adjust the distribution of the image picture FM. For example, directly increase the image for the object that is recognized incorrectly, so that the recognition accuracy of the recognition model MD for this object can be improved. For example, the distribution adjustment unit 220 can increase the image by 30% or 20% for the object recognition error, so as to improve the recognition accuracy.

[0089] Once there are new objects to be imported, return to step S110, and the new objects are trained.

[0090] Embodiments of the present disclosure propose a recognition system 1000 and its image augmentation and training method, which uses adaptive image augmentation technology to increase the number of image pictures FM, and various different settings are made to the object image OP and the environment image EP during the image augmentation process, which enriches the variability of the image picture FM. In addition, the distribution adjustment technology of the image augmentation and model training program PD2 and the verification program PD3 of the training result and recognition effectiveness improve the accuracy of the recognition model MD. Embodiments of the present disclosure can train a high-accuracy recognition model MD with a small number of image pictures FM.

[0091] In summary, although the present disclosure has been disclosed as above by embodiments, it is not intended to limit the present disclosure. Those skilled in the art to which the present disclosure belongs can make various modifications and decorations without departing from the spirit and scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the appended claims and their equivalent scope.

Claims

1. A method for augmenting and training of a recognition system, characterized by The method comprises: obtaining a plurality of image frames, each image frame comprising an object image; obtaining a plurality of environment images; segmenting the object image from each image frame, comprising: smoothing the edges of the object image to obtain a mask, the mask having an edge range in which a weight decreases outwardly; and superimposing the mask on each image frame to retain the object image; setting a plurality of image parameters; based on the plurality of object images, augmenting the plurality of image frames according to the plurality of image parameters, or based on the plurality of environment images, augmenting the plurality of image frames according to the plurality of image parameters, or based on the plurality of object images and the plurality of environment images, augmenting the plurality of image frames according to the plurality of image parameters, to increase the number of the plurality of image frames; and training a recognition model using the plurality of image frames.

2. The method of claim 1, wherein the plurality of image frames are captured according to different poses and / or different angles.

3. The method of claim 1, wherein the plurality of environment images are virtual environments.

4. The method of claim 1, wherein the plurality of image parameters are object rotation angles, object positions, object magnifications, object superimposition ratios, object size relationships, environment background colors, image sources, application domains, or object removal degrees.

5. The method of claim 1, wherein the plurality of object images are automatically labeled during the step of augmenting the plurality of image frames.

6. The method of claim 1, further comprising: adjusting the distribution of the plurality of image frames.

7. The method of claim 6, wherein the step of adjusting the distribution of the plurality of image frames comprises: determining whether the plurality of types of the plurality of image frames are balanced; if the plurality of types of the plurality of image frames are not balanced, augmenting the plurality of image frames for some of the plurality of types; determining whether the number of each type reaches a predetermined threshold value; and if the number of one of the plurality of types is less than the predetermined threshold value, augmenting the plurality of image frames for the type that is less than the predetermined threshold value.

8. The method of claim 6, further comprising: determining whether the recognition model is trained based on the training output data of the recognition model; if the recognition model is trained, determining whether the recognition model is accurate; and if the recognition model is not accurate, performing the step of adjusting the distribution of the plurality of image frames again.

9. The method of claim 8, wherein the step of determining whether the recognition model is trained comprises: ​ ​ determining whether the target detection average precision is greater than a predetermined percentage; if the target detection average precision is not greater than the predetermined percentage, the recognition model is not trained; determining whether the target loss function value is less than a predetermined loss value; and if the target loss function value is not less than the predetermined loss value, the recognition model is not trained.

10. The method of claim 9, wherein the step of determining whether the recognition model is trained further comprises: determining whether a change of the target loss function value is less than a predetermined multiple; and if the change of the target loss function value is not less than the predetermined multiple, the recognition model is not trained.

11. A recognition system characterized by The recognition system comprises: an image processing device comprising: an image capturing unit configured to obtain a plurality of object images and obtain a plurality of environment images, each object image comprising an object image; a segmentation unit configured to segment the object image from each object image, comprising: performing a smoothing process on an edge of the object image to obtain a mask, the mask having an edge range in which a weight decreases outwardly; and superimposing the mask on each object image to retain the object image; and a parameter setting unit configured to set a plurality of object parameters; and a model establishing device comprising: an object augmentation unit configured to augment the plurality of object images according to the plurality of object parameters, or augment the plurality of environment images according to the plurality of object parameters, or augment the plurality of object images and the plurality of environment images according to the plurality of object parameters, to increase the number of the plurality of object images; and a training unit configured to train a recognition model according to the plurality of object images.

12. The recognition system of claim 11, wherein the image capturing unit is configured to capture the plurality of object images according to different poses and / or different angles.

13. The recognition system of claim 11, wherein the segmentation unit is further configured to smooth the edge of the plurality of object images.

14. The recognition system of claim 11, wherein the plurality of environment images are virtual environments.

15. The recognition system of claim 11, wherein the plurality of object parameters are object rotation angles, object positions, object magnifications, object superimposition ratios, object size relationships, environment background colors, material taking images, application fields, or object moving out of frame degrees.

16. The recognition system of claim 11, wherein the object augmentation unit is further configured to automatically annotate the plurality of object images.

17. The recognition system of claim 11, wherein the model establishing device further comprises: a distribution adjustment unit configured to adjust the distribution of the plurality of object images.

18. The recognition system of claim 17, wherein if a plurality of types of the plurality of object images are imbalanced, the distribution adjustment unit augments the plurality of object images for part of the plurality of types; and If the number of one of the types is lower than a predetermined threshold, the distribution adjustment unit expands the number of images of the type that is lower than the predetermined threshold. 19.The recognition system of claim 17, further comprising: a verification device comprising: a training determination unit configured to determine whether the recognition model is trained based on training output data of the recognition model; an application unit configured to apply the recognition model to a recognition procedure; and an effectiveness determination unit configured to determine whether the recognition model is accurate. 20.The recognition system of claim 19, wherein the training determination unit determines whether the recognition model is trained based on a target detection average precision and a target loss function value.

Images