Synthetic image generation for machine learning model training

Abstract

An improved method and system for synthetic image data generation for use in machine learning model training. A graphical creation module and rendering module allow creation and manipulation of one or more virtual objects within a virtual environment, and allows manipulation of object and / or environmental variables to create poses of the object. A graphical skin can be applied to a geometric shape to mimic any real world object of interest. Multiple poses of the object may be obtained by planned rotation of the object or of a virtual camera. A bounding box around the object is generated and saved with the pose in a database for subsequent utilization in machine learning model training.

Description

Synthetic Image Generation For Machine Learning Model Training DatasetsTECHNICAL FIELD

[0001] The invention relates to computer modeling and simulation for creation of three-dimensional virtual simulations of real-world environmental elements useful for machine learning algorithm training .BACKGROUND ART

[0002] Real-time surveillance and obj ect detection, identi fication, and / or tracking by automated methods requires use of machine learning models upon which such automated systems operate . To achieve this a digiti zed image of a real- world obj ect of interest within its environment is typically captured by a physical camera and supplied to the system . The machine learning algorithm then processes the captured image for detection, identi fication, and / or tracking purposes . However, for machine learning capability to work accurately, ef ficiently, and ef fectively, it must first be trained with very large quantities of digiti zed images and obj ect speci fic data . Currently this training is performed in an entirely manual way, or is performed with a combination of manual and computer automated ef forts .

[0003] One important step in machine learning is the accurate tagging of obj ects of interest to train the machine learning model algorithm . Obtaining digiti zed images of a real-world obj ect of interest traditionally involves use of a physical camera capturing images of the obj ect within its environment ,from all angles . This is often accomplished through use of a green screen background and a turntable upon which the obj ect of interest is placed, and which is rotated while digital images of the obj ect are obtained . Environmental variations , for example lighting of the obj ect ' s surfaces , must also be simulated using limited studio lighting . Background images are then added to replace the green screen in the image to create an illusion that the obj ect was captured in that background . This task is complex, highly repetitive , unreasonably time consuming, unreasonably arti ficial in the sense that lighting of the obj ect will likely conflict with superimposed background lighting, and subj ect to errors introduced by human action . Moreover, positioning of additional obj ects in the capture frame is also necessary to accurately simulate real- world conditions , for example , with multiple items obscuring portions of the obj ect of interest on a conveyor system . This requires access to multiple additional obj ects and environmental structures , which further complicates the image gathering process and dataset development .

[0004] Accordingly, a need exists for a means by which any number of images of an obj ect or multiple obj ects may be generated automatically, with consistent environmental characteristics , oriented within an essentially infinitely variable environment , to create a large dataset . The present invention satis fies these needs and others as shown in the detailed description below .SUMMARY OF INVENTION

[0005] The synthetic image generation invention presented herein provides an interactive three dimensional arti ficial environment that mimics visuals of the real world . For example , one can recreate living room furniture and ambiance ,background obj ects , obj ect images from di f ferent perspectives , lighting and light reflections , and the like . With the help of the rich environment being created, this invention caters towards any industry where high quality product images with dynamic backgrounds are beneficial .

[0006] Synthetic image data generation for machine learning model training methods and systems embodiments as discussed herein are provided as set out in the appended claims section .BRIEF DESCRIPTION OF DRAWING ( S )

[0007] The present invention may be more fully understood by reference to the following detailed description of the preferred embodiments of the invention when read in reference to the accompanying drawings , wherein :FIG . 1 presents a systems high-level block diagram of the functional components of the novel system comprising a first embodiment ;FIG . 2 presents a block diagram of the system hardware components of the embodiment ;FIG . 3 presents a flow diagram of processing method steps of the embodiment ;FIG . 4 presents a flow diagram of processing method steps of an additional embodiment ;FIG . 5 presents a flow diagram of processing method steps that may be utili zed by the embodiments to obtain a first pose of an obj ect of interest ;FIG . 6 presents a flow diagram of processing method steps that may be utili zed by the embodiments to obtain one or more poses of an obj ect of interest ;FIG . 7 presents a depiction of a user interface of an embodiment displaying synthetically generated obj ects of interest in a synthetic environment ;FIG . 8 presents a depiction of the methods for obtaining one or more poses of obj ects of interest ;FIG . 9 presents a depiction of a user interface of an embodiment displaying a means for creating a synthetic obj ect of interest by applying a graphical skin over a geometric shape ;FIG . 10 presents a depiction of an obj ect of interest as it may exist within a synthetic store environment ; andFIG . 11 presents a depiction of the obj ect of interest as the embodiment surrounds the obj ect with a bounding box .

[0008] The above figures are provided for illustration and description only, and are not intended to limit the disclosed invention . Use of the same reference number in multiple figures is intended to designate the same parts or method steps .DESCRIPTION OF EMBODIMENTS

[0009] The discussion of embodiments herein references machine learning training methods by manual and automated means . For example , U . S . Patent Application Serial Number 17 / 932 , 063 filed on September 14 , 2022 , entitled, System and Method for Automatically Labelling Media ( 0. Mani , et al . ; incorporated by reference herein for all purposes ) describes automatic labeling of digiti zed images of real-world environmental elements for machine learning algorithm training .

[0010] When used herein, the terms "virtual" and " synthetic" are interchangeable , each describing something existing within a computer processing device environment and not in what is perceived as the five-sense physical world, but which may be representations of actual obj ects existing in the physical world .

[0011] When used herein, the term "pose" refers to the speci fic arrangement of a virtual obj ect of interest within a virtual environment , which is subsequently captured as a digital image for additional processing .

[0012] The system described herein utili zes commercially available computer processing hardware and commercially available software development environment tools and languages to establish the unique functionality through which the novel methods may be implemented and practiced . One of ordinary skill will understand that a "computer, " "computing device , " or "computer processing device" as used herein is any computing device , including laptop, desktop, workstation, or similar hand-held device capable of and adapted to run stored program software code , accept user input , store data( transient , volatile , non-volatile , short-term, long-term, or some combination thereof ) , and present to the user computer-generated output in the form of graphical representations, text, sounds, and the like. The system herein may be a single computer processor device or distributed among a plurality of networked computer processor devices. Such network may be wired, wireless, or some combination thereof.

[0013] Figure 1 presents a systems high-level block diagram of the functional components of the novel system (100) comprising a first embodiment. A first computer processing device (102) serves as a primary interface with the system for the user.The first computing device (102) is adapted (i.e., programmed) to run specialized application software (104) to perform the method steps described herein. Conventional programming languages are utilized depending on the hardware and networking means, and are well understood by one of ordinary skill, requiring no additional discussion herein.

[0014] The first computing device (102) operates a specialized application (104) interface and allows user login, three- dimensional model uploading, image capture sessions, system reporting, and the like, in addition to the novel functionality claimed herein. Additional functionality includes capturing data, importing object textures, importing three-dimensional models, storing object tagging data, and the like .

[0015] A second computing device (106) provides administrative functionality for the system, including cloud-based data storage (108) . For example, an administrator interface allows a user to manage user account registration, account payments, account configuration, authentication, and the like. While it is possible for a single computing device to manage the entirety of the functionality of the stated first and second computing devices, offloading the administrative tasks to another computing device allows for more efficient resourcemanagement. Moreover, the system is structured to allow multiple users (first computing devices) to access the administrative and other features over a conventional network.

[0016] Figure 2 presents a block diagram (200) of the system hardware components of the embodiment. A first computing device (202) provides computing architecture including a processor (204) capable of executing stored program instructions, an input module (212) and an output module (214) allowing interaction with a user, and computer memory (206) sufficient to render operable the functionality described herein. This includes the ability to run a graphical creation module (208) with which virtual objects of interest may be created, and a rendering module (210) with which the virtual objects may be manipulated as described herein. A conventional database device (216) may exist in internal memory (206) in whole or in part, or may be external to the computing device and accessible using a network connection.

[0017] Examples of the graphical creation module (208) include the commercially available three-dimensional visual effects software known as Maya, Blender, and the like. Maya provides an application programming interface (API) for accessing its programmed functionality, which includes creation of three- dimensional geometric virtual objects, rendering of surface textures, and the like. Software implementing the functionality herein accesses the graphical creation module (208) through the various API function calls to allow user interaction therewith.

[0018] Examples of the rendering module (210) include the commercially available Unity real-time three-dimensional development platform, RenderMan, Blender Cycles, and the like. Unity is preferred for its real-time interactivity. Others are focused on offline rendering and are suboptimal for real-timeapplications. Unity provides an API for accessing its programmed functionality, which includes rendering objects, virtual environments, and virtual environment variables, for example, illumination of the environment and / or object. Software implementing the functionality herein accesses the rendering module (210) through the various API function calls to allow user interaction therewith.

[0019] Figure 3 presents a flow diagram (300) of processing method steps of the embodiment. Depicted are processing steps performed by the graphical creation module (208) and rendering module (210) . The graphical creation module (208) generates, through user input, a three-dimensional virtual environment (302) . This virtual environment can be any suitable environment within which an object of interest may be found. Figures 7, 10, and 11 present examples of such virtual environments, but can be any environment reasonable or fanciful depending on need. Next, the graphical creation module generates, through user input, a three-dimensional virtual object of interest (304) , which is placed within the virtual environment. Processing is then continued with the rendering module (210) .

[0020] The rendering module (210) then arranges, through user input, the generated object of interest within the virtual environment (306) . Arrangement includes placement of the object of interest within the environment respective of other potential objects; orientation of the object of interest with respect to the viewer (or virtual camera) ; modification of scene lighting variables; and the like. Once arranged, the rendering module (210) then captures a pose of the object (308) and generates a bounding box around same (310) by normalizing a two-dimensional projection of the three- dimensional object according to screen resolution. Thecaptured pose is then provided in digital image format, annotated with object information (eg., identity, characteristics, etc., in a corresponding text file) , and is saved in conjunction with the bounding box coordinates in an object file that is stored in an object dataset (312) . The object dataset may then be provided to a machine learning model for training (314) .

[0021] Figure 4 presents a flow diagram (400) of processing method steps of an additional embodiment. As before, the graphical creation module (208) generates, through user input, a three-dimensional virtual environment (302) . This virtual environment can be any suitable environment within which an object of interest may be found. Figures 7, 10, and 11 present examples of such virtual environments, but can be any environment reasonable or fanciful. Next, the graphical creation module (208) generates, through user input, a three- dimensional virtual object of interest (304) , which is placed within the virtual environment. The rendering module (210) then arranges, through user input, the generated object of interest within the virtual environment (306) . Once arranged, the rendering module (210) then captures a pose of the object (308) and generates a bounding box around same (310) . The captured pose is then provided in digital image format, annotated with object information (eg., identity, characteristics, etc., in a corresponding text file) , and is saved in conjunction with the bounding box coordinates in an object file that is stored in an object dataset (312) . If additional images are desired (402) , for example, to generate a large dataset, the rendering module modifies an attribute, view, and / or position of the object and / or environment (404) , captures a new pose (308) , generates a new bounding box (310) , and saves the capture pose and bounding box coordinates in anobject dataset (312) . Once the desired object dataset size is reached, the object dataset may then be provided to a machine learning model for training (314) .

[0022] Figure 5 presents a flow diagram (500) of processing method steps that may be utilized by the embodiments to obtain a first pose of an object of interest. The rendering module (210) establishes a position of the virtual object of interest within the environment (502) by first positioning the virtual object along the three-dimensional coordinate space with respect to the environment, and establishing its rotation. A "virtual camera" is provided by the rendering module (210) , which is effectively the view provided to the user on the display (i.e., the user is looking through the view finder of the virtual camera) . With the virtual camera position established (502) , the camera may be rotated to a specific position around the object (504) , and a pose captured by obtaining a digital capture of the pose of the image (308) . This image data then becomes part of the object dataset (312) when combined with the annotation data and bounding box coordinates for the new pose.

[0023] Figure 6 presents a flow diagram of processing method steps that may be utilized by the embodiments to obtain one or more poses of an object of interest. As before, the rendering module (210) establishes a position of the virtual object of interest within the environment, and with the virtual camera position established (502) , captures a pose (308) by obtaining a digital capture of the pose of the image. The image is annotated with object information (eg., identity, characteristics, etc., ) in a corresponding text file, and the bounding box is generated with respect to the pose (310) . The image, annotation data, and bounding box coordinates are saved as object files (504) . If additional poses are desired (506)the camera position is incrementally rotated by a predetermined amount (508) and a new pose established and captured (308) , annotation data and bounding box generated (310) , and object files saved (504) . Once the desired number of poses is obtained (506) the object files are added to the object dataset (312) .

[0024] The rendering module (210) can also vary lighting of the object of interest, which is helpful in subsequent machine learning model training. By varying the scene lighting, more realistic environmental changes can be captured in a given pose mimicking real-world environmental. This makes the training far more robust and challenging during training when the machine learning model is ultimately used for detection of real-world objects of interest.

[0025] It is also possible for the rendering module (210) to simulate environmental effects such as wind, thereby causing perturbations in the virtual object that mimic real-world conditions that might be encountered and which a machine learning model should consider in training. As the object is perturbed, multiple captures of the pose and bounding box coordinates are obtained and added to the dataset thereby capturing the disturbances.

[0026] Although the above describes manual input by a user in generation and manipulation of environments and objects, automation is also envisioned. For example, it is possible to write software scripts to automate environment and object generation to increase system efficiency and to allow for rapid generation of massive datasets. Moreover, software scripts may automate arrangement, capture, and storage with minimal or no direct user input.

[0027] It is possible for the system to generate more than one object of interest for processing. Figure 7 presents adepiction of a user interface of an embodiment displaying synthetically generated objects of interest in a synthetic environment (700) . The objects of interest in this example are available from an image library, preformed, but may also be created from geometric shapes as described below. User generated images (as described below) may also be saved for subsequent recall and use in generating additional datasets.

[0028] In the depicted user interface display (702) the graphical creation module generated synthetic environment includes a background scene (704) with walls, windows, floors, furniture, cabinetry, and the like, simulating a real-world setting. For example but not limitation, a kitchen and table (718) is depicted upon which multiple objects of interest are present. In this instance the objects of interest include a milk carton (706) , chicken nuggets (708) , banana (710) , potato fries (712) , and an apple (714) , all supported within a serving tray (716) . Shown is a single pose for digital image capture. The individual objects may be manipulated separately or as a group, with the rendering module (210) establishing lighting and other environmental variables. The objects may be manipulated at will in the three-dimensional coordinate space to establish a desired pose.

[0029] Figure 8 presents an abstract depiction of the methods for obtaining one or more poses of objects of interest (800) . As stated previously, the rendering module (210) display provides a "virtual camera" (804) through which the user views the scene (802) . Poses may be obtained by rotating the camera (804) around the objects of interest (706, 708, 710, 712, 714, and 716) by a predetermined degree; rotating the objects of interest by a predetermined degree; and / or manipulating the lighting and environmental effects (806) . With each pose thesystem automatically computes a bounding box around each individual object and retains the coordinates.

[0030] In the present embodiment a virtual turn-table style manipulation technique is utilized, with object identification logic. The logic is in place to determine: the object's scale and adjust the height of the camera; the object line of sight (If there is any unwanted obstruction blocking view in between the camera and the object of interest, the camera will skip taking that synthetic image that can contribute to false data) ; if the object of interest is placed with more than one object (eg., food tray (716) with different food items) ; and the logic to annotate the objects correctly. This calculation changes for each of the camera positions. As the camera rotates around the objects by a predetermined number of degrees, the perspective changes and one object may be ahead of or beside another object, partially or completely obscuring its view. The logic determines if the object of interest is being obscured by 60% or more, and if so, the annotation data and image for the obscured object is not captured at this perspective. It is possible to vary this percentage amount threshold depending on the desired testing or machine learning model training strategy.

[0031] Figure 9 presents a depiction of a user interface of an embodiment (900) displaying a means for creating a synthetic object of interest by applying a graphical skin over a predefined geometric shape. The user interface (902) presents menu options allowing the user to select a predefined geometric shape (e.g., a cube, sphere, cylinder, or packet) and modify the shape (height, width, length, orientation) as desired using the graphical creation module (208) function calls. Depicted is a three-dimensional cube (six faces) thathas been compressed along the X axis to form the shape of a rudimentary rectangular cereal box (904) .

[0032] To complete the look of the cereal box object of interest the user is allowed to apply a predefined graphical skin to each of the object faces, and trim / position the skin as desired. Menu options (912) present each of the object faces. By selecting a face (912) the system presents the option to the user to select a graphic skin image from memory. As depicted, a graphic skin is applied to face 1 (906) , with face 2 (908) and face 3 (910) still awaiting application. The object may be rotated as necessary to aid in application. Once completed the object of interest may be saved (914) in an object library for later utilization.

[0033] Ideally, to ensure a full data collection, each graphical shape should be captured with approximately fifty different poses. These poses can be applied to any predefined object of interest by assigning to the object a shape from one of the four stated shapes. Nonetheless, it is possible to obtain greater or fewer than fifty poses depending upon testing or training requirements.

[0034] Figure 10 presents a depiction of an object of interest within a synthetic store environment (1000) . As stated previously, the graphical creation module (208) may be utilized to generate essentially any virtual environment within which one may expect to find an object of interest. The synthetic environment (1002) depicted includes a retail space (1004) , checkout counter (1006) , and retail checkout scanning device (1008) . On the checkout counter is an object of interest, namely, the completed cereal box (904) from above.

[0035] Figure 11 presents a depiction of the object of interest a bounding box (1100) generated by the system. The rendering module (310) establishes the scene by allowing manipulation ofthe environment variables, and generates a bounding box (1104) around the object. The virtual camera position may be modified by user controls (1110) , changing the pose at will. Depicted are manual controls for establishing (1110) and capturing (1108) the pose. On capture (1108) , the digital image is saved along with the annotation data and bounding box coordinates. Automated control through use of scripts is possible to rapidly achieve a series of poses, taking into account the above logic when multiple objects are encountered.

[0036] The synthetic object image data generated by this novel system and method is useful for any the following example use cases: 1) hardware (Router, Machines, Robots, Gadgets) ; 2) mobile device (Simulate damage, change colors, introduce dents on the body) ; 3) retail products training (Any edible, packed non-edible products) ; 4) car and automobiles (Simulate the dents, color change, scratches) ; 5) heavy machinery; 6) shelves inside the retail store; and 7) custom retail scanners (eg., DigitKart) . This list presents examples and is not meant to be limiting. Essentially any application of machine learning that requires synthetic object image training data ranging from heavy machinery to micro-cellular level in lifesciences can be developed and generated using these novel systems and methods.

[0037] As indicated above, aspects of this invention pertain to specific "method functions" implementable through various computer systems. In an alternate embodiment, the invention may be implemented as a computer program product for use with a computer system.

[0038] Those skilled in the art should appreciate that programs defining the functions of the present invention can be delivered to a computer in many forms, which include, but are not limited to: (a) information permanently stored on non-writeable storage media ( e . g . read only memory devices within a computer such as ROMs or CD-ROM disks readable only by a computer I / O attachment ) ; (b ) information alterably stored on writeable storage media ( e . g . floppy disks and hard drives ) ; or ( c ) information conveyed to a computer through communication media, such as a local area network, a telephone network, or a public network like the Internet . It should be understood, therefore , that such media, when carrying computer readable instructions that direct the method functions of the present invention, represent alternate embodiments of the present invention .

[0039] The described embodiments are considered in all respects as illustrative and not restrictive . Accordingly, the scope of the invention is established by the appended claims rather than by the foregoing description .

[0040] The recitation of method steps does not denote a limiting sequence for execution of the steps . Such method steps may therefore be performed in parallel or in a sequence other than that recited unless the claim expressly states otherwise .INDUSTRIAL APPLICABILITY

[0041] The invention is applicable and beneficial for any industry employing machine learning tools , and in which machine learning model training is required or desired .

Claims

AMENDED CLAIMS received by the International Bureau on 03 April 2025 (03.04.2025)We claim:

1. A computer automated method for synthetic image data generation, the method steps comprising: generating, with a graphical creation module (208) , a three-dimensional virtual environment (302) and at least one virtual object (304) , the at least one virtual object simulating a real-world object of interest within the virtual environment; arranging, with a rendering module (210) , the at least one virtual object within the virtual environment (306) to establish a first pose of the at least one object; capturing a first pose image of the first pose (308) ; generating automatically, with the rendering module, a two-dimensional first pose bounding box around the at least one virtual object, the first pose bounding box having coordinates within the virtual environment (310) ; and storing automatically the first pose image in an object image file and the first pose bounding box coordinates in an object coordinate file (312) , wherein the object image file and the object coordinate file comprise an object dataset stored in a database device (216) for subsequent use in training one or more machine learning model neural networks .

2. (Cancelled) .

3. The method of Claim 1, the method steps comprising: applying a graphical skin over the virtual object (900) .

4. The method of of Claim 1, the method steps comprising: modifying automatically, with the rendering module, an environmental attribute, a virtual object attribute, a virtual object view, and / or the virtual object position within the virtual environment to establish an additional pose (404) ; capturing automatically an additional pose image of the additional pose (308) ; generating automatically, with the rendering module, a two-dimensional additional pose bounding box around the at least one virtual object, the additional pose bounding box having coordinates within the virtual environment (310) ; storing automatically the additional pose image in an additional object image file and the additional pose bounding box coordinates in an additional object coordinate file, adding each to the object dataset (312) ; and repeating automatically the above steps to increase the object dataset to a predetermined amount (402) .

5. (Cancelled) .

6. The method of Claim 4, the method steps comprising: rotating the virtual object to establish the additional pose (500, 600) .

7. (Cancelled) .

8. The method of Claim 4, the method steps comprising: rotating, with the rendering module (210) , a virtual camera around the at least one virtual object by a predetermined degree to establish the additional pose (600) .

9. The method of Claim 4, the method steps comprising: capturing the additional pose image and the additional pose bounding box coordinates only if the at least one virtual object is not obscured by another virtual object by a predetermined amount.

10. The method of Claim 4, the method steps comprising: applying a graphical skin over the virtual object (900) .

11. A computer automated system for synthetic image data generation, the system comprising: a computer device (202) adapted to provide a graphical user interface (212, 214) for user interaction, a graphical creation module (208) , and a rendering module (210) to perform the automated processing steps comprising : generating, with the graphical creation module (208) , a three-dimensional virtual environment (302) and at least one virtual object (304) , the at least one virtual object simulating a real-world object of interest within the virtual environment; arranging, with the rendering module (210) , the at least one virtual object within the virtual environment (306) to establish a first pose of the at least one object; capturing a first pose image of the first pose (308) ;generating, with the rendering module, a two-dimensional first pose bounding box around the at least one virtual object, the first pose bounding box having coordinates within the virtual environment (310) ; and storing the first pose image in an object image file and the first pose bounding box coordinates in an object coordinate file (312) , wherein the object image file and the object coordinate file comprise an object dataset stored in a database device (216) for subsequent use in training one or more machine learning model neural networks .

12. (Cancelled) .

13. The system of Claim 11, the processing steps comprising: applying a graphical skin over the virtual object (900) .

14. The system of Claim 11, the processing steps comprising: modifying, with the rendering module, an environmental attribute, a virtual object attribute, a virtual object view, and / or the virtual object position within the virtual environment to establish an additional pose (404) ; capturing an additional pose image of the additional pose (308) ; generating, with the rendering module, a two-dimensional additional pose bounding box around the at least one virtual object, the additional pose bounding box having coordinates within the virtual environment (310) ;storing the additional pose image in an additional object image file and the additional pose bounding box coordinates in an additional object coordinate file, adding each to the object dataset (312) ; and repeating the above steps to increase the object dataset to a predetermined amount (402) .

15. (Cancelled) .

16. The system of Claim 14, the processing steps comprising: rotating the virtual object to establish the additional pose (500, 600) .

17. (Cancelled) .

18. The system of Claim 14, the processing steps comprising: rotating, with the rendering module, a virtual camera around the at least one virtual object by a predetermined degree to establish the additional pose (600) .

19. The system of Claim 14, the processing steps comprising: capturing the additional pose image and the additional pose bounding box coordinates only if the at least one virtual object is not obscured by another virtual object by a predetermined amount.

20. The system of Claim 14, the processing steps comprising: applying a graphical skin over the virtual object (900) .

Images