Method and computing system for generating training image sets using automatic semantic segmentation labels

Abstract

The present application relates to a method and a computing system for generating a training image set using automatic semantic segmentation labels. A computer-implemented method for generating a training image set includes: retrieving model data corresponding to a three-dimensional (3-D) model of a target object from at least one storage device; and creating a plurality of two-dimensional (2-D) synthetic images from the model data. The 2-D synthetic image includes multiple views of the 3-D model. The method also includes creating a plurality of semantic segmentation images by identifying a plurality of pixels defining a target object in the 2-D synthetic image, and assigning semantic segmentation labels to the pixels of the identified target object. The method further includes generating link data associating each semantic segmentation label with a corresponding one of the 2-D synthetic images, and storing a training image set including the 2-D synthetic image, the semantic segmentation label, and the link data.

Description

Technical Field

[0001] The present invention relates to the field of machine vision, and more particularly to a method and a computing system for generating a training image set using automatic semantic segmentation labels. Background Art

[0002] At least some known machine vision systems are trained to recognize objects detected by an image sensor (e.g., objects detected by a camera mounted on a machine). For example, at least some known unmanned aerial vehicles ("UAVs") utilize machine vision systems that are trained to recognize objects relevant to autonomous navigation and / or autonomous pursuit of various mission objectives of the UAV. For another example, at least some known autonomous vehicles utilize machine vision systems that are trained to recognize objects relevant to autonomous driving and / or autonomous pursuit of various objectives of the autonomous vehicle. Such machine vision systems are typically trained using a suitable machine learning algorithm applied to a training image set.

[0003] Such training images typically include metadata, sometimes referred to as "labels," to facilitate machine learning. For example, the training images may be semantically segmented to identify at least one object of interest or "target" object in the training images. The semantically segmented images may include a mask, such as a pre-selected color superimposed on each target object in each training image to train the applied machine learning algorithm to associate detected contours with the correct target object. The training images may also include additional labels, such as the orientation of the target object, the configuration of the target object (e.g., if the target object is an aircraft, the configuration may include the position of wing flaps), the presence and type of any obstacles to the view of the target object, etc.

[0004] However, in order to add accurate semantic segmentation labels to the training images, each target object in the original image may require an operator to identify the object, and a mask of the appropriate color must be manually drawn on each target object in the original image. In addition, this manual fitting is subjective and is often not strictly accurate on a pixel-by-pixel basis; i.e., each manually created mask often includes multiple image pixels that are not part of the target object, and / or does not include multiple image pixels that are part of the target object. These pixel-level inconsistencies in the masks may ultimately limit the effectiveness of the trained system in identifying the target objects. Such operations rely on the skills of the operator and take longer than expected.

[0005] Therefore, the time and effort required to generate a complete set of training images can be very intensive, as a very large number of training images, in some cases on the order of 10,000 to 100,000,000 training images, may be required to obtain robust and stable object recognition performance. A large number of images is required because, to ensure that the trained model is robust, for each target object, the image set must typically include different viewpoints (e.g., the target object observed from different angles and distances), different backgrounds (e.g., the target object appears in different weather conditions and at different times of the day), occlusions (e.g., branches in front of the street sign target object), and different potential sensor effects (e.g., camera lens glare under certain lighting conditions). In many cases, acquiring native images of the target object relevant to a particular machine vision application under such a variety of conditions is itself a complex and very time-consuming process (e.g., arranging the physical camera positions to capture raw images of the target object at different angles and distances, different occlusions, different physical lighting conditions, etc.). Summary of the invention

[0006] A method for generating a training image set is described herein. The method is implemented on a computing system including at least one processor in communication with at least one storage device. The method includes retrieving model data corresponding to a three-dimensional (3-D) model of a target object from at least one storage device using at least one processor; and creating a plurality of two-dimensional (2-D) synthetic images based on the model data. The 2-D synthetic image includes multiple views of the 3-D model. The method also includes using at least one processor to create a plurality of semantically segmented images by identifying a plurality of pixels defining a target object in the 2-D synthetic image and assigning a semantically segmented image to the pixels of the identified target object. The method also includes using at least one processor to generate link data associating each semantically segmented image with a corresponding one of the 2-D synthetic images, and storing a training image set including the 2-D synthetic image, the semantically segmented image, and the link data.

[0007] A computing system for generating a training image set is described herein. The computing system includes at least one processor in communication with at least one storage device. The at least one processor is configured to retrieve model data corresponding to a three-dimensional (3-D) model of a target object from at least one storage device, and create multiple two-dimensional (2-D) synthetic images based on the model data. The 2-D synthetic image includes multiple views of the 3-D model. The at least one processor is also configured to create multiple semantic segmentation images by identifying multiple pixels defining a target object in the 2-D synthetic image and assigning a semantic segmentation image to the pixels of the identified target object. The at least one processor is also configured to generate link data associating the semantic segmentation image with a corresponding one of the 2-D synthetic images, and store a training image set including the 2-D synthetic image, the semantic segmentation image, and the link data.

[0008] A non-transitory computer-readable storage medium is described herein, on which computer-executable instructions for generating a training image set are included. When executed by at least one processor in communication with at least one storage device, the computer-executable instructions cause the at least one processor to retrieve model data corresponding to a three-dimensional (3-D) model of a target object from at least one storage device, and create multiple two-dimensional (2-D) synthetic images based on the model data. The 2-D synthetic image includes multiple views of the 3-D model. The computer-executable instructions also cause the at least one processor to create multiple semantically segmented images by identifying multiple pixels defining a target object in a 2-D synthetic image and assigning a semantically segmented image to the pixels of the identified target object. The computer-executable instructions also cause the at least one processor to generate link data that associates each semantically segmented image with a corresponding 2-D synthetic image, and store a training image set including 2-D synthetic images, semantically segmented images, and link data.

[0009] There are various improvements of the features related to the above aspects. Other features may also be incorporated into the above aspects. These improvements and additional features may exist alone or in any combination. For example, the various features discussed below with respect to any of the illustrated examples may be incorporated into any of the above aspects alone or in any combination. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1A is a schematic diagram of a computer-generated view of a three-dimensional model of a first example target object according to an embodiment.

[0011] Figure 1B is a schematic diagram of a computer-generated view of a three-dimensional model of a second example target object according to an embodiment.

[0012] Figure 2A According to the implementation method Figure 1AA schematic representation of a portion of an example training image set of a first example target object is shown, including 2-D synthetic images for multiple angular views and a corresponding semantically segmented image for each 2-D synthetic image. Figure 2A (a) is according to the implementation method Figure 2A A detailed diagram showing the Figure 2A Multiple pixels of the semantic segmentation image shown.

[0013] Figure 2B According to the implementation method Figure 1B A schematic representation of a portion of another example training image set of a second example target object is shown, including 2-D synthetic images for multiple angular views and a corresponding semantically segmented image for each 2-D synthetic image. Figure 2B (b) is according to the implementation method Figure 2B A detailed diagram showing the Figure 2B Multiple pixels of the semantic segmentation image shown.

[0014] Figure 2C According to the implementation method Figure 1A A schematic representation of a portion of an example training image set of a first example target object is shown, including 2-D synthetic images for multiple range views.

[0015] Figure 3 According to an embodiment, it can be used to generate a training image set, such as Figure 2A and 2B A schematic block diagram of an example of a computing system of a set of examples shown, according to such as Figure 1A and Figure 1B A three-dimensional model such as the one shown.

[0016] Figure 4A Depending on the implementation, Figure 3 An example of a 2-D synthetic base image generated by the system shown, based on a Figure 1A and Figure 1B A three-dimensional model of a target object such as the three-dimensional model shown.

[0017] Figure 4B is an example of occlusion image data corresponding to an occluder according to an embodiment.

[0018] Figure 4C Depending on the implementation, Figure 3 An example of a 2-D composite image created by the system shown, according to Figure 4A The 2-D synthetic base image shown and Figure 4B The occlusion image data shown.

[0019] Figure 5ADepending on the implementation, Figure 3 Another example of a 2-D synthetic base image created by the system shown, based on a Figure 1A and Figure 1B A three-dimensional model of a target object such as the three-dimensional model shown.

[0020] Figure 5B is an example of another 2-D composite image that may be created by the system shown in FIG. 4 according to an embodiment. Figure 5A The 2-D synthetic base image shown has simulated extrinsic sensor effects and background added to it.

[0021] Fig. 6A is an example of a physical test pattern that may be viewed by a camera used by a machine vision system in accordance with an embodiment.

[0022] Figure 6B is acquired by a camera according to an embodiment Fig. 6A Examples of acquired images of physical test patterns are shown.

[0023] Fig. 7A is an example of a method of generating a set of training images according to an embodiment, such as Figure 2A and Figure 2B The flowchart of the example set shown is based on Figure 1A and Figure 1B A three-dimensional model of a target object such as the three-dimensional model shown, using a method such as Figure 3 The computing system shown.

[0024] Figure 7B According to the implementation method Fig. 7A Continuation of the flowchart.

[0025] Figure 7C According to the implementation method Fig. 7A and 7B Continuation of the flowchart.

[0026] Although specific features of the various examples may be shown in some drawings and not in others, this is for convenience only. Any feature of any drawing may be referenced and / or claimed in combination with any feature of any other drawing.

[0027] Unless otherwise specified, the drawings provided herein are intended to illustrate features of examples of the present disclosure. It is believed that these features can be applied to a variety of systems including one or more examples of the present disclosure. As such, the drawings are not intended to include all conventional features known to those of ordinary skill in the art required to practice the examples disclosed herein. DETAILED DESCRIPTION

[0028] An example of a computer-implemented method for generating a training image set of a target object as described herein includes creating multiple two-dimensional (2-D) composite images of a target object from multiple perspective views of a three-dimensional (3-D) model of the target object. Based on an input rotation orientation and a distance from a selected viewpoint, a simulated or "synthetic" perspective view of the target object can be obtained from the 3-D model using a suitable rendering algorithm, which is included in many commercially available computer-aided design products and need not be discussed in depth for the purposes of this disclosure.

[0029] These examples also include, for each synthetic image, identifying a plurality of pixels defining a target object in the 2-D synthetic image, automatically constructing a semantically segmented image of the 2-D synthetic image based on the identified pixels, and generating link data associating the semantically segmented image with a corresponding two-dimensional synthetic image. The example further includes storing a training image set in a memory, the training image set comprising the two-dimensional synthetic image, the semantically segmented image, and the link data. In other words, the plurality of 2-D synthetic images are stored as part of the training image set to be used as native images corresponding to the semantically segmented images during training. Thus, the systems and methods of the present disclosure replace the time and resource intensive physical camera and scene adjustments required by prior art methods for obtaining a diverse set of native images of a target object with high speed, automatic generation of precisely controlled synthetic images based on a single 3D object model.

[0030] In addition, because the 2-D synthetic image is cleanly generated from the 3-D object model, there are no uncontrolled or unnecessary elements in the image, so the pixels representing the target object in each 2-D synthetic image can be accurately identified by the computing system, and the computing system can apply appropriate algorithms to locate the pixels that are part of the target object. The computing system automatically assigns or "fills" the semantic segmentation image to the identified pixels without the intervention of operator input. The semantic segmentation image is, for example, a pre-selected color or other identifier of a class of objects to which the target object belongs, and can therefore be used by the training algorithm to help the machine vision system correctly identify the target object as belonging to the class. Therefore, the system and method of the present disclosure replaces the painstaking effort and subjective judgment required by the prior art methods for semantic segmentation labeling with high-speed, automatic generation of semantic segmentation images that are objectively accurate on a pixel-by-pixel basis, because each semantic segmentation is based on the pixels of the target object in the 2-D synthetic image.

[0031] Unless otherwise indicated, the terms "first," "second," etc. are used herein merely as labels and are not intended to impose order, position, or hierarchy requirements on the items to which these terms refer. Furthermore, for example, reference to a "second" item does not require or preclude the existence of, for example, a "first" or lower-numbered item or a "third" or higher-numbered item.

[0032] Figure 1A1 is a schematic diagram of a computer-generated view of a three-dimensional model of a first instance of target object 100, and more specifically, aircraft 102. For purposes of this disclosure, a “target object” refers to an object that a machine vision system is to be trained to discern from a general background. In some examples, aircraft 102 is an instance of target object 100, which may be a computer-generated view of a first instance of a three-dimensional model of a first instance of aircraft 102, such as a machine vision system 362. Figure 3 Other unmanned vehicles, such as land, sea or space vehicles, may also be considered. Figure 1B 1 is a schematic diagram of a second instance of target object 100, more specifically a computer-generated view of a three-dimensional model of road sign 104. In some examples, road sign 104 is an instance of target object 100, which may be viewed using, for example, machine vision system 362 (e.g., Figure 3 Although target objects 100 are represented herein as airplanes 102 and road signs 104, it should be understood that the present disclosure encompasses any target object 100 that may be associated with machine vision system 362 in any application. For example, machine vision system 362 is used in applications other than an aerial flight environment or an automotive environment.

[0033] Figure 3 3 is a schematic block diagram of an example computing system 300 for generating a training image set 350 for a target object 100. Specifically, the computing system 300 includes at least one processor 302 configured to generate a semantically segmented image 352 based on a three-dimensional (3-D) model of the target object 100. The semantically segmented image 352 is a two-dimensional image that includes the semantically segmented image 202 (in Figure 2A and 2B ). In the example, the semantically segmented image 202 is a pre-selected color assigned to the pixels 342 corresponding to the target object 100.

[0034] The at least one processor 302 may be configured to perform one or more operations described herein via programming the at least one processor 302. For example, the at least one processor 302 is programmed to execute the model data manipulation module 320, the image processing module 322, the data linking module 324, and / or other suitable modules that perform the steps described below.

[0035] In this example, the computing system 300 includes at least one storage device 304 operably coupled to at least one processor 302, and the at least one processor 302 is programmed by encoding operations as one or more computer executable instructions 306 and providing the computer executable instructions 306 in the at least one storage device 304. In some examples, the computer executable instructions are provided as a computer program product by embodying the instructions on a non-transitory computer readable storage medium. The at least one processor 302 includes, but is not limited to, for example, a graphics card processor, another type of microprocessor, a microcontroller, or other equivalent processing devices that are capable of executing commands or programs of computer readable data for executing the model data manipulation module 320, the image processing module 322, the data link module 324, and / or other suitable modules as described below. In some examples, the at least one processor includes, for example, but not limited to, multiple processing units coupled in a multi-core configuration. In some examples, the at least one processor 302 includes a graphics card processor programmed to execute the image processing module 322 and a general-purpose microprocessor programmed to execute the model data manipulation module 320, the data link module 324, and / or other suitable modules.

[0036] In this example, at least one storage device 304 includes one or more devices that enable storage and retrieval of information such as executable instructions and / or other data. At least one storage device 304 includes one or more computer-readable media, such as, but not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), solid-state disk, hard disk, read-only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), and / or non-volatile RAM (NVRAM) memory. The above memory types are examples only, and are therefore not limited to the types of memory that can be used as at least one storage device 304. At least one storage device 304 is configured to store, but is not limited to, application source code, application object code, source code portions of interest, object code portions of interest, configuration data, execution events, and / or any other type of data.

[0037] In this example, computing system 300 includes a display device 372 coupled to at least one processor 302. Display device 372 presents information, such as a user interface, to an operator of computing system 300. In some examples, display device 372 includes a display adapter (not shown) coupled to a display device (not shown), such as a cathode ray tube (CRT), a liquid crystal display (LCD), an organic LED (OLED) display, and / or an "electronic ink" display. In some examples, display device 372 includes one or more display devices.

[0038] In this example, computing system 300 includes a user input interface 370. User input interface 370 is coupled to at least one processor 302 and receives input from an operator of computing system 300. User input interface 370 includes, for example, a keyboard, a pointing device, a mouse, a stylus, a touch-sensitive panel, such as but not limited to a touchpad or a touch screen, and / or an audio input interface, such as but not limited to a microphone. A single component such as a touch screen can function as both display device 372 and user input interface 370.

[0039] In some examples, the computing system 300 includes a communication interface 374. The communication interface 374 is coupled to the at least one processor 302 and is configured to be coupled to communicate with one or more remote devices, such as, but not limited to, a network server, and to perform input and output operations for such devices. For example, the communication interface 374 includes, but is not limited to, a wired network adapter, a wireless network adapter, a mobile telecommunications adapter, a serial communication adapter, and / or a parallel communication adapter. The communication interface 374 receives data from one or more remote devices and / or sends data to one or more remote devices for use by the at least one processor 302 and / or storage by the at least one storage device 304.

[0040] In other examples, computing system 300 is implemented in any suitable manner that enables computing system 300 to perform the steps described herein.

[0041] refer to Figures 1A to 3 , at least one processor 302 can access model data 308 corresponding to the three-dimensional model of the target object 100. In this example, the model data 308 is stored in the at least one storage device 304, and the at least one storage device 304 is programmed to retrieve the model data 308 from the at least one storage device 304. For example, the model data 308 is developed using a suitable computer-aided design (CAD) application and formatted as CAD data. For another example, the model data 308 is developed from a physical instance of the target object 100 using a suitable photogrammetry technique or a suitable 3-D scanning technique.

[0042] In this example, model data 308 includes element data 310 associated with at least one element 110 of target object 100 . Figure 1A Elements 110 of aircraft 102 identified in FIG. 1 include wings, a fuselage, and landing gear, and Figure 1B The elements 110 of the road sign 104 identified in FIG. 1 include a base and text. It should be understood that both the aircraft 102 and the road sign 104 are included in Figure 1A and Figure 1BIn this example, the model data 308 includes a unique identifier 312 and a geometric shape 316 for each element 110 of the target object 100. More specifically, the unique identifier 312 of each element 110 is different from the unique identifier 312 of each other element 110 included in the model data 308.

[0043] The geometry 316 enables the at least one processor 302 to determine the position and orientation of the corresponding element 310 from any viewing angle and distance, such as when drawing an image of the target object 100 from the model data 308. In some examples, the target object 100 includes at least one configurable element 110, that is, the geometry 316 can change based on the configuration of the element 110. For example, for the target object 100 is an aircraft 102, one of the elements 110 is an extendable or retractable landing gear. In this example, for the element 110 of the configurable target object 100, the element data 310 includes configuration data 314, and the configuration data 314 defines how to modify the geometry 316 for each configuration of the element 110. Alternatively, the target object 100 (e.g., the road sign 104) does not include a configurable element 110, so the model data 308 does not include the configuration data 314.

[0044] The at least one processor 302 is further programmed to generate a plurality of views 240 of the target object 100 from the model data 308 using the view definition data 332. Each view 240 of the plurality of views of the target object 100 corresponds to a perspective rendering of the target object 100 for a corresponding orientation of the target object 100 and a distance from the target object 100. In other words, the view 240 is a perspective view of the target object 100, images of which will be included in the training image set 350 for training the machine vision system 362. More specifically, the training image set 350 includes a 2-D synthetic image 340 and a corresponding semantically segmented image 352 for each view 240, and the training algorithm 360 is programmed to use the corresponding 2-D synthetic image pairs 340, which function as native "real" images and semantically segmented images 352, to train the machine vision system 362 to recognize the target object 100.

[0045] The view definition data 332 defines the viewing angle of each view 240 of the target object 100. In this example, the view definition data 332 is stored in the at least one storage device 304, and the at least one processor 302 is programmed to retrieve the view definition data 332. In some examples, the view definition data 332 includes a predefined set of orientations and / or distances for which the views 240 of the target object 100 are to be generated to create a comprehensive training image set 350. For example, the at least one processor 302 is programmed to receive the view definition data 332 as a list of values ​​in, for example, a text file.

[0046] For example, the view definition data 332 defines the plurality of angular viewing angles 242, and the at least one processor 302 is programmed to create the 2-D composite image 340 by rotating the 3-D model of the target object 100 through the plurality of angular viewing angles 242 and creating one of the 2-D composite images at each of the plurality of angular viewing angles 242. For example, Figure 2A As shown in the upper left figure 240, each angular view angle 242 is defined according to orthogonal axes x, y and z based on an angular rotation away from the initial forward-facing position of the target object 100, wherein the rotation rotates the pitch angle about the x-axis, the yaw angle about the y-axis and the roll angle about the z-axis (in Figure 2A , the z-axis is off-page). In some instances, the view definition data 332 includes an angular viewing angle 242 as a set of absolute rotation values ​​about predetermined orthogonal axes x, y, and z, such as pitch, yaw, and roll, and the at least one processor 302 is programmed to sequentially apply each value set to the model data 308 to generate each view 240. Alternatively, the view definition data 332 includes a rotation increment for each predefined axis x, y, and z, and the at least one processor 302 is programmed to sequentially advance the view of the 3D model by the corresponding rotation increment to generate each view. Although the terms "pitch," "yaw," and "roll" are commonly used in aeronautical applications, the angular viewing angle 242 defined using these rotations about the predefined axes x, y, z may be applied to any target object 100, such as Figure 2B Alternatively, any predetermined axis and / or coordinate system suitable for defining the plurality of angular viewing angles 242 may be used.

[0047] For example, Figure 2A is a schematic representation of a portion of an example training image set 350 of aircraft 102, and Figure 2B 3 is a schematic representation of a portion of another example training image set 350 for road signs 104. Each training image set 350 includes 2-D synthetic images 340 of a corresponding target object 100 for a plurality of views 240, each view 240 corresponding to an angle view map 242; and a corresponding semantic segmentation image 352 for each 2-D synthetic image 340. More specifically, in Figure 2A , each 2-D composite image 340 is a rendering of aircraft 102 at a particular angle of view 242 based on model data 308 of aircraft 102. Figure 2B , each 2-D composite image 340 is a rendering of the model data 308 from the road sign 104 at a particular angular view 242 .

[0048] For another example, the view definition data 332 further defines a plurality of distance viewing angles 244, and the at least one processor 302 is programmed to create the 2-D composite image 340 by scaling the 3-D model through the plurality of distance viewing angles 244 and, for each distance viewing angle 244, creating one of the 2-D composite images 340 at each of the plurality of angle viewing angles 242. As applied to the above example, Figure 2A and Figure 2B The training image set 350 shown in will further include repetitions of at least some of the illustrated angle views 242 at closer and / or farther distance distance views 244. As an example, Figure 2C is a schematic representation of a portion of a training image set 350 for a target object 100 embodied as an aircraft 102, including 2-D synthetic images 340 generated for views 240 defined by constant angle views 242 at multiple distance views 244. More specifically, Figure 2C The three views 240 shown in FIG. 2 include distance perspectives 244 from left to right, corresponding to progressively closer vantage points, ie, progressively decreasing distances, from the viewpoint to the target object.

[0049] In some instances, the view definition data 332 includes the distance perspectives 244 as a set of absolute distance values, and the at least one processor 302 is programmed to sequentially apply each distance value to the model data 308 at each angular perspective 242 to generate views 240 for the training image set 350. Alternatively, the view definition data 332 includes distance increments, and the at least one processor 302 is programmed to sequentially scale the view of the 3-D model by the distance increment at each angular perspective 242 to generate each view 240.

[0050] Alternatively, view definition data 332 defines any suitable set of views 240 of target object 100 in any suitable manner that enables training image set 350 to function as described herein.

[0051] In this example, to create each 2-D composite image 340, a suitable rendering algorithm is applied to the model data 308 using the corresponding view 240 defined by the view definition data 332 (e.g., defined by a particular orientation of and distance from the target object 100) as the "viewpoint" from which the model data 308 is rendered. Suitable rendering algorithms, such as but not limited to ray tracing algorithms, are known and need not be discussed in depth for purposes of this disclosure. The UNITY PRO product sold by Unity Technologies ApS of San Francisco, California provides a ray tracing algorithm for 3-D models. In addition, the PiXYZ software plug-in provided by Metaverse Technologies France can be used with the UNITY PRO product to import model data 308 formatted as computer-aided design (CAD) data.

[0052] As described above, prior art systems for creating training image sets would require an operator to capture native images of physical instances of the target object 100 and orchestrate a time and resource intensive series of physical camera placements and / or target object adjustments to obtain a diverse set of views 240 of the target object. The systems and methods of the present disclosure replace these manual efforts with high-speed, automatic generation of precisely controlled 2-D synthetic images 340 at any number of views 240 from model data 308 for use as native images for the training image set 350.

[0053] Figure 2A (a) in the equation is Figure 2A Detail of part C, Figure 2B (b) in the Figure 2B Detailed views of portions D of the 2-D images, each showing a plurality of pixels 342 used to construct a semantically segmented image 352. In this example, the process of creating the semantically segmented image 352 includes using at least one processor 302 to identify pixels 342 that define a target object 100 in a 2-D synthetic image 340. For example, in the process of generating a 2-D synthetic image 340 for a particular view 240, the image processing module 322 applies a rendering algorithm to the model data 308. Correctly rendering the view 240 requires that the image processing module 322 identify all pixels 342 that show any portion of the target object 100. To create the semantically segmented image 352, at least one processor is programmed to automatically assign the semantically segmented image 202, such as a pre-selected color, to the pixels 342 of the target object 100 identified by the image processing module 322.

[0054] The semantically segmented images 202 are associated with corresponding object categories in the training algorithm 360. For example, the training algorithm 360 is programmed to associate bright red with the category of “airplane” and Figure 2AIn the illustrated example, at least one processor 302 assigns semantically segmented image 202 to pixel 342 corresponding to aircraft 102 as a bright red color to create semantically segmented image 352. Additionally or alternatively, at least one processor 302 is programmed to assign any other suitable semantically segmented image to identified pixel 342, such as an encoded class identifier other than a preselected color, which enables training set 350 to function as described herein.

[0055] In this example, after pixels 342 in 2-D synthetic image 340 are appropriately modified to create semantically segmented image 352, semantically segmented image 352 is stored as a portable network graphic (PNG) image file in at least one storage device 304. Alternatively, semantically segmented image 352 is stored in any suitable format that enables training image set 350 to function as described herein.

[0056] In some examples, the process of creating the semantically segmented image 352 includes the at least one processor 302 identifying pixels 342 along a boundary of the target object 100. Boundary pixels are pixels 342 of the target object 100 that are adjacent to any pixel that is not part of the target object 100, or are located on the edge of the 2-D synthetic image 340. In other words, for a 2-D synthetic image 340 where the target object 100 partially extends beyond the edge of the rendered image, the boundary pixels 342 are defined as including pixels 342 of the target object 100 along the edge of the rendered image. The at least one processor 302 is programmed to "fill in" the pixels 342 in the 2-D synthetic image 340 that are surrounded by the boundary pixels. Alternatively, the pixels 342 of the target object 100 in the 2-D synthetic image 340 are identified in any suitable manner to create the semantically segmented image 352.

[0057] Because at least one processor 302 automatically determines pixels 342 within the target object 100 based on the rendering of the 2-D synthetic image 340 and automatically colors the set of pixels 342 using the pre-selected color 202, the computing system 300 generates a semantically segmented image 352 with pixel-level accuracy in a high-speed process, which can reduce (e.g., eliminate) the operation of manually learning or acquiring images.

[0058] As described above, in some examples, the target object 100 includes at least one element 110 that can be presented in multiple configurations, and the corresponding element data 110 of the model data 308 includes configuration data 314 representing each configuration. The at least one processor 302 is programmed to create the 2-D synthetic image 340 by selectively applying the configuration data 314 for each configuration to the 3-D model and creating a 2-D synthetic image 340 for each view 240 for each configuration. For example, if the model data 308 of the aircraft 102 includes the configuration data 314 of the element 110 as a landing gear, then in Figure 2A In the example training image set 350 of FIG. 1 , the illustrated 2-D synthetic image 340 and semantically segmented image 352 pair with the landing gear extended will be repeated with the landing gear retracted.

[0059] The process of generating the training image set 350 further includes using the at least one processor 302 to generate link data associating each semantically segmented image 352 with the corresponding 2-D synthetic image 340. In this example, the at least one processor 302 generates the link data by including at least a portion of the file name of the corresponding 2-D synthetic image 340 in the file name of the file containing the semantically segmented image 352. For example, the matching portion of the file name includes the orientation angle and / or distance of the view 240 for which the 2-D synthetic image 340 and the semantically segmented image 352 were generated. Alternatively, the at least one processor 302 generates the link data by including such link information in the corresponding file headers or other metadata of the corresponding 2-D synthetic image 340 and the semantically segmented image 352. The training algorithm 360 is configured to parse the training image set 350 for corresponding 2-D synthetic image 340 and semantically segmented image 352 pairs by comparing the corresponding file names, file headers, and / or other metadata.

[0060] Alternatively, at least one processor 302 generates the link data as a separate data structure that includes information (e.g., file path, memory location) sufficient to identify where the corresponding 2-D synthetic image 340 and semantic segmentation image 352 pairs are located. For example, the data structure is a comma separated value (CSV) file or a table in a database. The training algorithm 360 is configured to parse the training image set 350 for the corresponding 2-D synthetic image 340 and semantic segmentation image 352 pairs by parsing the records in the data structure for the link information.

[0061] Alternatively, at least one processor 302 generates linking data associating each semantically segmented image 352 with a corresponding 2-D synthetic image 340 in any suitable manner that enables training image set 350 to function as described herein.

[0062] Figure 4Ais an example of a 2-D composite base image 400 created from model data 308 and view definition data 332 . Figure 4B is an example of occlusion image data 450 corresponding to occluder 452 . Figure 4C An example of a 2-D composite image 340 created from a 2-D composite base image 400 and occlusion image data 450 is referred to as 2-D composite image 440. In the example of FIG.

[0063] An occluder 452 is any object that partially blocks or "occludes" the view of the target object 100. As described above, in some instances, the machine vision system 362 may need to recognize the target object 100 even if the target object 100 is partially blocked or blocked by another object. Therefore, the training image set 350 can be generated to include some views 240 in which the target object 100 appears to be blocked by the occluder 452. In some instances, the at least one processor 302 is programmed to store the occlusion image data 450 in the at least one storage device 304 and retrieve the occlusion image data 450 for use in creating the 2-D synthetic image 340 with the partially blocked view 240 of the target object 100. More specifically, the at least one processor 302 creates a plurality of 2-D synthetic base images 400, each base image 400 corresponding to one of the plurality of views 240 of the 3-D model defined by the view definition data 332, in the same manner as described above with respect to the 2-D synthetic image 340 with the unobstructed / unchanged view. In some instances, the 2-D synthetic base image 400 is also used as a 2-D synthetic image 340 having an unobstructed / unchanged view 240 of the target object 100 , in addition to being used to create other 2-D synthetic images 340 having altered views 240 of the target object 100 that are obstructed or otherwise.

[0064] At least one processor 302 creates a 2-D composite image 340 having an occluded view 240 of the target object 100 by superimposing the occlusion image data 450 on each 2-D composite base image 400. For example, Figure 4CAs shown, tree branches 454 are superimposed above the aircraft 102, simulating an occluded or blocked view 240 of the aircraft 102 from behind the tree. In the same manner, a semantically segmented image 352 for a 2-D synthetic image 340 of an obstructed view may also be created by superimposing the occlusion image data 450 on the semantically segmented image 352 created from the 2-D synthetic base image 400. Thus, the computer system 300 may automatically create a training image set 350 including obstructed views 240 of the target object 100 based on any number of views 240 defined by the view definition data 332, without having to spend a lot of time and arduous tasks to adjust the physical camera position to capture native occluded images of the target object at different angles and distances.

[0065] Figure 5A is an example of another 2-D composite base image 500 created from the model data 308 and the view definition data 332 as described above. Figure 5B is an example of a 2-D composite image 340 created from a 2-D composite base image 500 and having simulated extrinsic sensor effects 502 and a background 504 added thereto.

[0066] In some cases, the image machine vision system 362 sees in the field of view through its one or more cameras varies based on background aspects, such as weather or time of day (e.g., bright sunlight or cloudy, low illumination). In some cases, the presentation of training images with a single consistent background can potentially reduce the effectiveness of the training image set 350. In some instances, the at least one processor 302 is further programmed to create the 2-D composite image 340 with one or more simulated backgrounds 504 for at least some views 240 of the target object 100. The simulated backgrounds 504 can each correspond to, for example, a landscape, (e.g., urban or rural), a time of day, and / or weather conditions. Figure 5B The example background 504 shown in 2-D includes leaf trees 506 extending from a trunk located along the right-hand side of the 2-D composite image 340 , and two cumulus clouds 505 in the upper left quadrant of the 2-D composite image 340 .

[0067] More specifically, at least one processor 302 is programmed to create a plurality of 2-D synthetic base images 500, each 2-D synthetic base image 500 corresponding to one of the views 240 from the view definition data 332, and then superimpose each 2-D synthetic base image 500 on a different background 504 to create a 2-D synthetic image 340. For example, the 2-D synthetic base images 500 are created from the model data 308 at the various views 240 defined by the view definition data 332, such as using a suitable ray tracing algorithm as described above, and each 2-D synthetic base image 500 is superimposed on each different background 504 to generate a corresponding 2-D synthetic image 340 or each background 504. Thus, the computing system 300 is able to generate a training set of images of the target object 100 under various environmental conditions without waiting for or relying on physical background scene adjustments.

[0068] In this example, at least one processor 302 identifies portions of the 2-D synthetic base image 500 that do not correspond to the target object 100, marks these portions so that pixels of the background 504 can be deleted when the 2-D synthetic base image 500 is superimposed on the background 504 (e.g., the at least one processor 302 considers these portions as a "green screen"), and superimposes the modified 2-D synthetic base image 500 on the background 504 to create a corresponding 2-D synthetic image 340. For example, Figure 5B Shown from Figure 5A 2-D synthetic base image 500 , where the target object 100 is a road sign 104 , which is superimposed on an example background 504 including trees 506 and clouds 505 .

[0069] Additionally or alternatively, a lighting effects algorithm is applied to the 2-D synthetic base image 500 to create a 2-D synthetic image 340 that corresponds to a particular time of day or other ambient light conditions in which the target object 100 may be viewed. Alternatively, the at least one processor 302 is programmed to create the 2-D synthetic image 340 with the background 504 from the 2-D synthetic base image 500 in any suitable manner that enables the training image set 350 to function as described herein, or is not programmed to include the background 504.

[0070] In some cases, the image machine vision system 362 sees in the field of view through its one or more cameras varies based on extrinsic sensor effects (i.e., camera effects caused by the environment), such as lens glare caused by sunlight, deformation caused by raindrops on the camera lens. In some cases, presenting training images that do not account for extrinsic sensor effects may reduce the effectiveness of the training image set 350. In some instances, at least one processor 302 is programmed to apply at least one simulated extrinsic sensor effect 502 to the 2-D synthetic base image 500 to create at least some of the 2-D synthetic images 340. In other words, at least one processor 302 is programmed to intentionally distort the non-distorted synthetic image to simulate real-world environmental sensor effects.

[0071] More specifically, as described above, at least one processor 302 is programmed to create 2-D synthetic base images 500 based on the view definition data 332, and then apply the simulated extrinsic sensor effects 502 to the 2-D synthetic base images 500 to create one or more 2-D synthetic images 340 based on each 2-D synthetic base image 500. For example, the simulated extrinsic sensor effects 502 are generated by applying a corresponding extrinsic sensor effect algorithm to the 2-D synthetic base image 500. Figure 5B In the example shown, a "lens flare" extrinsic sensor effect 502 is added to Figure 5A 500 and is created by a lighting modification algorithm propagating from a point near the upper right corner of the 2-D synthetic base image 500. In this example, in combination with the background 504, the extrinsic sensor effect 502 simulates the effect of direct sunlight shining through the branches of the tree 506 into the camera lens. For another example, the raindrop algorithm simulates raindrops on the camera lens by applying local distortion to one or more random locations on the 2-D synthetic base image 500 to create the corresponding 2-D synthetic image 340. Alternatively, the at least one processor 302 is programmed to take such extrinsic sensor effects into account when creating the 2-D synthetic images 340 in any suitable manner to enable the training image set 350 to function as described herein, or is not programmed to take into account the extrinsic sensor effects 502.

[0072] Similarly, in some examples, at least one processor 302 is further programmed to apply simulated intrinsic sensor effects in the 2-D composite image 340 . Fig. 6A is an example of a physical test pattern 600 that may be viewed by a camera used by machine vision system 362 . Figure 6Bis an example of an acquired image 650 of a physical test pattern 600 acquired by a camera. The physical test pattern 600 is a checkerboard pattern defined by horizontal straight lines 602 and vertical straight lines 604. However, due to intrinsic sensor effects of the camera, the acquired image 650 of the physical test pattern 600 is distorted such that the horizontal straight lines 602 become curved horizontal lines 652 and the vertical straight lines 604 become distorted vertical lines 654. Without further modification, the 2-D synthetic image 340 generated from the model data 308 using, for example, a ray tracing algorithm does not include distortions caused by intrinsic sensor effects, such as those shown in the acquired image 650. In some instances, a mismatch between the 2-D synthetic images 340 and the semantically segmented images 352 in the training image set 350 and what the image machine vision system 362 sees in its field of view through one or more cameras may result, which may reduce the effectiveness of the training image set 350.

[0073] In this example, the at least one processor 302 is programmed to take such intrinsic sensor effects into account when creating the 2-D synthetic image 340. In other words, the at least one processor 302 is programmed to intentionally distort the non-distorted synthetic image. More specifically, as described above, the at least one processor 302 is programmed to create the 2-D synthetic base image 500 according to the view definition data 332, and then apply the simulated intrinsic sensor effects to the 2-D synthetic base image to create the 2-D synthetic image 340. For example, the 2-D synthetic base image is created at the view 240 defined by the view definition data 332 according to the model data 308, such as using a suitable ray tracing algorithm as described above, and then the intrinsic sensor effect mapping algorithm is applied to the 2-D synthetic base image to generate the 2-D synthetic image 340.

[0074] For example, one such intrinsic sensor effect mapping algorithm maps the x and y coordinates of each 2-D synthetic base image to xd and yd coordinates to generate a corresponding 2-D synthetic image 340 according to the following formula:

[0075] xd = x(1+k1 r2+k2 r4); and

[0076] yd=y(1+k1 r2+k2 r4);

[0077] Where r = the radius from the center of the 2D synthetic base image to (x, y).

[0078] For example, factors k1 and k2 for a particular camera are determined by comparing an acquired image 650 captured by the camera to a physical test pattern 600. For a camera with a fisheye lens, an additional factor k3 may also be determined and applied using a suitable expansion mapping. Alternatively, at least one processor 302 is programmed to account for such intrinsic sensor effects in creating 2-D synthetic images 340 in any suitable manner to enable training image set 350 to function as described herein, or is not programmed to include intrinsic sensor effects.

[0079] Additionally or alternatively, at least one processor 302 is programmed to apply any suitable additional processing in creating and / or processing the 2-D synthetic images 340 and / or the semantically segmented images 352. For example, at least some known instances of the training algorithm 360 perform better on a training image set 350 having a relatively low image resolution. At least one processor 302 may be programmed to reduce the image resolution of the 2-D synthetic images 340 prior to storage in the training image set 350, and to have the semantically segmented images 352 with a correspondingly reduced image resolution. For another example, at least some known instances of the training algorithm 360 perform better on a training image set 350 that does not include a large amount of unsegmented background images. At least one processor 302 may be programmed to crop the 2-D synthetic images 340 prior to storage in the training image set 350, and to crop the semantically segmented images 352 accordingly.

[0080] In some examples, the at least one processor 302 is programmed to create at least one additional label for each 2-D synthetic image 340. For example, the at least one additional label includes a perspective label corresponding to the perspective, such as angular rotation and / or distance, of a corresponding one of the views 240 of the 3-D model used to create the 2-D synthetic image 340. For another example, the at least one other label includes a background label corresponding to one of the plurality of backgrounds 504 used to create the 2-D synthetic image 340. For another example, the at least one additional label includes an occlusion label corresponding to an occlusion 452 for which the occlusion image data 450 is included in the 2-D synthetic image 340. Additionally or alternatively, the at least one additional label includes any suitable identifier of a feature of the corresponding 2-D synthetic image 340 or data used to create the corresponding 2-D synthetic image 340. The at least one processor 302 is programmed to generate additional linking data associating the at least one additional label with the corresponding 2-D synthetic image 340, and store the additional linking data in the at least one storage device 304 as part of the training image set 350. In some embodiments, the additional linking data is the inclusion of at least one additional tag in at least one of the file name and the file header of the corresponding 2-D composite image 340. Alternatively, the additional linking data includes at least one additional tag in other metadata of the 2-D composite image 340 or in a separate data structure for providing linking data.

[0081] Fig. 7A is a flow chart of an example method 700 for generating a training image set, such as training image set 350. As described above, method 700 is implemented on a computing system including at least one processor in communication with at least one storage device, such as computing system 300 including at least one processor 302 in communication with at least one storage device 304. Figure 7B and 7C yes Fig. 7A Continuation of the flowchart.

[0082] In this example, method 700 includes retrieving 702 model data 308 corresponding to a three-dimensional (3-D) model of target object 100 from at least one storage device 304. In some examples, method 700 also includes retrieving 704 occlusion image data 450 corresponding to occluder 452 from at least one storage device 304.

[0083] In this example, method 700 also includes creating 706 plurality of 2-D composite images 340 based on model data 308. Plurality of 2-D composite images 340 includes plurality of views 240 of 3-D model.

[0084] In some instances, the plurality of views 240 includes a plurality of angular perspectives 242, and the step of creating 706 the 2-D composite image 340 includes rotating 708 the 3-D model through the plurality of angular perspectives 242 and creating 710 one of the 2-D composite images 340 at each of the plurality of angular perspectives 242. In some instances, the plurality of views 240 also includes a plurality of distance perspectives 244, and the step of creating 706 the 2-D composite image 340 includes scaling 712 the 3-D model through the plurality of distance perspectives, and creating 714 one of the 2-D composite images 340 at each of the plurality of angular perspectives 242 for each of the distance perspectives 244.

[0085] In some examples, creating 706 the 2-D synthetic image 340 includes creating 716 a plurality of 2-D synthetic base images 400 and / or 500, each corresponding to one of the views 240 of the 3-D model. In some examples, creating 706 the plurality of 2-D synthetic images 340 also includes overlaying 718 the 2-D synthetic base images 340 onto a plurality of different backgrounds 504. Additionally or alternatively, creating 706 the plurality of 2-D synthetic images 340 also includes applying 720 an extrinsic sensor effect to the 2-D synthetic base image 400 or 500. Additionally or alternatively, creating 706 the plurality of 2-D synthetic images 340 also includes applying 722 an intrinsic sensor effect to the 2-D synthetic base image 400 or 500. Additionally or alternatively, creating 706 the plurality of 2-D synthetic images 340 includes overlaying 726 the occlusion image data 450 onto each of the 2-D synthetic base images 340.

[0086] In some examples, target object 100 can be presented in multiple configurations, such as by including configurable elements 110, and model data 308 includes configuration data 314 representing each configuration. Creating 706 a plurality of 2-D composite images 340 includes selectively applying 728 the configuration data 314 for each configuration to the 3-D model and creating 730 a plurality of 2-D composite images 340 at each of a plurality of views of each configuration.

[0087] In this example, method 700 also includes creating 732 a plurality of semantically segmented images 352, including the steps of identifying 734 a plurality of pixels 342 defining a target object 100 in a 2-D composite image 340; and assigning 736 the semantically segmented image 202, such as a preselected color, to the identified pixels 342 of the target object.

[0088] In this example, the method 700 also includes generating 738 link data 354 that associates each semantically segmented image 352 with a corresponding one of the 2-D synthetic images 340. In some examples, the step of generating 738 the link data 354 includes including at least a portion of the file name of the corresponding 2-D synthetic image 340 in the file name of the semantically segmented image 352. Additionally or alternatively, the step of generating 738 the link data 354 includes including such link information in the corresponding file headers or other metadata of the corresponding 2-D synthetic image 340 and the semantically segmented image 352.

[0089] In some instances, the method 700 further includes, for at least one of the 2-D synthetic images 340, creating 742 at least one additional label, the at least one additional label including at least one of a viewpoint label, a background label, and an occlusion label; generating 744 additional linking data associating the at least one additional label with the corresponding at least one 2-D synthetic image 340; and storing 746 the additional linking data as part of the training image set 350. In some instances, the step of storing 746 the additional linking data includes including 748 the at least one additional label in at least one of a file name and a file header of the at least one 2-D synthetic image 340. Additionally or alternatively, the step of storing 746 the additional linking data includes including such linking information in a corresponding file header or other metadata of the corresponding at least one 2-D synthetic image 340 and / or the semantically segmented image 352.

[0090] In this example, method 700 also includes storing 750 , for example in at least one storage device 304 , a training image set 350 , the training image set 350 including the 2-D synthetic image 340 , the semantically segmented image 352 , and the linkage data 354 .

[0091] The above-described examples of computer-implemented methods and systems for generating a training image set for a target object utilize 2-D synthetic images created from views of a computer model of the target object. These examples include: identifying a plurality of boundary pixels in the 2-D synthetic image that define the perimeter of the target object in the 2-D synthetic image; automatically constructing a semantic segmentation image for the 2-D synthetic image based on the boundary pixels; and generating link data to associate the semantic segmentation image with the corresponding 2-D synthetic image. The example also includes storing a training image set that includes a 2-D synthetic image, a semantic segmentation image, and link data. In other words, the 2-D synthetic image is used as a native image stored as part of the training image set. In some instances, at least one of occlusion image data, non-intrinsic sensor effects, intrinsic sensor effects, and a changing background image is added to the 2-D synthetic image to create a more powerful training image set.

[0092] Example technical effects of the methods, systems, and devices described herein include at least one of the following: (a) high-speed, automatic generation of semantic segmentation images for training image sets; (b) generation of objectively accurate semantic segmentation images on a pixel-by-pixel basis; (c) generation of training image sets of target objects under various environmental conditions without waiting for or relying on physical scene adjustments; (d) simulation of various non-intrinsic and / or intrinsic sensor effects in computer-generated semantic segmentation images without the need for any physical camera and / or physical scene adjustments.

[0093] Furthermore, the present disclosure includes implementations according to the following clauses:

[0094] 1. A method (700) for generating a set of training images (350) using automatic semantic segmentation labels, the method being implemented on a computing system (300), the computing system comprising at least one processor (302) in communication with at least one storage device (304), the method comprising using the at least one processor to:

[0095] retrieving target object model data (308) corresponding to a three-dimensional (3-D) model of the target object (100) from at least one storage device;

[0096] creating a plurality of two-dimensional (2-D) composite images (340) from the target object model data, each 2-D composite image corresponding to one of the plurality of views of the 3-D model;

[0097] Creating a plurality of semantic segmentation images (352) for each 2-D composite image includes the following steps:

[0098] identifying a plurality of boundary pixels (342), the boundary pixels defining a perimeter of a target object in the 2-D composite image; and

[0099] automatically establishing one of the semantic segmentation images corresponding to the 2-D synthetic image based on the boundary pixels;

[0100] generating link data (354) associating the semantically segmented image with a corresponding 2-D synthetic image; and

[0101] A training image set including 2-D synthetic images, semantically segmented images, and linkage data is stored in at least one storage device.

[0102] 2. The method (700) of clause 1, wherein the plurality of views comprises a plurality of angular perspectives, and wherein creating the plurality of 2-D composite images (340) comprises using at least one processor (302) to:

[0103] Rotate 3-D models through multiple angles of view; and

[0104] One of the 2-D composite images is created at each of the plurality of angular viewing angles.

[0105] 3. The method (700) of clause 2, wherein the plurality of views further comprises a plurality of distance perspectives, the method further comprising: using at least one processor (302) to create the plurality of 2-D composite images (340) by:

[0106] Scaling 3-D models through multiple distance perspectives; and

[0107] For each distance view angle, one of the 2-D composite images is created at each of a plurality of angular views.

[0108] 4. A method (700) according to any of clauses 1 to 3, wherein the target object (100) is capable of being presented in a plurality of configurations, and the target object model data (308) includes configuration data (314) representing each configuration, the method further comprising creating, using at least one processor (302), a plurality of 2-D composite images (340) by:

[0109] selectively applying the configuration data for each configuration to the 3-D model; and

[0110] For each configuration, a plurality of 2-D composite images are created at each of the plurality of views.

[0111] 5. The method (700) of any of clauses 1-4, further comprising: using at least one processor (302) to create a plurality of 2-D composite images (340) by:

[0112] creating a plurality of 2-D synthetic base images (500), each 2-D synthetic base image corresponding to one of the views of the 3-D model; and

[0113] Each 2-D composite base image is superimposed on a plurality of different backgrounds (504) to create a 2-D composite image.

[0114] 6. The method (700) of any of clauses 1-5, further comprising: using at least one processor (302) to:

[0115] retrieving occlusion image data (450) corresponding to the occluder (452) from at least one storage device (304); and

[0116] A plurality of 2-D composite images (340) are created by:

[0117] creating a plurality of 2-D synthetic base images (500), wherein each 2-D synthetic base image corresponds to one of the plurality of views of the 3-D model; and

[0118] The occlusion image data is superimposed on each 2-D synthesis base image to create a 2-D synthesis image.

[0119] 7. The method (700) of any of clauses 1-6, further comprising: using at least one processor (302) to create a plurality of 2-D composite images (340) by:

[0120] creating a plurality of 2-D synthetic base images (500), wherein each 2-D synthetic base image corresponds to one of the plurality of views of the 3-D model; and

[0121] At least one of an extrinsic sensor effect and an intrinsic sensor effect is applied to the 2-D synthetic base image to create a 2-D synthetic image.

[0122] 8. The method (700) of any one of clauses 1 to 7, further comprising: for each 2-D composite image (340), using at least one processor (302) to:

[0123] Create at least one additional tag, including at least one of the following:

[0124] a perspective tag corresponding to the perspective of the corresponding view of the 3-D model used to create the 2-D composite image;

[0125] a background label corresponding to one of a plurality of backgrounds used to create the 2-D composite image; and

[0126] an occlusion tag corresponding to an occluder (452) comprising occlusion image data (450) in the 2-D composite image;

[0127] generating additional linking data (354) associating at least one additional tag with the corresponding 2-D composite image; and

[0128] The additional link data is stored in at least one storage device (304) as part of the training image set (350).

[0129] 9. The method (700) of clause 8, wherein, in the additional linking data (354), at least one additional tag is included in at least one of a file name and a file header of the 2-D composite image (340).

[0130] 10. The method (700) of any one of clauses 1 to 9 further comprises: using at least one processor (302) to generate link data (354) by including at least a portion of the file name of the corresponding 2-D composite image (340) in the file name of the semantically segmented image (352).

[0131] 11. A computing system (300) for generating a training image set (350) using automatic semantic segmentation labels, the computing system comprising at least one processor (302) in communication with at least one storage device (304), wherein the at least one processor is configured to:

[0132] retrieving target object model data (308) corresponding to a three-dimensional (3-D) model of the target object (100) from at least one storage device;

[0133] creating a plurality of two-dimensional (2-D) composite images (340) from the target object model data, each 2-D composite image corresponding to one of the plurality of views of the 3-D model;

[0134] Creating a plurality of semantic segmentation images (352) for each 2-D composite image includes the following steps:

[0135] identifying a plurality of boundary pixels (342), the boundary pixels defining a perimeter of a target object in the 2-D composite image; and

[0136] automatically establishing one of the semantic segmentation images corresponding to the 2-D synthetic image based on the boundary pixels;

[0137] generating link data (354) associating the semantically segmented image with a corresponding 2-D synthetic image; and

[0138] A training image set including 2-D synthetic images, semantically segmented images, and linkage data is stored in at least one storage device.

[0139] 12. The computing system (300) of clause 11, wherein the plurality of views comprises a plurality of angular perspectives, and wherein the at least one processor (302) is further configured to create the plurality of 2-D composite images (340) by:

[0140] Rotate 3-D models through multiple angles of view; and

[0141] One of the 2-D composite images is created at each of the plurality of angular viewing angles.

[0142] 13. The computing system (300) of clause 12, wherein the plurality of views further comprises a plurality of distance perspectives, and wherein the at least one processor (302) is further configured to create the plurality of 2-D composite images (340) by:

[0143] Scaling 3-D models through multiple distance perspectives; and

[0144] For each distance view, a 2-D composite image is created at each of the plurality of angular views.

[0145] 14. A computing system (300) according to any of clauses 11 to 13, wherein the target object (100) can be presented in a plurality of configurations, and the target object model data (308) includes configuration data (314) representing each configuration, and wherein the at least one processor is further configured to create the plurality of 2-D composite images (340) by:

[0146] selectively applying the configuration data for each configuration to the 3-D model; and

[0147] For each configuration, a plurality of 2-D composite images are created at each of the plurality of views.

[0148] 15. The computing system (300) of any one of clauses 11 to 14, wherein the at least one processor (302) is further configured to create the plurality of 2-D composite images (340) by:

[0149] creating a plurality of 2-D synthetic base images (500), each 2-D synthetic base image corresponding to one of the views of the 3-D model; and

[0150] Each 2-D composite base image is superimposed on a plurality of different backgrounds (504) to create a 2-D composite image.

[0151] 16. The computing system (300) of any of clauses 11 to 15, wherein the at least one processor (302) is further configured to retrieve occlusion image data (450) corresponding to the occluder (452) from the at least one storage device (304), and wherein the processor is further configured to create the plurality of 2-D composite images (340) by:

[0152] creating a plurality of 2-D synthetic base images (500), wherein each 2-D synthetic base image corresponds to one of the plurality of views of the 3-D model; and

[0153] The occlusion image data is superimposed on each 2-D synthesis base image to create a 2-D synthesis image.

[0154] 17. The computing system (300) of any one of clauses 11 to 16, wherein the at least one processor (302) is further configured to create the plurality of 2-D composite images (340) by:

[0155] creating a plurality of 2-D synthetic base images (500), wherein each 2-D synthetic base image corresponds to one of the plurality of views of the 3-D model; and

[0156] At least one of an extrinsic sensor effect and an intrinsic sensor effect is applied to the 2-D synthetic base image to create a 2-D synthetic image.

[0157] 18. The computing system (300) of any of clauses 11 to 17, wherein, for each of the 2-D composite images (340), the at least one processor (302) is further configured to:

[0158] Create at least one additional tag, including at least one of the following:

[0159] a perspective tag corresponding to the perspective of the corresponding view of the 3-D model used to create the 2-D composite image;

[0160] a background label corresponding to one of a plurality of backgrounds used to create the 2-D composite image; and

[0161] an occlusion tag corresponding to an occluder (452) comprising occlusion image data (450) in the 2-D composite image;

[0162] generating additional linking data (354) associating at least one additional tag with the corresponding 2-D composite image; and

[0163] The additional link data is stored in at least one storage device (304) as part of the training image set (350).

[0164] 19. A computing system (300) according to any of clauses 11 to 18, wherein at least one processor (302) is further configured to generate link data (354) by including at least a portion of a file name of a corresponding 2-D synthetic image (340) in at least one of a file name and a file header of a file containing the semantically segmented image (352).

[0165] 20. A non-transitory computer-readable storage medium having computer-executable instructions embodied thereon for generating a training image set (350) using automatic semantic segmentation labeling, wherein when executed by at least one processor (302) in communication with at least one storage device (304), the computer-executable instructions cause the at least one processor to:

[0166] retrieving target object model data (308) corresponding to a three-dimensional (3-D) model of the target object (100) from at least one storage device;

[0167] creating a plurality of two-dimensional (2-D) composite images (340) from the target object model data, each 2-D composite image corresponding to one of the plurality of views of the 3-D model;

[0168] Creating a plurality of semantic segmentation images (352) for each 2-D composite image includes the following steps:

[0169] identifying, by a processor, a plurality of boundary pixels (342), the boundary pixels defining a perimeter of a target object in the 2-D composite image;

[0170] automatically establishing, by the processor, one of the semantically segmented images corresponding to the 2-D composite image based on the boundary pixels; and

[0171] generating link data (354) associating the semantically segmented image with a corresponding 2-D synthetic image; and

[0172] A training image set including 2-D synthetic images, semantically segmented images, and linkage data is stored in at least one storage device.

[0173] The systems and methods described herein are not limited to the specific examples described herein, but rather, components of the systems and / or steps of the methods may be utilized independently and separately from other components and / or steps described herein.

[0174] As used herein, an element or step described in the singular and beginning with the word "a" or "an" should be understood to not exclude multiple elements or steps, unless such exclusion is explicitly stated. In addition, reference to "an example" or "an example" of the present disclosure is not intended to be interpreted as excluding the existence of additional examples that also include the described features.

[0175] This written description uses examples to disclose various examples, including the best mode, to enable those skilled in the art to practice those examples, including making and using any device or system and performing any combined method. The patentable scope is defined by the technical solution and may include other examples that occur to those skilled in the art. If such other examples have structural elements that are not different from the literal language of the technical solution, or if they include equivalent structural elements that are not substantially different from the literal language of the technical solution, it is intended that these other examples be included within the scope of the technical solution.

Claims

1. A method (700) for generating a training image set (350) using automatic semantic segmentation labels, the method being implemented on a computing system (300) comprising at least one processor (302) in communication with at least one storage device (304), the method comprising using the at least one processor to: Target object model data (308) corresponding to the three-dimensional model of the target object (100) is retrieved from the at least one storage device, wherein: The target object (100) comprises at least one configurable element, the geometry (316) being variable based on the configuration of the at least one element, wherein the target object model data (308) comprises configuration data (314) defining how the geometry of the target object is modified for each configuration of the at least one element; Creating a plurality of two-dimensional composite images (340) from the target object model data, each of the two-dimensional composite images corresponding to one of the plurality of views of the three-dimensional model, wherein creating the plurality of the two-dimensional composite images (340) comprises the following steps: selectively applying configuration data of each of said configurations to said three-dimensional model; and for each of said configurations, creating a plurality of said two-dimensional composite images at each of a plurality of said views; Creating a plurality of semantically segmented images (352) for each two-dimensional composite image includes the following steps: identifying a plurality of boundary pixels (342) that define a perimeter of the target object in the two-dimensional composite image; and Automatically establishing the semantic segmentation image corresponding to the two-dimensional composite image based on the boundary pixels, including assigning the semantic segmentation image to pixels surrounded by the boundary pixels; generating link data associating the semantically segmented image with a corresponding two-dimensional composite image (354); and A training image set including the two-dimensional composite image, the semantic segmentation image and the link data is stored in the at least one storage device.

2. The method (700) according to claim 1, wherein: The plurality of views comprises a plurality of angular perspectives, and wherein creating the plurality of two-dimensional composite images (340) comprises using the at least one processor (302) to: rotating said three-dimensional model through a plurality of said angular viewing angles; and One of the two-dimensional composite images is created at each of a plurality of the angular viewing angles.

3. The method (700) according to claim 2, wherein: The plurality of views further include a plurality of distance viewing angles, and the method further includes using the at least one processor (302) to create the plurality of two-dimensional composite images (340) by: scaling the three-dimensional model through a plurality of the distance viewing angles; and For each of the distance viewing angles, one of the two-dimensional composite images is created at each of a plurality of the angular viewing angles.

4. The method (700) of any one of claims 1 to 3, further comprising creating, using the at least one processor (302), a plurality of the two-dimensional composite images (340) by: creating a plurality of two-dimensional synthetic base images (500), each of the two-dimensional synthetic base images corresponding to one of the views of the three-dimensional model; and Each of the two-dimensional composite base images is superimposed on a plurality of different backgrounds (504) to create the two-dimensional composite image.

5. The method (700) according to any one of claims 1 to 3, further comprising using the at least one processor (302) to: retrieving occlusion image data (450) corresponding to the occluder (452) from the at least one storage device (304); and The plurality of two-dimensional composite images (340) are created by: A plurality of two-dimensional composite base images are created (500), wherein: Each of the two-dimensional synthetic base images corresponds to one of a plurality of views of the three-dimensional model; and The occlusion image data is superimposed on each of the two-dimensional synthesis base images to create the two-dimensional synthesis image.

6. The method (700) of any one of claims 1 to 3, further comprising creating, using the at least one processor (302), a plurality of the two-dimensional composite images (340) by: A plurality of two-dimensional composite base images are created (500), wherein: Each of the two-dimensional synthetic base images corresponds to one of a plurality of views of the three-dimensional model; and At least one of an extrinsic sensor effect and an intrinsic sensor effect is applied to the two-dimensional synthetic base image to create the two-dimensional synthetic image.

7. The method (700) of any one of claims 1 to 3, further comprising, for each of the two-dimensional composite images (340), using the at least one processor (302) to: At least one additional tag is created, the at least one additional tag comprising at least one of the following: a viewing angle tag corresponding to a viewing angle of a corresponding view of the three-dimensional model used to create the two-dimensional composite image; a background tag corresponding to one of a plurality of backgrounds (504) used to create the two-dimensional composite image; and An occlusion tag corresponding to an occlusion object (452), wherein the two-dimensional composite image includes occlusion image data (450) for the occlusion object (452); generating additional linking data associating the at least one additional tag with the corresponding two-dimensional composite image (354); and The additional link data is stored in the at least one storage device (304) as part of the training image set (350).

8. The method (700) according to claim 7, wherein: In the additional link data (354), the at least one additional tag is included in at least one of a file name and a file header of the two-dimensional composite image (340).

9. The method (700) according to any one of claims 1 to 3, further comprising: The link data (354) is generated using the at least one processor (302) by including at least a portion of a file name of the corresponding two-dimensional composite image (340) in a file name of the semantically segmented image (352).

10. A computing system (300) for generating a training image set (350) using automatic semantic segmentation labels, the computing system comprising at least one processor (302) in communication with at least one storage device (304), wherein: The at least one processor is configured to: retrieving target object model data (308) corresponding to a three-dimensional model of a target object (100) from the at least one storage device, wherein the target object (100) comprises at least one configurable element, a geometry (316) being variable based on a configuration of the at least one element, wherein the target object model data (308) comprises configuration data (314) defining how the geometry of the target object is modified for each configuration of the at least one element; Creating a plurality of two-dimensional composite images (340) from the target object model data, each of the two-dimensional composite images corresponding to one of the plurality of views of the three-dimensional model, wherein creating the plurality of the two-dimensional composite images (340) comprises the following steps: selectively applying configuration data of each of said configurations to said three-dimensional model; and for each of said configurations, creating a plurality of said two-dimensional composite images at each of a plurality of said views; Creating a plurality of semantically segmented images (352) for each two-dimensional composite image includes the following steps: identifying a plurality of boundary pixels (342), the boundary pixels defining a perimeter of the target object in the two-dimensional composite image; and automatically establishing the semantically segmented image corresponding to the two-dimensional composite image based on the boundary pixels, including assigning the semantically segmented image to pixels surrounded by the boundary pixels; generating link data associating the semantically segmented image with the corresponding two-dimensional composite image (354); and A training image set including the two-dimensional composite image, the semantic segmentation image and the link data is stored in the at least one storage device.

11. The computing system (300) of claim 10, wherein: The plurality of views comprises a plurality of angular viewing angles, and wherein the at least one processor (302) is further configured to create the plurality of two-dimensional composite images (340) by: rotating said three-dimensional model through a plurality of said angular viewing angles; and One of the two-dimensional composite images is created at each of a plurality of the angular viewing angles.

12. The computing system (300) of claim 11, wherein: The plurality of views further comprises a plurality of distance viewing angles, and wherein the at least one processor (302) is further configured to create the plurality of two-dimensional composite images (340) by: scaling said three-dimensional model through a plurality of said distance viewing angles; and For each of the distance viewing angles, one of the two-dimensional composite images is created at each of a plurality of the angular viewing angles.

13. The computing system (300) according to any one of claims 10 to 12, in, The at least one processor (302) is further configured to create a plurality of the two-dimensional composite images (340) by: creating a plurality of two-dimensional synthetic base images (500), each of the two-dimensional synthetic base images corresponding to one of the views of the three-dimensional model; and Each of the two-dimensional composite base images is superimposed on a plurality of different backgrounds (504) to create the two-dimensional composite image.

14. The computing system (300) according to any one of claims 10 to 12, in, The at least one processor (302) is further configured to create a plurality of the two-dimensional composite images (340) by: creating a plurality of two-dimensional synthetic base images (500), each of the two-dimensional synthetic base images corresponding to one of the views of the three-dimensional model; and At least one of an extrinsic sensor effect and an intrinsic sensor effect is applied to the two-dimensional synthetic base image to create the two-dimensional synthetic image.

15. The computing system (300) according to any one of claims 10 to 12, wherein: For each of the two-dimensional composite images (340), the at least one processor (302) is further configured to: At least one additional tag is created, the at least one additional tag comprising at least one of the following: a viewing angle tag corresponding to a viewing angle of a corresponding view of the three-dimensional model used to create the two-dimensional composite image; a background tag corresponding to one of a plurality of backgrounds used to create the two-dimensional composite image; and An occlusion tag corresponding to an occlusion object (452), wherein the two-dimensional composite image includes occlusion image data (450) for the occlusion object (452); generating additional linking data associating the at least one additional tag with the corresponding two-dimensional composite image (354); and storing the additional link data in the at least one storage device (304) as part of the training image set (350); and The at least one processor (302) is further configured to generate the link data (354) by including at least a portion of a file name of the corresponding two-dimensional composite image (340) in at least one of a file name and a file header of a file containing the semantically segmented image (352).

Images