Systems and methods for reducing the amount of data storage in machine learning

Abstract

A method for reducing the size of machine learning data stored in a vehicle storage device, the method comprising: Receiving, by a processor, data comprising a three-dimensional model; wherein the three-dimensional model is 3D data acquired by one or more sensors; Analyzing, by the processor, a scene within the three-dimensional model to identify an object of interest and one or more additional objects; Capturing, by the processor, the object of interest and the one or more additional objects as multiple two-dimensional images; Removing, by the processor, the one or more additional objects from the multiple two-dimensional images to obtain multiple cropped two-dimensional images of the object of interest;The processor stores multiple cropped two-dimensional images of the object of interest in the vehicle storage device; the processor receives a background image from an image sensor, wherein the background image is intended to surround the object of interest in subsequent process steps; the processor categorizes the background image based on one or more attributes of the background; the processor adds a selected attribute to the multiple cropped two-dimensional images, wherein the selected attribute is at least one of a weather attribute, a lighting attribute, and a shading attribute; the processor synthesizes one or more synthesized images from the multiple cropped two-dimensional images with the added attribute and the background image;and annotating, by the processor, of one or more synthesized images with the exact position of the object of interest.

Description

CROSS REFERENCEThis application claims priority to U.S. patent application Ser. No. 15 / 919,376 filed Mar. 13, 2018, the contents of which are hereby incorporated by reference in their entirety.Technical FieldThe present invention relates generally to reducing the storage space required for training data used for machine learning, and more particularly relates to systems and methods for generating two-dimensional images from three-dimensional models, wherein the two-dimensional images are used as training data.BACKGROUNDCurrently, computing devices, particularly those associated with machine learning for autonomous and semi-autonomous vehicles, store three-dimensional (3D) models that are used as training data for machine learning. This type of training data is computationally intensive and requires a lot of memory space. Therefore, it may be difficult for on-board vehicle processors to effectively utilize the training data.SUMMARYIn one embodiment, a method for reducing the amount of machine learning data is provided. The method includes receiving, by a processing device, data including a three-dimensional model. A scene within the three-dimensional model is analyzed by the processing device to identify an object of interest and one or more additional objects. The processing device captures the object of interest and the one or more additional objects as a plurality of two-dimensional images. The processing device removes the one or more additional objects from the plurality of two-dimensional images to obtain a plurality of scrambled two-dimensional images. The processing device stores the plurality of scrambled two-dimensional images.In another embodiment, a system is provided for reducing the amount of data storage used to include training data in machine learning. The system includes a processing device and a non-transitory processor readable storage medium. The non-transitory processor readable storage medium includes one or more programming instructions that, when executed, cause the processing device to receive data such as a three-dimensional model. The non-transitory processor readable storage medium further includes one or more programming instructions that, when executed, cause the processing device to analyze a scene within the three-dimensional model to identify an object of interest and one or more additional objects, capture the object of interest and the one or more additional objects as multiple two-dimensional images, remove the one or more additional objects from the multiple two-dimensional images to obtain multiple scrambled two-dimensional images, and store the multiple scrambled two-dimensional images.In another embodiment, a system is provided for reducing the amount of data storage used to include training data in machine learning. The system includes a processing device, an image capture device communicatively coupled to the processing device, an external device communicatively coupled to the processing device, and a non-transitory processor readable storage medium. The non-transitory processor readable storage medium includes one or more programming instructions that, when executed, cause the processing device to receive data such as a three-dimensional model, analyze a scene within the three-dimensional model to identify an object of interest and one or more additional objects, and capture the object of interest and the one or more additional objects as multiple two-dimensional images. Further, the non-transitory processor readable storage medium includes one or more programming instructions that, when executed, cause the processing device to transmit the plurality of two-dimensional images to the external device and receive a signal from the external device indicating that the plurality of two-dimensional images are realistic images. The non-transitory processor readable storage medium further includes one or more programming instructions that, when executed, cause the processing device to remove the one or more additional objects from the plurality of two-dimensional images to obtain a plurality of scrambled two-dimensional images and store the plurality of scrambled two-dimensional images.These and additional features provided by embodiments described herein will be better understood from the following detailed description taken in conjunction with the drawings.BRIEF DESCRIPTION OF THE DRAWINGSThe embodiments illustrated in the drawings are illustrative and exemplary in nature and are not intended to limit the subject matter defined by the claims. The following detailed description of the exemplary embodiments may be understood when read in conjunction with the following drawings, in which like structures are denoted by the same reference numerals and in which: FIG. 1 schematically illustrates an example network including components for a system that generates two-dimensional images from three-dimensional models, in accordance with one or more of the embodiments shown and described herein; FIG. 2A schematically illustrates example hardware components of a vehicle computing device that may be used to generate two-dimensional images from three-dimensional models, in accordance with one or more of the embodiments shown and described herein; FIG. 2B schematically illustrates an example memory component including example logic components, in accordance with one or more of the embodiments shown and described herein; FIG. 2C schematically illustrates an example data storage device including example data components, in accordance with one or more of the embodiments shown and described herein; FIG. 3 shows a flowchart of an example method for generating two-dimensional images from three-dimensional models, in accordance with one or more of the embodiments shown and described herein; FIG. 4A schematically illustrates an example view of a scrambled two-dimensional image, in accordance with one or more of the embodiments shown and described herein; FIG. 4B schematically illustrates an example view of a background image in accordance with one or more of the embodiments shown and described herein; FIG. 4C schematically illustrates an example view of weather attributes fused to a scrambled two-dimensional image, in accordance with one or more of the embodiments shown and described herein; and FIG. 4D schematically illustrates an example view of a synthesized image in accordance with one or more of the embodiments shown and described herein.DETAILED DESCRIPTIONWith general reference to the figures, the embodiments described herein relate to systems and methods for reducing the amount of data storage and / or processing power required for machine learning training data by replacing three-dimensional (3D) models with multiple two-dimensional (2D) images. The 3D models may be, for example, design models such as CAD drawings, 3D data captured by one or more sensors, and / or the like. The 3D models are analyzed by a semantic segmentation method. In particular, a specific scene within the 3D model may be analyzed to identify an object of interest, and one or more additional objects may also be identified. Multiple 2D images of the object of interest and one or more additional objects are captured from different angles, zoom levels, and / or the like. The plurality of 2D images are cropped such that only the object of interest remains and the one or more additional objects are removed.Moreover, a background image is captured by an image capturing device. The background image itself originates from real-time live images of a background surrounding an object of interest, such as a vehicle. The image capturing device may capture images in real time with respect to weather and environmental information. The background image includes information from the background surrounding the object of interest, such as other objects and attributes. The background image is categorized, and selected attributes such as a weather attribute, an illumination attribute, and / or a shading attribute are then added to the plurality of scrambled 2D images. The plurality of scrambled 2D images and the selected attribute are synthesized and annotated such that the 2D images can be used as training data for machine learning. As a result, the 2D images replace 3D models as training data, which reduces the amount of data storage used for the training data. Moreover, the systems and methods described herein may generate data that may be utilized by relatively less powerful processors, such as processors located in in-vehicle computing devices.Referring to the drawings, FIG. 1 shows an example network 100 including components for a system for generating scrambled 2D images from 3D models used as training data for machine learning, in accordance with embodiments shown and described herein. As illustrated in FIG. 1, a computer network 105 may include a wide area network (WAN) such as the Internet, a local area network (LAN), a mobile communication network, a public service telephone network (PSTN), a personal area network (PAN), a metropolitan area network (MAN), a virtual private network (VPN), and / or another network. The computer network 105 may generally be configured to electronically connect one or more devices, such as computing devices and / or components thereof. Example devices may include, but are not limited to, a vehicle 110, a user computing device 120, and a server computing device 130.The vehicle 110 may generally be any vehicle having one or more onboard computing devices, particularly computing devices including hardware for processing data, storing data, and capturing images in or around the vehicle. Thus, the vehicle 110 and / or components thereof may perform one or more computing functions, such as receiving data, capturing image data (e.g., 2D images) with an image capture device 115, processing the captured images, storing the processed images, and providing processed images to machine learning algorithms, as will be described in more detail herein.The user computing device 120 may be generally used as an interface between a user and the other components connected to the computer network 105. Thus, the user computing device 120 may be used to perform one or more user-oriented functions, such as receiving one or more inputs from a user or providing information to the user, as described in more detail herein. Accordingly, the user computing device 120 may include at least one display and / or input hardware, as will be described in more detail herein. In the event that the server computing device 130 requires monitoring, updating, and / or correction, the user computing device 120 may be configured to provide the desired monitoring, updating, and / or correction. The user computing device 120 may also be used to input additional data into a corpus of data stored on the server computing device 130. For example, the user computing device 120 may be. Software programming or the like related to viewing, interpreting, and / or capturing 2D images and 3D models, as well as software programming related to generating 2D images from 3D models.The server computing device 130 may receive data from one or more sources, generate data, store data, index data, search for data, and / or transmit data to the user computing device 120 and / or the vehicle 110 (or components thereof). In some embodiments, the server computing device 130 may use one or more machine learning algorithms used for the purposes of analyzing data received from the vehicle 110, such as multiple 2D images captured, as described in more detail herein. Additionally, the server computing device 130 may be used to generate data, such as for executing one or more semantic segmentation functions, as will be described in more detail herein. It should be appreciated that the vehicle onboard computer systems may function with the server computing device such that the vehicle onboard computer systems perform one or more semantic segmentation functions and / or may use one or more machine learning algorithms.It should be understood that although the user computing device 120 is shown as a personal computer and the server computing device 130 is shown as a server, these are not limiting examples. In some embodiments, any type of computing device (e.g., mobile computing device, personal computer, server, etc.) may be used for any of these components. Moreover, although each of these computing devices is shown in FIG. 1 as a single piece of hardware, this is also just one example. Both the user computing device 120 and the server computing device 130 may represent multiple computers, servers, databases, components, and / or the like.FIG. 2A schematically illustrates example hardware components of the vehicle 110 that may be used in machine learning data generation. Although the components illustrated in FIG. 2A are described with respect to the vehicle 110, it should be appreciated that similar components may also be used for the user computing device 120 (FIG. 1 ) and / or the server computing device 130 (FIG. 1 ) without departing from the scope of the present invention.The vehicle 110 may include a vehicle component 200 having a non-transitory computer readable medium for executing the various processes described herein and embodied as hardware, software, and / or firmware, in accordance with the embodiments shown and described herein. Although in some embodiments the vehicle component 200 may be configured as a general purpose computer with the required hardware, software, and / or firmware, in other embodiments the vehicle component 200 may also be configured as a special purpose computer specifically configured to perform the functionality described herein. For example, the vehicle component 200 may be a device particularly configured to use machine learning algorithms for purposes of autonomous or semi-autonomous control of the vehicle 110. In another example, the vehicle component 200 may be a device particularly configured to use machine learning algorithms for purposes of monitoring a driver and / or passenger of the vehicle, predicting movements or actions of the driver and / or passenger, and providing feedback and / or executing commands. It should be appreciated that still other machine learning applications in vehicles are contemplated. In embodiments where the vehicle component 200 is a general purpose computer, the systems and methods described herein provide a mechanism for improving functionality by replacing 3D models with 2D images so that machine learning processes can be performed by less powerful processors, by processors requiring less resources, and / or in less time.Referring again to FIG. 2A, the vehicle component 200 may generally be a vehicle-integrated computer system. In some embodiments, the vehicle component 200 may be comprised of multiple vehicle computer systems.As also illustrated in FIG. 2A, the vehicle component 200 may include a processing device 204, I / O hardware 208, network interface hardware 210, nonvolatile memory component 212, system interface 214, data storage device 216, and image capture device 115. A local interface 202, such as a bus or the like, may connect the various components together.The processing device 204, such as a computer processing unit (CPU), may be the central processing unit of the vehicle component 200 that performs computations and logic operations for executing a program. The processing device 204, alone or in conjunction with the other components, is an example processing device, an example computing device, an example processor, or a combination thereof. The processing device 204 may include any processing component configured to receive and execute instructions (e.g., from the data storage device 216 and / or the storage component 212).The memory component 212 may be configured as a transitory and / or non-transitory computer readable medium and as such may include random access memory (including SRAM, DRAM, and / or other types of random access memory), read only memory (ROM), flash memory, registers, compact discs (CD), digital versatile discs (DVD), and / or other types of memory components. The memory component 212 may include one or more programming instructions that, when executed by the processing device 204, cause the processing device 204 to execute various processes, such as the processes described herein with respect to FIG. 3. Referring again to FIG. 2A, the programming instructions stored in the memory component 212 may be embodied as multiple software logic modules, with each logic module providing programming instructions for executing one or more tasks, as described in more detail below with respect to FIG. 2B.The network interface hardware 210 may include any wired or wireless network hardware, such as a modem, a LAN port, a wireless fidelity (Wi-Fi) card, a WiMax card, mobile communication hardware, and / or other hardware for communicating with other networks and / or devices. For example, network interface hardware 210 may establish a communication link between vehicle 110 and the other components of network 100 shown in FIG. 1, including, for example, server computing device 130.Referring again to FIG. 2, the data storage device 216, which may generally be a storage medium, may include one or more data containers for storing received and / or generated data. The data storage device 216 may be any physical storage medium including, for example, a hard disk drive (HDD), a memory, removable memory, and / or the like. Example data that may be included in the data storage device 216 is described below with reference to FIG. 2C. It should be appreciated that the amount of available storage space in the data storage device 216 may be limited due to its position in the vehicle component 200 in some embodiments. Therefore, it may be necessary to minimize the size of the data stored thereon, as will be described in more detail herein.Referring again to FIG. 2A, the I / O hardware 208 may communicate information between the local interface 202 and one or more other components of the vehicle 110. For example, the I / O hardware 208 may serve as an interface between the vehicle component 200 and other components such as navigation systems, instrument clusters, cellular telephone systems, infotainment systems, and / or the like. In some embodiments, I / O hardware 208 may be used to send one or more commands to the other components of vehicle 110.The system interface 214 may generally provide the vehicle component 200 with the ability to connect to one or more external devices, such as the user computing device 120 and / or the server computing device 130 illustrated in FIG. 1. Communication with external devices may be through various communication ports (not shown). An example communication port may be connected to a communication network.Referring again to FIG. 2A, the image capturing device 115 may be communicatively coupled to the local interface 202 and may be coupled to the processing device 204 via the local interface 202. The image capturing device 115 may be any imaging device, sensor, or detector suitable for obtaining images. As used herein, the term "images" or "image" refers to video images (i.e., a sequence of consecutive images), photos (including still images extracted from video images), and / or image data. Any suitable commercially available image capture device 115 may be used without departing from the scope of the invention. In some embodiments, the image capture device 115 may be coupled to one or more other components that provide additional functionality for imaging, such as one or more sensors.The image capturing device 115 may include an objective (not shown) or may be coupled to an objective. The lens is not limited by the invention and may generally be any optical component configured to focus the light incident on the image pickup device 115 so that a useful image can be obtained. In some embodiments, the objective may be a fixed objective that cannot be adjusted. In other embodiments, the lens may be either manually or automatically adjustable by the processing device 204 to magnify, reduce, and / or adjust the focus of the light entering the image capturing device 115.Referring to FIG. 2B, in some embodiments, the program instructions included in the storage component 212 may be embodied as multiple software modules, each module providing program instructions for executing one or more tasks. For example, FIG. 2B schematically illustrates the memory component 212 including example logic components according to one or more of the embodiments shown and described herein. As shown in FIG. 2B, the storage component 212 may be configured to store various processing logic, such as operation logic 220, creation logic 222, simulation logic 224, and / or optimization logic 226 (each of which may be embodied as, for example, a computer program, firmware, or hardware). The operation logic 220 may include an operating system and / or other software for managing components of the vehicle component 200 (FIG. 2A ). Furthermore, the operation logic 220 can comprise one or more software modules for data transmission and / or data analysis.Referring again to FIG. 2B, the creation logic 222 may include one or more software modules for collecting data from one or more sources (e.g., the image capture device 115, the server computing device 130 shown in FIG. 1, and / or the like) and / or for converting data, as will be described in more detail herein. In addition, the creation logic 222 may generate and / or process the 2D images such that data may be extracted from the 2D images or such that the data may be analyzed within the 2D images or the 3D model. The simulation logic 224 may include one or more software modules to determine whether particular data is being synthesized, such as the 2D images fused to information regarding the live background images, as described in more detail herein. The simulation logic 224 may determine whether the 2D image and the 3D model are in the same domain (e.g., whether the respective 2D image and / or 3D model is semantically segmented for analysis, is a realistic image, is in original format, and / or the like). The simulation logic 224 may reside on various computing devices. For example, one or more of the functionalities and / or components described herein may be provided by the user computing device 120 and / or the server computing device 130, which may be coupled to the storage component 212 via the network 100 to enable access to the simulation logic 224. For example, the processing device 204 (FIG. 2A ) may access the simulation logic 224 to transmit and retrieve the 3D model data, and then may use the server computing device 130 and / or the like to process the 3D model data. Optimization logic 226 may include one or more software modules for determining and annotation positions, either in 2D and / or 3D, and may annotation objects, pixels, and / or the like using bounding bodies, pixel-by-pixel annotations, coordinate-based annotations, and / or the like, as discussed in more detail herein.FIG. 2C schematically illustrates a block diagram of various data included in a memory device (e.g., data storage device 216). For example, as shown in FIG. 2C, the data storage device 216 may include multiple stored 3D model data 228, such as CAD files or data processed using semantic segmentation methods, as discussed further herein. Moreover, it should be appreciated that the 3D model data 228 may be vehicle construction data or a technical model with only a single image of a specific object without background. It will also be appreciated that the 3D model data 228 may also be data captured by autonomous vehicle sensors (not shown), where the 3D model data may include multiple images of objects and / or data of objects. The 3D model data 228 may be received from, for example, the server computing device 130 (FIG. 1 ) or may be received from, for example, autonomous vehicle sensors (not shown). It should be appreciated that the plurality of stored 3D model data 228 need not be stored permanently, but instead may be stored temporarily so that the data may be extracted therefrom.The data storage device 216 may further include, for example, a plurality of 2D image data 230 that may be extracted from the 3D models, as will be discussed in greater detail herein. Moreover, the plurality of 2D image data 230 may be received by the image capture device 115 (FIG. 1 ) and / or the server computing device 130 (FIG. 1 ). The data storage device 216 further includes a plurality of captured background images 232, a plurality of captured illumination and shading attributes 234, and / or a plurality of captured weather attributes 236. The plurality of captured background images 232 and the captured attributes 234, 236 may be received by the image capture device (FIG. 2A ) and / or the server computing device 130 (FIG. 1 ). The plurality of captured background images 232 and the captured attributes 234, 236 may be captured in real-time or may be created, as will be discussed in more detail herein. In some embodiments, the captured background images 232 may include, for example, buildings, roads, bicycles, pedestrians, the entire object of interest, or a portion thereof, and / or the like. In some embodiments, the plurality of captured lighting and / or shading attributes 234 may include whether sunny, cloudy, day, night, etc. The plurality of detected weather attributes 236 may include, for example, fog, rain, snow, and the like. The background images, when captured by the image capture device 115 (FIG. 2A ), are processed by the processing device 204 (FIG. 2A ) and / or the storage component 212 (FIG. 2A ).As mentioned above, the various components described with respect to FIGS. 2A-2C may be used to execute one or more processes and / or generate data that may be executed by less powerful processors and / or processors that require less resources, such as vehicle-based computing devices.The method 300 for reducing the size of machine learning training data will now be discussed with reference to FIGS. 3 and 4A. At block 302, an input may be received. The input may generally include data including a 3D model, such as a 3D CAD file from the server computing device 130 (FIG. 1 ). The input may be received via the system interface 214 and may be analyzed by the processing device 204 (FIG. 2A ), for example, using semantic segmentation methods in block 304. The use of semantic segmentation causes the processing device 204 (FIG. 2A ) to subdivide the image into semantically meaningful parts, classify each part and / or pixel into one of the predetermined classes, such as by a predetermined number of pixels, or by object identification and / or labeling. That is, in semantic segmentation, each image, segment, and / or pixel is labeled and / or classified such that each pixel in the scene can be identified and understood. It should be appreciated that the 3D model may already be in semantic segmented format when received, so the step at block 304 may be omitted. Further, it should be appreciated that the 3D model may be a model generated from information received from multiple sensors on the vehicle 110 (FIG. 1 ), such as autonomous or semi-autonomous vehicle sensors.Referring again to FIGS. 3 and 4A, the scene of the 3D model is analyzed by the processing device 204 to identify an object of interest 404 and one or more additional objects. The scene in the 3D model may include various objects surrounding a particular object of interest, or may be free of other objects. The scene in the 3D model may be, for example, vehicle construction data or a technical model that has only a single image of a specific object without one or more additional objects. In another example, the scene in the 3D model may include one or more additional objects, such as objects included when a model is sampled from data received from autonomous vehicle sensors or the like. The object of interest 404 is captured from the 3D model in block 306. The object of interest 404 shown in FIG. 4A is an entire vehicle. However, it should be appreciated that the object of interest may be a part of the vehicle, such as a hood, a driver of the vehicle, a passenger, and / or the like. It should be appreciated that, as discussed above, in some 3D models, additional objects may surround the object of interest. For example, 3D models generated by autonomous imaging systems may include additional objects. In a road scene captured by the autonomous imaging system, additional objects such as roads, structures, pedestrians, another car, a bicycle, and / or the like may be present. In other embodiments, the scene may originate from the vehicle interior where additional objects such as a gauge, steering wheel, gear lever, navigation device, and the like may be present.At block 308, the processing device 204 (FIG. 2A ) removes any additional objects surrounding the object of interest 404 in the 2D images, as best seen in FIG. 4A, such that the plurality of scrambled 2D images 402 of the object of interest are obtained. The plurality of scrambled 2D images 402 are captured from various angles, zoom levels, and / or the like. For example, if the object of interest 404 has mixed pixels or is unclear where the object of interest and the additional objects intersect, the processing device 204 (FIG. 2A ) and the storage component 212 (FIG. 2A ) may rotate the 3D model to determine the exact shape of the object of interest 404. Therefore, the plurality of scrambled 2D images 402 match the exact shape and contour of the object of interest 404 from the 3D model.The plurality of scrambled 2D images 402 may be scrambled using various attributes, such as a road width attribute 411 (FIG. 4B ), a road angle attribute 413 (FIG. 4D ), a surrounding vehicle angle (not shown), and / or the like. These attributes can be used to establish relationships between objects of interest and 3D surfaces and / or establish relationships between the scene attributes and an object presence. As such, the attribute can be used to identify the objects of interest from additional objects, as well as to describe a variation within the class so that the object of interest can be cut out of the scene. For example, in the road scene described above, each object in the scene may be assigned an attribute so that the objects can be identified. Based on this identification, the object of interest 404 may be cut out of the scene, leaving only the object of interest and removing the one or more additional objects. Additionally, the weather attributes 418 (FIG. 4D ) and the lighting and / or shading attributes (not shown) are also removed from the plurality of scrambled 2D images 402.In FIG. 4A, one of the plurality of scrambled 2D images 402 is shown. The object of interest 404 in the 3D model includes additional objects. These objects, such as the background information 406, the weather attributes, and the illumination and / or shading attributes, are removed when the object of interest 404 is cropped. As a result, the object of interest 404 is isolated from all background information 406 and / or attributes. It should be appreciated that the object of interest 404 is illustrated as a realistic computer generated 2D image. It will also be appreciated that the image may be a high resolution image generated by a generative adversarial network (GAN).Referring again to FIGS. 3 and 4A, it may be determined in block 310 whether the plurality of scrambled 2D images 402 are realistic images. The determination may be made by an external device such as the server computing device 130 or the like. As such, the plurality of two-dimensional images may be sent to the external device for analysis, and the processing device 204 may receive a signal from the external device indicating that the plurality of two-dimensional images are realistic images. However, it is understood that the determination may also be made locally without using an external device.Realistic images are generally understood to mean images that have data sets from the real world. In this respect, this model enables a fast conversion of 3D surface measurement data of real objects into a structured representation for realistic images. That is, in real world images, many information about the environment, the illumination, surface materials, shapes, and the like are embedded. On the other hand, non-realistic images, such as the semantic segmented images from the 3D model (computer graphics image), are generally not rich in textures and shadows. Thus, if it is determined in block 310 that the image is not the realistic image, then the semantic segmented images are converted into realistic images in block 312. It should be appreciated that converting the semantic segmented images into the plurality of scrambled 2D images 402 is performed using methods known to those skilled in the art. For example, the original images can be converted into realistic images by means of neural network or deep learning. After the image is converted to the realistic image, the plurality of scrambled 2D images 402 are re-verified in block 310.Referring now to FIGS. 3 and 4B, in some embodiments, the processing device 204 may receive background images 408 from the image capturing device 115 at block 314. The background images 408 are images and information that the imaging device 115 can capture and that surrounds the object of interest 404. The background images 408 may include multiple background attributes, as best seen in FIG. 4B. In a non-limiting example, the plurality of background attributes may include a structure such as a building 410, a traffic light head 412, a sidewalk 414, a traffic light mast 416, and / or a road 420. Attributes may be defined as a particular type of data in pixels and / or data points that make up the 2D image. Thus, generally, each observation and / or data point may include many different attributes, such that the different attributes produce a representation of a given dimension (number of attributes, extracted features). Thus, a model can be used to search for the attributes and other features, which in turn can generate new attributes (extract attributes from attributes) that are more complex objects than the previous ones. Non-limiting examples of attributes may include a building structure attribute, a road width attribute, a road angle attribute, a pedestrian and / or bicycle angle attribute, another vehicle width and / or direction attribute, another vehicle angle attribute, a lighting and / or shading attribute, a weather attribute, and / or the like. It should be appreciated that since the object of interest 404 is not present in the background images 408, the entire background may be analyzed for attributes, as discussed above.It should be appreciated that the background images 408 may also include the object of interest 404 or a portion of the object of interest. As such, it should be appreciated that any type of software and / or algorithm known to those skilled in the art for pixel matching may be used to recognize and exclude the object of interest 404 from the background images 408. For example, pixel-by-pixel matching may be used to determine the object of interest 404 from the 3D model and to match the object of interest in the background images 408 such that the object of interest may be removed from the background images 408.Referring again to FIGS. 3 and 4B, at block 316, the background images 408 may be created. For example, semantic segmentation methods may be used on the background images 408 such that the processing device 204 (FIG. 2A ) and / or the server computing device 130 (FIG. 1 ) may interpret the background objects and / or the background attributes. In some embodiments, the background images 408 may be created rather than captured. Thus, in some embodiments, the background images 408 may be converted to realistic 2D images as discussed herein such that the background objects and / or the background attributes may be interpreted.Referring to FIGS. 3 and 4B-4D, at block 318, the background images 408 are categorized by the processing device 204 (FIG. 2A ) based on one or more attributes of the background. The attributes shown in FIG. 4B include the building 410, the traffic light head 412, the sidewalk 414, the traffic light mast 416, and the road 420. The attributes in FIG. 4D include the weather attributes 418, the building 410, the traffic light head 412, the sidewalk 414, the traffic light mast 416, and the road 420. In block 320, a selected attribute of the one or more attributes may be added to the plurality of scrambled 2D images 402. The attributes selected may be the weather attributes 418 (FIG. 4D ) and the lighting and / or shading attributes (not shown). These attributes are added to the plurality of scrambled 2D images 402 based on the background images 408. That is, the weather attributes 418, and the lighting and / or shading attributes, as categorized in block 316, are now added to the plurality of scrambled 2D images 402, as best seen in FIG. 4C. Thus, if the weather attributes 418 included rain at block 314, the attribute has been categorized as such at block 316. Then, in block 318, the corresponding weather attributes 418 would be added to the plurality of scrambled 2D images 402.It should be appreciated that, in block 308, the backgrounds are removed from the plurality of scrambled 2D images 402 such that the backgrounds represent the model background. As such, the remote attributes may not be real-time attributes. Thus, to provide real-time data to the plurality of scrambled 2D images 402, the captured attributes in the background images 408 of block 314 are added to the plurality of scrambled 2D images 402, as best seen in FIG. 4C.In FIG. 4C, the plurality of scrambled 2D images 402 are shown with the weather attributes 418 added. The plurality of scrambled 2D images 402 now include the object of interest 404 and the weather attributes 418. As shown, the weather attribute 418 for the current background is rain. However, it should be appreciated that, as discussed herein, based on the background images captured (FIG. 2A ) by the image capture device 115 (FIG. 2A ) or generated by the server computing device 130 (FIG. 1 ), any weather attribute may be incorporated into the plurality of scrambled 2D images 402.It will also be appreciated that attributes are used in machine learning such that the machine learning algorithms are trained to probe the data to find the structure or attributes, even if the attributes are dynamic or the structure is not known. As such, the attributes are inserted into the plurality of scrambled 2D images 402, such that machine learning can identify the attributes and learn from the data of the attributes.Referring now to FIGS. 3 and 4D, in block 322, the background images 408 are synthesized. That is, the pixels of the background images 408 are combined with the pixels of the plurality of scrambled 2D images 402, such that a two-dimensional copy of the 3D model can be generated with real-time background information. However, because the background images 408 and the plurality of scrambled 2D images 402 may originate from two different domains, the domains may be fused into a single domain during the synthesizing process. That is, the plurality of scrambled 2D images 402 and the background images 408, which are still semantically segmented or the like, may be converted into photorealistic images such that now both images are fused together and are in the same domain. It will be appreciated that the conversion of the semantically segmented images into the realistic images and / or the synthesis of images is performed using methods known to those skilled in the art, as described above.In FIG. 4D, the merging of the background images 408 and the plurality of scrambled 2D images 402 to form the synthesized 2D images 424 is shown. The object of interest 404 is positioned on the road 420, and the background still includes the same attributes, the building 410, the head 412, the light mast 416, the sidewalk 414, and the road 420. Moreover, now the weather attributes 418 and the illumination / shading attributes (not shown) display a complete plurality of 2D images for that scene.Referring again to FIGS. 3 and 4D, in block 324, the exact position of the synthesized images 424 is annotated. Such annotation may include the use of a hull annotation 422 and / or a pixel-by-pixel annotation (not shown). Hull annotation 422 refers generally to a drawn frame around pixels to be annotated when the pixels are grouped. For example, as shown in FIG. 4D, the bounding box annotation 422 may surround the pixels of the vehicle. This use of the bounding box annotation 422 may convey information such as the type of object, the size of the object, and / or the like. For example, the data of the pixels within the hull annotation 422 surrounding the object of interest 404 may be used to determine whether the object of interest is the vehicle based on the shape of the vehicle, the shading of the vehicle, the size of the vehicle, and / or the like. Further, computer programs and algorithms can interpret the data within the hull annotation 422 to determine the vehicle type, the size of the vehicle, the position of the vehicle within the scene, the relationship of the vehicle to other hulls, and / or the like. On the other hand, the pixel-by-pixel annotation (not shown) uses each pixel in the scene as the data points so that each pixel can be assigned a value so that the processing device 204 (FIG. 2A ) can use programming and / or algorithms to interpret the data, such as pixel shading, to understand the scene.It should be understood that annotation is not limited to 2D annotation (x, y), but may also use 3D annotation (x, y, z), such that annotation and synthesized images 424 are compatible with 3D machine learning algorithms. After annotation is completed, the synthesized images 424 are stored in block 326.It should be appreciated that, as described herein, while the method 300 describes the vehicle component 200 as generating the 2D images, the server computing device 130 may also generate these images. The vehicle 110 may capture the background images 408 while the background images 408 and the other data are sent by the vehicle 110 to the server computing device 130 such that the server computing device 130 executes the method 300. Moreover, the vehicle 110 does not need to capture the background images, and the server computing device 130 may use its own or other background images to generate the 2D images as described in the method 300.It will also be appreciated that the stored images include enough data sets to provide the machine learning algorithms with the same realistic data and information as much larger files that require more memory space and more powerful processors, such as 3D CAD drawings. In addition, the stored 2D images allow the 3D CAD files to be removed from the vehicle construction data memory.In other embodiments, the image data acquisition may be performed from inside the vehicle. In these embodiments, human pose detection of objects and pixel-by-pixel segmentation algorithms can be used. In one embodiment, instead of synthesizing the background images at block 314, the driver and / or passenger may be synthesized. In this respect, the posture of a human model can be annotated. Moreover, in other embodiments, as discussed above, the interior of the vehicle may be synthesized. For example, the instrument boxes of the vehicle, the navigation, the infotainment system, the shift lever, the steering wheel, and the like may be synthesized. This may provide a way to detect devices located in the vehicle and detect how the driver responds, which a driver does, how a driver moves, and / or the like. Therefore, data about driver information and responses may be captured so that these responses and interactions with the vehicle interior may be mimicked in autonomous, semi-autonomous, and / or keyless vehicles.It should now be appreciated that the methods, systems, and vehicle components described herein may serve to reduce the amount of data storage used to include training data in machine learning and / or reduce the processing power required for training data used in machine learning. The method includes using semantic segmentation to analyze a scene within the 3D model, determining an object of interest from the analysis, and cropping the object of interest into multiple 2D images. The multiple 2D images are cropped so that only objects of interest remain and the background objects are removed. The plurality of 2D images are also cropped with respect to certain attributes, such as a road width, a road angle, a surrounding vehicle angle, and / or the like. Moreover, an image pickup device takes the real-time background images. The background images are created and the objects in the background are categorized according to associated attributes. Certain attributes are added to the multiple 2D images and the other attributes are synthesized with the multiple scrambled 2D images so that a replicated 2D image is generated with the real-time background. After the synthesis, the newly generated images can be annotated and stored. This minimizes data storage because the multiple 2D images when taken together generally require less data storage than the 3D models, but still include all information of the 3D models.Although specific embodiments have been illustrated and described herein, it is to be understood that various other changes and modifications may be made without departing from the spirit and scope of the claimed subject matter. Moreover, while various aspects of the claimed subject matter have been described herein, these aspects need not be used in combination. It is therefore intended that the appended claims also encompass all such changes and modifications as fall within the scope of the claimed subject matter.

Claims

A method for reducing a size of machine learning data stored in a vehicle storage device, the method comprising: receiving, by a processor, data comprising a three-dimensional model; wherein the three-dimensional model is 3D data captured by one or more sensors; analyzing, by the processor, a scene within the three-dimensional model to identify an object of interest and one or more additional objects; capturing, by the processor, the object of interest and the one or more additional objects as multiple two-dimensional images; removing, by the processor, the one or more additional objects from the multiple two-dimensional images to obtain multiple scrambled two-dimensional images of the object of interest; storing, by the processor, the plurality of scrambled two-dimensional images of the object of interest in the vehicle storage device; receiving, by the processor, a background image from an image sensor, the background image having a background to surround the object of interest in later steps of the method; categorizing, by the processor, the background image based on one or more attributes of the background; adding, by the processor, a selected attribute to the plurality of scrambled two-dimensional images, the selected attribute being at least one of a weather attribute, an illumination attribute, and a shading attribute; synthesizing, by the processor, one or more synthesized images from the plurality of scrambled two-dimensional images with the attribute added and the background image; and annotating, by the processor, the one or more synthesized images with the exact position of the object of interest.The method of claim 1, wherein the annotating comprises creating an envelope around the object of interest.The method of claim 1, wherein the annotation comprises using a pixel-by-pixel annotation of the object of interest.The method of claim 1, wherein the plurality of scrambled two-dimensional images are scrambled using at least one of a road width attribute, a road angle attribute, or an attribute of an angle of a surrounding vehicle.The method of claim 1, wherein the plurality of background attributes are selected from a building, a traffic light head, a sidewalk, a traffic light mast, or a road.A system for reducing an amount of data storage used to include training data in machine learning, the system comprising: a processor; an image sensor communicatively coupled to the processor, the image sensor capturing a plurality of background images; and a non-transitory processor readable storage medium, the non-transitory processor readable storage medium comprising one or more programming instructions that, when executed, cause the processor to: receive, by the processor, data comprising a three-dimensional model, the three-dimensional model being 3D data captured by one or more sensors; analyze, by the processor, a scene within the three-dimensional model to identify an object of interest and one or more additional objects; capturing, by the processor, the object of interest and the one or more additional objects as a plurality of two-dimensional images; removing, by the processor, the one or more additional objects from the plurality of two-dimensional images to obtain a plurality of scrambled two-dimensional images of the object of interest; storing, by the processor, the plurality of scrambled two-dimensional images of the object of interest in a vehicle storage device; receiving, by the processor, a background image from the image sensor, the background image having a background to be later surrounded by the object of interest; categorizing, by the processor, the background image based on one or more attributes of the background; Adding, by the processor, a selected attribute to the plurality of scrambled two-dimensional images, wherein the selected attribute is at least one of a weather attribute, an illumination attribute, and a shading attribute; synthesizing, by the processor, one or more synthesized images from the plurality of scrambled two-dimensional images with the attribute added and the background image; and annotation, by the processor, the one or more synthesized images having the exact position of the object of interest.The system of claim 6, wherein the processor is caused to annotate the one or more synthesized images by creating at least one hull around the object of interest.The system of claim 6, wherein the processor is caused to annotate the one or more synthesized images at least by pixelwise annotating the object of interest.The system of claim 6, wherein the plurality of scrambled two-dimensional images are scrambled using at least one of a road width attribute, a road angle attribute, or an attribute of an angle of a surrounding vehicle.The system of claim 6, wherein the plurality of background attributes are selected from a building, a traffic light head, a sidewalk, a traffic light mast, or a road.A system for reducing an amount of data storage used to include training data in machine learning, the system comprising: a first processor; an image sensor communicatively coupled to the first processor; a second processor external to the first processor and communicatively coupled to the first processor; a non-transitory processor readable storage medium, the non-transitory processor readable storage medium comprising one or more programming instructions that, when executed, cause the first processor to: receive data comprising a three-dimensional model, wherein the three-dimensional model is 3D data captured by one or more sensors; analyze a scene within the three-dimensional model to identify an object of interest and one or more additional objects; capturing the object of interest and the one or more additional objects as a plurality of two-dimensional images; sending the plurality of two-dimensional images to the second processor; receiving a signal from the second processor indicating that the plurality of two-dimensional images are realistic images; removing the one or more additional objects from the plurality of two-dimensional images to obtain a plurality of scrambled two-dimensional images of the object of interest; storing the plurality of scrambled two-dimensional images of the object of interest in a vehicle storage device; receiving a background image from the image sensor, the background image having a background to surround the object of interest later; categorizing the background image based on one or more attributes of the background; adding a selected attribute to the plurality of scrambled two-dimensional images, wherein the selected attribute is at least one of a weather attribute, an illumination attribute, and a shading attribute; synthesizing one or more synthesized images from the plurality of scrambled two-dimensional images with the attribute added and the background image; and annotation the one or more synthesized images with the exact position of the object of interest.The system of claim 11, wherein the processing device is caused to annotate the one or more synthesized images by creating at least one hull around the object of interest or using a pixel-by-pixel annotation of the object of interest.The system of claim 11, wherein the plurality of scrambled two-dimensional images are scrambled using at least one of a road width attribute, a road angle attribute, or an attribute of an angle of a surrounding vehicle.The system of claim 11, wherein the plurality of background attributes are selected from a building, a traffic light head, a sidewalk, a traffic light mast, or a road.

Images