Method of training and / or testing machine learning system
By dynamically adjusting the spatial layout during image generation and utilizing semantic label maps and edge detection methods, the problem of balancing diversity and accuracy in image generation for machine learning systems is solved, thereby improving the quality of training and testing data for automated driving systems.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ROBERT BOSCH GMBH
- Filing Date
- 2025-10-27
- Publication Date
- 2026-05-08
AI Technical Summary
Existing machine learning systems struggle to balance the diversity and accuracy of image generation during training and testing, particularly in autonomous driving applications where insufficient edge representation of key objects leads to inadequate accuracy in important regions and insufficient diversity in unimportant regions in generated images.
By providing instructions, layout specifications, and classification specifications for generating synthetic images, image pixels are divided into groups with different relevance levels. The spatial layout is dynamically adjusted during the generation process. Masks are created using semantic label maps and edge detection methods to emphasize edge information of key classes and weaken edge information of non-key classes, thereby generating images that meet specific application requirements.
It improves the accuracy and diversity of training and testing data for machine learning systems in specific applications, enhances accuracy in critical areas and flexibility in non-critical areas, and improves the performance of automated driving systems.
Smart Images
Figure CN121998127A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to methods for training and / or testing machine learning systems. Furthermore, the invention also relates to machine learning systems, computer programs, devices, and storage media used for this purpose. Background Technology
[0002] Generative diffusion models such as Stable Diffusion, when combined with ControlNet, ushered in a new era for controlled spatial layout applications. These models, fine-tuned based on proprietary image datasets, can transform images from driving simulators into highly realistic outputs that closely resemble images from vehicle cameras. Furthermore, these images can be dynamically modified using text prompts.
[0003] The paper "High-resolution image synthesis with latent diffusion models" by Rombach, Robin, et al., Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2022, discloses a common image synthesis solution. Summary of the Invention
[0004] According to various aspects of the present invention, a method, a machine learning system, a computer program, a data processing apparatus, and a computer-readable storage medium are provided. Other features and details of the invention are disclosed in the corresponding dependent claims, description, and drawings. Features and details described in the context of the method according to the invention also correspond to the machine learning system according to the invention, the computer program according to the invention, the data processing apparatus according to the invention, and the computer-readable storage medium according to the invention, and vice versa.
[0005] According to one aspect of the invention, a method for training and / or testing a machine learning system, particularly for a specific (especially technical) application, includes the following steps (preferably executed automatically):
[0006] - Provides at least one instruction for image generation processing to generate a composite image, the composite image representing a scenario specific to the application or a specific technology application.
[0007] - Provide at least one layout specification that specifies the spatial constraints of the generation process, wherein the spatial constraints are particularly associated with different pixels of the image.
[0008] - Provide a classification specification that assigns different classes to the represented scene (especially an image).
[0009] - Divide the different classes of the classification specification into at least two groups representing different levels of relevance to the application, wherein the groups and / or relevance levels are preferably manually predefined, and preferably, each pixel in the layout specification is then mapped to one of the groups.
[0010] - Based on the defined classes, determine at least one modification to the layout specification, preferably by removing restrictions on pixels mapped to specific groups.
[0011] - Initiate the generation process to generate the image based on the at least one instruction and at least one modified layout specification.
[0012] This method allows for improved training and / or testing of machine learning systems for specific technology applications by generating synthetic images that accurately reflect the target environment. Using different groups, the method can limit generative processing to important regions of the image while allowing flexibility in less critical areas. This results in more diverse and representative training data, thereby improving the accuracy and performance of the machine learning system. By focusing on relevant classes, the system can learn to identify and interpret key information within the synthetic image, thus improving decision-making in specific applications.
[0013] Each of the steps in the above method can be performed automatically. For example, instructions and / or at least one layout specification and / or classification specification can be provided as digital data, such as based on user input. Segmentation and / or determination can be performed by a computer program using a predefined set of rules. Initiation of the generative process can be performed using a digital interface with an image generative model that uses at least one instruction and at least one modified layout specification as digital input. The at least one instruction may include text prompts and / or at least one initial image, particularly from a simulator (such as a driving simulator) and / or from a camera, etc.
[0014] Generative models, particularly generative diffusion models such as Stable Diffusion, can transform images from simulators like driving simulators into realistic outputs that closely resemble images from vehicle cameras. In other words, based on images from driving simulators, generative models can generate synthetic images with higher realism. Furthermore, the generated images can be dynamically modified using text input (especially prompts).
[0015] The generated images can then be used to train a machine learning system. This can be accomplished using two main inputs: a layout specification, specifically in the form of a conditional spatial layout, which can be constructed using edge detection methods such as Canny or HED; and instructions, such as descriptive text prompts, to define overall image features, such as snow-covered roads.
[0016] This invention specifically addresses the needs of automated driving and proposes a method for achieving a balance between variability and accuracy in image generation. One objective could be to create edge representations that emphasize key objects (such as roads and road users) while allowing the generative model to freely interpret less important areas, thereby increasing the variability of the image output without compromising the accuracy of key elements.
[0017] A combination of a base image, semantic label map, text description, and layout information (e.g., from HED or CannyEdge) can be used. A mask can be created based on the semantic label map to isolate important classes. This mask can then be used to filter layout conditions, removing edge information from non-critical classes. The layout information can then be used to generate an image. This method aims to achieve a balance between image diversity and fidelity.
[0018] The method may further include providing training and / or evaluation data for training and / or testing the machine learning system based on the generated images. Specifically, this means that the training and / or evaluation data may include the generated images and / or further modified generated images (optionally, through further processing steps). This allows for obtaining high-quality training and / or testing data for any particular application using the method according to the invention. This is particularly advantageous for applications such as object and / or scene detection based on images recorded by vehicles.
[0019] The method may further include: using the generated images as training and / or evaluation data to train and / or test the machine learning system, particularly for a specific technical application, which is in particular object and / or scene detection based on images recorded by a vehicle.
[0020] The generation process can also be subject to different spatial constraints, specifically controlled by the at least one modified layout specification. Thus, the generation process can be more constrained in spatial regions of the image where pixels are classified into at least a first group highly relevant to the application, and less constrained in spatial regions of the image where pixels are classified into at least a second group less relevant to the application. Therefore, the spatial layout of the generated image can be dynamically adjusted based on the classification of its elements. This means that regions containing pixels classified as highly relevant to the application (e.g., roads or vehicles in an autonomous driving scenario) will have a more defined, more constrained spatial layout. Conversely, regions with pixels classified as less relevant (e.g., background scenery) will have a more relaxed, more flexible layout. The more constrained portions of the image can be controlled in the generation process in a manner predetermined by the application. This targeted control over image composition enhances the accuracy and realism of key elements while allowing for greater variation and diversity in less important areas.
[0021] The layout specification can also specify spatial constraints associated with the different classes. Furthermore, the step of determining at least one modification may include removing those spatial constraints associated with at least one group (e.g., at least one second group mentioned above) of non-critical classes specifically representing the application, particularly based on Canny Edges edge information. Thus, more focused image generation can be achieved by modifying spatial layout constraints based on class relevance. This specifically means that the layout specification can prioritize edge information of critical classes while weakening edges associated with less important classes (e.g., background features).
[0022] Initial synthetic images and / or sensor images representing the scene can also be provided, particularly using a simulator, and the generated images are based on the initially provided images, specifically for use as training and / or evaluation data for the machine learning system. In other words, the method can utilize existing synthetic or sensor images as a starting point. This initial dataset can be used as a basis for generating new synthetic images, thereby enhancing the training data for the machine learning system. Utilizing pre-existing images can accelerate training processing and improve the accuracy of the model.
[0023] The provided classification specification can offer different classes in the form of categories for classifying images, specifically classifying the different objects represented in each image. Classification can be performed based on the pixels of the image and the provided categories. This pixel-based classification enables accurate detection and / or identification and / or segmentation of objects within synthetic images.
[0024] A semantic labeling graph can also be provided for the represented scene. Furthermore, the division of different classes can include creating masks from the semantic labeling graph to isolate those classes relevant to the application, thereby dividing the different classes into critical and non-critical class groups. Thus, masks can be created based on the semantic labeling graph to highlight classes that are crucial to a specific application. These masks can then be used to refine layout specifications, emphasizing the spatial arrangement of important elements while allowing greater flexibility in representing less critical areas.
[0025] Furthermore, the scenario can be a traffic scenario, and the machine learning system is trained and / or tested for use in driving systems such as driver assistance systems and / or automated driving systems. Therefore, the generated synthetic images can depict realistic traffic scenarios, enhancing the training data used for driver assistance and automated driving systems. For example, this approach can allow the machine learning system to learn specific traffic situations, object interactions, and road conditions relevant to a particular application. Improved accuracy and diversity of the training data can enhance performance in real-world driving scenarios.
[0026] In another aspect of the invention, a machine learning system can be provided that uses images generated by the method according to the invention as training and / or evaluation data for training and / or testing. Therefore, the machine learning system according to the invention can have the same advantages as those described in detail with reference to the method according to the invention.
[0027] In another aspect of the invention, a computer program, particularly a computer program product, may be provided, comprising instructions that, when executed by at least one computer, cause the computer(s) to perform the method according to the invention. Therefore, the computer program according to the invention may have the same advantages as those described in detail with reference to the method according to the invention.
[0028] In another aspect of the invention, an apparatus for data processing can be provided, configured to perform the method according to the invention. For example, a computer can be provided as the apparatus, executing a computer program according to the invention. The computer may include at least one processor that can be used to execute the computer program. Furthermore, a non-volatile data memory can be provided, in which the computer program can be stored, and from which the processor can read the computer program for execution.
[0029] According to another aspect of the invention, a computer-readable storage medium can be provided, comprising a computer program and / or instructions according to the invention, which, when executed by a computer, cause the computer to perform the steps of the method according to the invention. The storage medium can be configured as a data storage device, such as a hard disk and / or non-volatile memory and / or memory card and / or solid-state drive. The storage medium can, for example, be integrated into a computer.
[0030] Furthermore, the method according to the invention can be implemented as a computer-based method. Alternatively, or additionally, at least one of the disclosed method steps can be computer-based and / or automated.
[0031] According to the present invention, a trained and / or tested machine learning system can be used in a vehicle. The vehicle may be designed, for example, as a motor vehicle and / or a passenger vehicle and / or at least partially automated / autonomous driving vehicle. The vehicle may have vehicle equipment, for example, for providing autonomous driving functions and / or driver assistance systems. The vehicle equipment may be designed to at least partially automatically control and / or accelerate and / or brake and / or steer the vehicle.
[0032] Machine learning systems (particularly in the form of machine learning models) are preferably trained for classification, particularly for object detection. Training may be aimed at training the machine learning system / model using a training dataset for classifying image data (e.g., images generated based on pixels and / or pixel values, preferably based on edges or pixel attributes of the image data) (particularly for image classification). The initial images used for generating the processing may, for example, originate from recordings of at least one sensor (preferably at least one camera), preferably recordings of the vehicle environment, and particularly preferably recordings of the camera environment and / or the vehicle environment during (vehicle) travel.
[0033] Specific technical applications may include at least one of the following: classifying (preferably detecting) objects in images received from cameras of a vehicle (particularly a driving system); scene recognition based on images received from cameras of the driving system; vehicle control based on the output of a machine learning system; classification; classification of objects in images; classification for identifying road signs, vehicles, or pedestrians within a vehicle environment; or support or optimization of a vehicle control system based on image recognition. Furthermore, the application may include at least one of the following: monitoring and / or analysis of sensor data for driver assistance systems, autonomous driving functions, or real-time traffic assessment. Additionally, or alternatively, the application may include tasks in autonomous industrial systems, such as defect detection in manufacturing processes, quality control through visual inspection, or classification of products and materials based on image analysis. Classification may be based on images received from at least one sensor (particularly a camera) of a vehicle (preferably an autonomous vehicle).
[0034] Therefore, an application-specific scenario could be a traffic scenario, as the application could be used to identify traffic scenarios and / or detect objects within them. In this case, the different classes representing the scenario might contain different specific traffic-related scenarios that are therefore critical to the application, such as classes for other vehicles and pedestrians. These classes might also contain less critical classes, such as background or different weather conditions. Alternatively, or additionally, an application-specific scenario could be an industrial scenario, such as a scenario within a production process.
[0035] Classification can be provided for various technical applications (especially at least one of the aforementioned applications). One example is the use of classification in vehicles. Based on classification, particularly based on at least one classification result, at least one control action can be initiated and / or executed, preferably for the vehicle or another technical system.
[0036] The classification results may include and / or be specific to at least one of the following: the category of the object, the identification of the object, the location of the object and / or obstacle (e.g., in or next to the direction of travel), the presence of the obstacle, a description of the traffic scene, hazard or warning information, the number of objects, the type and / or location of lane markings and / or lane boundaries, the location and / or status of the traffic signal system, the location of the lane, and so on.
[0037] Based on the classification results, at least one control action of the vehicle can be initiated and / or executed. The control action may include at least one of the following: braking, steering, acceleration, overtaking maneuver, emergency braking, activation of an alarm system, activation of a hazard warning system, activation of turn signals, light control, etc.
[0038] For example, classification can be used to identify obstacles, whether they are directly in the direction of travel or beside it. Based on the location (e.g., based on the expected vehicle trajectory), corresponding control actions, such as braking or steering, can be initiated.
[0039] "Classification" and "image classification" can also include "object detection" or "object detection in an image." Specifically, this means classifying whether an object exists in a particular region of an image. Additionally, the terms "classification" and "image classification" can also refer to "semantic segmentation," particularly in the form of pixel-by-pixel classification.
[0040] Therefore, training can yield at least one trained machine learning model that can be used for classification and / or object detection. For example, it can provide specific (technical) applications and inference within vehicles. Attached Figure Description
[0041] Further advantages, features, and details of the invention will become apparent from the following description, in which embodiments of the invention will be described in detail with reference to the accompanying drawings. In this context, the features mentioned in the claims and description may be essential to the invention, individually or in any combination.
[0042] Figure 1 Methods, computer programs, storage media, and apparatus according to embodiments of the present invention are illustrated.
[0043] Figure 2 An example of a synthetic image generated by a diffusion model is shown.
[0044] Figure 3 Another visual representation of an embodiment of the present invention is shown. Detailed Implementation
[0045] Figure 1 A computer program 20 with instructions is shown, which, when executed by a computer 10, causes the computer 10 to perform a method 100 according to an embodiment of the present invention. Furthermore, Figure 1 A computer-readable storage medium 15 according to an embodiment of the present invention and a method 100 for training and / or testing a machine learning system 50 for a specific technical application, particularly for a vehicle 60, are shown.
[0046] The following is combined with Figure 1 and Figure 3 Method 100 is described exemplarily.
[0047] According to the first method step 101, at least one instruction 310 (such as a text description, another image, or other content) is provided to the image generation process 340 to generate a synthetic image 320 representing an application-specific scene. If the application is for a vehicle 60, the scene could be a traffic scene.
[0048] According to step 102 of the second method, at least one particularly conditional layout specification 330 is provided, which specifies the space constraints for generating the process 340.
[0049] Then, according to step 103 of the third method, a classification specification 350 is provided, which provides different classes for the represented scene. For this purpose, the classification specification 350 may include a semantic label graph. These classes may include categories such as "lane," "car," "wheel," "vehicle," and "house."
[0050] In step 104 of the fourth method, the different classes in the classification specification can be divided into at least two groups representing different levels of relevance to the application. For example, this could be a high-relevance group and another low-relevance group. The first group with higher relevance could include classes such as "lane" or "vehicle". The second group with lower relevance could include "background" or "snow scene".
[0051] Next, according to step 105 of the fifth method, at least one modification to the layout specification 330 is determined based on the partitioned classes. This allows the generation process 340 to be initiated in step 106 of the sixth method to generate image 320 based on at least one instruction 310 and at least one modified layout specification 330. The modification can be made in such a way that spatial regions of an image corresponding to at least one group with lower relevance are given greater freedom in the generation process compared to spatial regions corresponding to other groups. In other words, the generation process 340 may be more constrained in the spatial regions of image 320 where pixels of image 320 are classified into at least the first group with higher relevance to the application, while less constrained in the spatial regions of image 320 where pixels of image 320 are classified into at least the second group with lower relevance to the application.
[0052] Therefore, the method described according to embodiments of the present invention allows the generation process 340 to be performed with different spatial constraints, which is controlled by at least one modified layout specification 330.
[0053] As is well known, the synergy between Stable Diffusion and ControlNet can be used to create images that simulate those captured by vehicle cameras. Figure 2Examples of synthetic images generated by a diffusion model are shown. These synthetic images can be specifically produced from two main inputs: conditional spatial layouts, primarily constructed using edge detection techniques such as Canny and HED; and descriptive textual cues, which define overall image characteristics, such as snow-covered pavement. Typically, the layout conditions are based on Canny Edge, which is computationally efficient and can be derived from both real and simulated images.
[0054] The embodiments of the present invention address the needs of automated driving technology and propose a method for balancing diversity and fidelity in image generation. This method particularly focuses on leveraging class information obtainable from semantic label graphs derived from driving simulators or other sources to manipulate layout conditions specifically for key classes (e.g., Canny and HED Edges) in the domain.
[0055] In autonomous driving, not all objects are equally important; for example, background elements such as trees may unduly constrain image generation. Therefore, embodiments of the present invention utilize the possibility of creating edge representations that emphasize key objects such as roads and road users, while allowing generative models the freedom to interpret less important areas, thereby enhancing the diversity of the output image without compromising the accuracy of key elements.
[0056] As described by Zhang, Lvmin, AnyiRao, and Maneesh Agrawala in their paper “Adding conditional control to text-to-image diffusion models”, Proceedings of the IEEE / CVF International Conference on Computer Vision, 2023, one current approach involves assigning global weights to spatial layout constraints relative to text cues. However, this approach lacks specificity in focusing on important image regions, thus limiting its effectiveness in sophisticated applications such as autonomous driving.
[0057] exist Figure 2 The paper presents examples of synthetic images that can be generated using the proposed method. Results show that different images have different backgrounds because the background is not constrained by layout conditions (Canny Edge), thus increasing layout diversity.
[0058] Figure 3This visually demonstrates a synthetic image generation method that does not require selective filtering of important edge regions. It shows how buildings (and their layouts, such as windows, etc.) are constrained by Canny Edge. However, using the proposed method according to embodiments of the invention, the network will be able to "imagine" any suitable background following the training data distribution without being overly constrained as in the example here.
[0059] The method according to embodiments of the present invention relies on distinguishing "semantic classes," particularly "important classes," within an automated driving context. Each pixel in an image belongs to a specific class, which can be identified through semantic segmentation or provided by driving simulators such as Nvidia DriveSim. The core of this method lies in selectively focusing on important classes (such as roads and vehicles) during image generation, while downplaying the importance of other classes. Figure 1 As shown, the process consists of two stages:
[0060] Phase I: Training
[0061] At this stage, any generative model trained using Canny Edge can be adapted to the method described in Stage II, but it is recommended to apply similar class filtering during training for optimal results.
[0062] Phase II: Reasoning
[0063] At this stage, the process involves using a combination of a base image, a semantic label map, textual descriptions, and layout information (e.g., from HED or Canny Edge). A mask is created from the semantic label map to isolate important classes. This mask is then used to filter layout conditions, removing edge information associated with non-critical classes. Subsequently, an image is generated using this customized layout information. This method aims to strike a balance between image diversity and fidelity, a characteristic often unresolved in the current spatial layout conditions of generative models.
[0064] The main advantage of this strategy is that it can increase the variation of the generated images without over-constraining them, especially in areas less important for autonomous driving. This allows for a delicate balance between fidelity and diversity in image generation processing.
[0065] The above description of the embodiments has been presented in the context of examples to illustrate the present invention. Of course, without departing from the scope of the invention, the individual features of the embodiments can be freely combined with each other, provided it is technically reasonable.
[0066] Cross-references to related applications
[0067] This application claims priority to European patent application filed on 7 November 2024 (previous application number: 24211597.0). All disclosures of the above application are incorporated herein by reference.
Claims
1. A method (100) for training and / or testing a machine learning system (50) for a specific technology application, the method comprising the following steps: - Provides at least one instruction (310) (101) for image generation processing (340) to generate a composite image (320), the composite image representing a scene specific to the application. - Provide (102) at least one layout specification (330), the layout specification specifying the space constraints of the generation process (340), - Provides (103) a classification specification (350), which provides different classes for the represented scenarios. - Divide the different classes of the classification specification (104) into at least two groups representing different levels of relevance to the application. - Based on the class division, determine (105) at least one modification to the layout specification (330), - Initiate (106) the generation process (340) to generate the image (320) based on the at least one instruction (310) and at least one modified layout specification (330).
2. The method (100) according to claim 1, characterized in that, The method (100) further includes at least one of the following steps: - Training and / or evaluation data for training and / or testing of the machine learning system (50) are provided based on the generated images (320), the training and / or evaluation data specifically including the generated images (320) and / or further modified generated images (320). - Use the generated images (320) as training and / or evaluation data to train and / or test the machine learning system (50), particularly for a specific technical application, which is object and / or scene detection based on images recorded by a vehicle.
3. The method (100) according to claim 1 or 2, characterized in that, The generation process (340) is subject to different spatial constraints controlled by the at least one modified layout specification (330), and thus is subject to more constraints in the spatial regions of the image (320) where the pixels of the image (320) are classified into at least a first group that is more relevant to the application, and less constraints in the spatial regions of the image (320) where the pixels of the image (320) are classified into at least a second group that is less relevant to the application.
4. The method (100) according to any one of the preceding claims, characterized in that, The layout specification (330) specifies space constraints associated with the different classes, and the step of determining (105) at least one modification includes: removing those space constraints associated with at least one group of non-critical classes that specifically represent the application, the space constraints being particularly based on edge information from Canny Edges.
5. The method (100) according to any one of the preceding claims, characterized in that, Provide, in particular generate, initial synthetic images and / or sensor images representing the scene, and the generated images (320) are generated based on the initially provided images, specifically for use as training or evaluation data for the machine learning system (50).
6. The method (100) according to any one of the preceding claims, characterized in that, The provided classification specification (350) provides different classes in the form of categories for classifying images, particularly for classifying different objects represented in each image, wherein the classification is performed based on the pixels of the image (320) and the provided categories.
7. The method (100) according to any one of the preceding claims, characterized in that, A semantic labeling graph is provided for the represented scene, and the division of different classes (104) includes: creating a mask from the semantic labeling graph to isolate those classes that are related to the application, thereby dividing the different classes into critical class groups and non-critical class groups.
8. The method (100) according to any one of the preceding claims, characterized in that, The scenario is a traffic scenario, and the machine learning system (50) is trained and / or tested for use in driver assistance and / or automated driving systems, the application of which specifically includes at least one of the following: classification and preferably detection of objects in images received from the camera of the driving system; scene recognition based on the images; and vehicle control based on the output of the machine learning system (50).
9. A machine learning system (50) that is trained and / or tested using images (320) generated by the method (100) according to any one of claims 1 to 8 as training and / or evaluation data.
10. A computer program (20) comprising instructions that, when executed by at least one computer (10), cause the computer (10) to perform the method (100) according to any one of claims 1 to 8.
11. A data processing apparatus (10) comprising means for performing the method (100) according to any one of claims 1 to 8.
12. A computer-readable storage medium (15) comprising instructions that, when executed by a computer (10), cause the computer (10) to perform the steps of the method (100) according to any one of claims 1 to 8.