Method for generating synthetic traffic sign data
By extracting the 3D model of traffic signs and placing them in real-world background images, the problem of scarcity of traffic sign data in autonomous vehicles is solved, and realistic synthetic data is generated, which improves the performance and robustness of traffic sign detection model.
Patent Information
- Application Number
- CN202410569496.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-09
- Filing Date
- 2024-05-09
- Publication Date
- 2024-11-12
AI Technical Summary
In autonomous or semi-autonomous vehicle development, when generating realistic synthetic traffic sign data to train traffic sign detection models, we face the problem of scarcity of rare traffic sign category data and difficulty in collecting data in regional/national specific variants.
By using information indicating the direction of the detected traffic sign, the front of the traffic sign is extracted from the sample image, forming a 3D model, and placing it in a real-world background image to generate realistic synthetic traffic sign data. This method provides complete control over the location, size and orientation of traffic signs, enhancing the authenticity and diversity of data.
Achieve faster and efficient generation of realistic synthetic traffic sign data, reducing the need for real-world data collection, and improving the accuracy and robustness of traffic sign detection models, especially in the face of new and unprecedented When data.
Smart Images

Figure CN118941701A_ABST
Abstract
Description
Technical Field
[0001] The inventive concept relates to the field of autonomous vehicles. In particular, the present invention relates to a method and apparatus for generating synthetic traffic sign data and for training a traffic sign detection model using the generated synthetic traffic sign data. Background Art
[0002] In recent years, with the development of technology, image capture and processing technology has been widely used in different technical fields. In particular, vehicles produced today are usually equipped with some form of vision or perception system to realize new functions.
[0003] More and more modern vehicles have advanced driver-assistance systems (ADAS) to improve vehicle safety and more generally road safety. ADAS (represented, for example, by adaptive cruise control (ACC), collision avoidance systems, forward collision warnings, lane support systems, etc.) is an electronic system that assists the driver of a vehicle. Today, research and development are ongoing in many technical fields associated with the field of ADAS and the field of autonomous driving (AD). ADAS and AD can also be mentioned under the general term automated driving system (ADS), which corresponds to all different automation levels defined, for example, by the SAE J3016 driving automation levels (0-5).
[0004] An important task for ADAS and AD systems is traffic sign recognition, which today is typically accomplished by detection in images and classification based on trained machine learning models. One of the key challenges in developing systems for traffic sign recognition is collecting large and representative datasets of traffic sign images.
[0005] With today's available machine learning algorithms, one can be relatively confident that good performance can be achieved in classifying traffic signs for those signs that have good support in the dataset. However, this is not the case for rare traffic sign categories or for regional / country specific variants of common categories where little or no data is available.
[0006] For some of these traffic sign categories, the scarcity of traffic signs in the real world makes it difficult and extremely expensive to collect enough samples and annotate them. In addition, regional / country specific variations and emerging privacy or data protection laws make further data collection difficult or even impossible.
[0007] The solution to this problem lies in synthetic data generation, which can be used to effectively increase support for rare and / or important traffic sign categories. Synthetic data generation refers to the process of creating new data that is similar in structure and format to the original dataset but is not based on real-world observations. Nevertheless, synthetic data generation is a challenging research area in itself. When generating synthetic traffic sign data, the images must be realistic enough so that the neural networks trained on these images will be fully operational and desirable in all possible real-world situations. Generating synthetic data that meets this requirement is a challenge. Therefore, an improved solution for generating synthetic traffic sign data is needed. Summary of the invention
[0008] The technology disclosed herein seeks to mitigate, alleviate or eliminate one or more of the above-mentioned deficiencies and shortcomings in the prior art to address various problems related to the generation of synthetic traffic sign data in the development of autonomous or semi-autonomous vehicles.
[0009] The inventors have achieved a new and improved way of generating realistic synthetic data of traffic signs in an easier and efficient manner.
[0010] Various aspects and embodiments of the disclosed invention are defined below and in the accompanying independent and dependent claims.
[0011] According to a first aspect of the disclosed technology, a computer-implemented method for generating synthetic traffic sign data for training a traffic sign detection model is provided. The method includes: extracting the front face of a detected traffic sign from a sample image depicting a vehicle surrounding environment including the detected traffic sign, using information indicating the orientation of the detected traffic sign. The method further includes: forming a 3D model of the traffic sign, the 3D model having the extracted front face of the detected traffic sign. The method further includes: generating synthetic traffic sign data by placing the 3D model of the traffic sign in one or more background images. The 3D model is placed in one or more background images based on the placement of real-world traffic signs in a dataset of real-world images.
[0012] A possible related advantage of the presently disclosed technique is that it provides an accurate 3D model that gives full control over the placement of detected traffic signs to generate synthetic data. Furthermore, the use of a 3D model may provide the effect of producing more realistic synthetic data as opposed to simply placing a 2D model of a traffic sign in a background image. The presently disclosed technique further makes the implementation of a 3D method more feasible because it may alleviate the complexity of time and resource requirements that may otherwise exist.
[0013] The present technology further provides an improved way of placing 3D models in background images, which results in more realistic data. In particular, by utilizing the placement distribution of traffic signs in a dataset of real-world images of traffic signs, more synthetic data can be generated in an automatic and more efficient manner, while achieving enhanced authenticity in the placement of 3D models compared to the prior art. This in turn can have the effect of producing a traffic sign detection model that learns to identify and summarize patterns that are more relevant to the problem being solved. The model also becomes more robust to changes and noise that may exist in real-world data. This will result in improved accuracy and better performance of the model when applied to new, never-before-seen data. This can further help reduce bias in the model. In general, using realistic and representative training data is crucial for developing machine learning models that perform well in the real world.
[0014] Furthermore, some embodiments enable synthetic traffic sign data generation to be performed continuously as a vehicle travels on a road in an efficient manner, thereby reducing the time it takes to generate sufficient data to develop a traffic sign detection model with desired performance.
[0015] According to a second aspect, a computer program product is provided, the computer program product comprising instructions, which, when executed by a computing device, cause the computing device to perform a method according to any embodiment of the first aspect. According to an alternative embodiment of the second aspect, a (non-transitory) computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a processing system, the one or more programs comprising instructions for performing a method according to any embodiment of the first aspect. When applicable, the above features of the first aspect also apply to the second aspect. In order to avoid unnecessary repetition, reference may be made to the above content.
[0016] According to a third aspect, a method for training a traffic sign detection model is provided. The method comprises: obtaining a training data set, the training data set comprising synthetic traffic sign data generated by any embodiment of the first aspect. The method further comprises: training the traffic sign detection model on the obtained training data set. The step of training the traffic sign detection model is performed in a server or in a vehicle. When applicable, the above features of the first and second aspects also apply to the third aspect. In order to avoid unnecessary repetition, reference may be made to the above content.
[0017] According to a fourth aspect, a device for generating synthetic traffic sign data for training a traffic sign detection model is provided. The device includes a control circuit. The control circuit is configured to extract the front of the detected traffic sign from a sample image for depicting the vehicle surroundings including the detected traffic sign using information indicating the orientation of the detected traffic sign. The control circuit is further configured to form a 3D model of the traffic sign, the 3D model having the extracted front of the detected traffic sign. The control circuit is further configured to generate synthetic traffic sign data by placing the 3D model of the traffic sign in one or more background images. The 3D model is placed in one or more background images based on the placement distribution of the real-world signs in a data set of images of the real world. When applicable, the above features of the first, second and third aspects also apply to the fourth aspect. In order to avoid unnecessary repetition, reference may be made to the above content.
[0018] According to a fifth aspect, a vehicle is provided. The vehicle comprises one or more sensors. The vehicle further comprises: an apparatus for generating synthetic traffic sign data for training a traffic sign detection model according to any embodiment of the fourth aspect. When applicable, the above features of the first, second, third and fourth aspects also apply to the fifth aspect. In order to avoid unnecessary repetition, reference may be made to the above content.
[0019] As used herein, the term "non-transitory" is intended to describe computer-readable storage media (or "memory"), excluding propagating electromagnetic signals, but is not intended to otherwise limit the types of physical computer-readable storage devices encompassed by the phrase "computer-readable medium or memory." For example, the terms "non-transitory computer-readable medium" or "tangible memory" are intended to encompass types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in a non-transitory form on a tangible, computer-accessible storage medium may be further transmitted via a transmission medium or signals such as electronic, electromagnetic, or digital signals, which may be communicated over a communication medium such as a network and / or wireless link. Thus, as used herein, the term "non-transitory" is a limitation on the medium itself (i.e., tangible, rather than a signal), rather than a limitation on the persistence of data storage (e.g., RAM versus ROM).
[0020] The various aspects and preferred embodiments disclosed may be appropriately combined with each other in any manner obvious to a person skilled in the art, such that one or more features or embodiments disclosed with respect to one aspect may also be considered disclosed with respect to another aspect or embodiment of another aspect.
[0021] Further embodiments are defined in the dependent claims. It should be emphasized that when used in this specification, the term "comprises / comprising" is used to indicate the presence of the stated features, integers, steps or components. It does not exclude the presence or addition of one or more other features, integers, steps, components or groups thereof.
[0022] These and other features and advantages of the disclosed technology will be further elucidated below with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The foregoing aspects, features, and advantages of the disclosed technology will be more fully understood by reference to the following illustrative and non-limiting detailed description of example embodiments of the present disclosure when used in conjunction with the accompanying drawings, in which:
[0024] Figure 1 is a schematic flow chart representation of a method for generating synthetic traffic sign data for training a traffic sign detection model according to some embodiments.
[0025] Figure 2 is a schematic flow chart representation of a method for training a traffic sign detection model according to some embodiments.
[0026] Figure 3 is a schematic illustration of an apparatus for generating synthetic traffic sign data for training a traffic sign detection model according to some embodiments.
[0027] Figure 4 is a schematic illustration of a vehicle according to some embodiments.
[0028] Figure 5 is a schematic illustration of a distributed system for generating synthetic traffic sign data according to some embodiments. DETAILED DESCRIPTION
[0029] The present disclosure will now be described in detail with reference to the accompanying drawings, in which some example embodiments of the disclosed technology are shown. However, the disclosed technology may be embodied in other forms and should not be construed as being limited to the disclosed example embodiments. The disclosed example embodiments are provided to fully convey the scope of the disclosed technology to those skilled in the art. Those skilled in the art will appreciate that the steps, services, and functions explained herein may be implemented using separate hardware circuits, using software that works in conjunction with a programmed microprocessor or general-purpose computer, using one or more application specific integrated circuits (ASICs), using one or more field programmable gate arrays (FPGAs), and / or using one or more digital signal processors (DSPs).
[0030] It will also be understood that when the present disclosure is described in the form of a method, the present disclosure may also be embodied in an apparatus including one or more processors, one or more memories coupled to the one or more processors, in which a computer code is loaded to implement the method. For example, in some embodiments, the one or more memories may store one or more computer programs that, when executed by the one or more processors, cause the apparatus to perform the steps, services, and functions disclosed herein.
[0031] It should also be understood that the terms used herein are only used for the purpose of describing specific embodiments, and are not intended to be limiting. It should be noted that, as used in this specification and the appended claims, the terms "one", "an", "the" and "said" are intended to indicate the presence of one or more elements, unless the context clearly specifies otherwise. Therefore, for example, in certain contexts, references to "a unit" or "the unit" may refer to more than one unit. In addition, the words "comprise", "include", "contain" do not exclude other elements or steps. It should be emphasized that, when used in this specification, the term "comprises / contains" is used to specify the presence of the features, wholes, steps or parts. It does not exclude the presence or addition of one or more other features, wholes, steps, parts or their groups. The term "and / or" should be interpreted as "both" and each as the meaning of an alternative.
[0032] It will be understood that although the terms "first", "second", etc. may be used herein to describe various elements or features, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the embodiments. The first element and the second element are both elements, but they are not the same element.
[0033] Some aspects of the currently disclosed technology provide a method for generating synthetic traffic sign data used in the training of traffic sign detection models. As known in the field of machine learning, the term "synthetic data" refers to artificially generated data, such as created by an algorithm or model, rather than data collected from the real world. In the context of the present disclosure, synthetic data (in this case, a synthetic image) is a combination of an image of the real world (e.g., as a background) and one or more synthetic objects (e.g., computer-generated objects or objects extracted from another source) added to the image of the real world. In other words, the synthetic image can be regarded as a composite or enhanced image in which the synthetic objects are superimposed or placed in the background image. Synthetic data is often used when there is a lack of sufficient available training data, or when collecting data is too expensive, too sensitive (e.g., for privacy reasons), or too difficult to collect. By generating synthetic data, machine learning models can be trained on larger and more diverse data sets, which can improve their accuracy and generalization capabilities. In the currently disclosed technology, these principles are applied in the field of autonomous or semi-autonomous vehicle development. More specifically, it is used in the training of a traffic sign detection model configured to detect and / or classify traffic signs in the vehicle surroundings. Therefore, the term "synthetic traffic sign data" should be understood as synthetic data of images including traffic signs.
[0034] The generation of synthetic traffic sign data as described herein is based on the concept of generating an improved 3D model of a traffic sign, which provides full geometric control over the model. The 3D model is formed based on the assumption that the surface of the traffic sign is substantially flat compared to most other objects in the vehicle's surroundings. This results in a simple and efficient way to form the 3D model, without the need for advanced algorithms or models, since the flatness of the sign can be exploited when recognizing and extracting the sign from the image.
[0035] After forming a 3D model of the sign for which synthetic traffic sign data is to be generated, the present technique provides a way to place the traffic sign in a new image or scene in a realistic manner that represents the real-world context to a higher degree than prior art techniques. This will be explained further below.
[0036] Figure 1 1 is a schematic flow chart representation of a method 100 for generating synthetic traffic sign data. The synthetic traffic sign data is intended to be used in training a traffic sign detection model. The various steps will be described in more detail below. Even though illustrated in a particular order, the steps of the method 100 may be performed in any suitable order and may be performed multiple times. Thus, although Figure 1 The specific order of method steps can be shown, but the order of these steps can be different from what is described. In addition, two or more steps can be performed simultaneously or partially simultaneously. For example, the steps represented as S112 and S114 can be performed at any time during method 100 based on a specific implementation. This variant will depend on the selected software and hardware system and on the designer's choice. All these variants are within the scope of the present invention. Similarly, software implementation can be completed with standard programming techniques with rule-based logic and other logics to complete various steps. The above-mentioned and described embodiments are only given as examples, and should not be limited to the present invention. Other solutions, uses, purposes and functions within the scope of the present invention claimed in the patent claims described below should be apparent to those skilled in the art.
[0037] The method includes extracting (S106) a front face of a detected traffic sign from a sample image depicting a surrounding environment of the vehicle. The surrounding environment includes the detected traffic sign. The front face of the detected traffic sign is extracted based on information indicating an orientation of the detected traffic sign.
[0038] The front side of a traffic sign is to be understood as the front surface of the sign, which includes visual information and symbols intended to convey specific information to drivers or pedestrians. In other words, extracting the front side can be understood as obtaining, separating or segmenting the front side from the rest of the sample image for further processing. Extracting (S106) the front side of the detected traffic sign may include forming (S120) a 2D representation of the detected traffic sign. This may be achieved by projecting or transforming the detected traffic sign into 2D based on information indicating the orientation. The outline or shape of the traffic sign may be determined using an edge finding algorithm or a segmentation model on the projected traffic sign.
[0039] In the context of this disclosure, the term "sample image" should be interpreted as a real-world image of a traffic sign for which synthetic data is to be generated. In other words, a sample image can be understood as an image of a sample of a traffic sign.
[0040] Since traffic signs are usually placed on the side or above the road, the sample image is likely to not be able to directly depict the traffic sign. Therefore, the orientation can be used to compensate for this. The orientation of the sign should be understood as the angle of the traffic sign relative to the camera used to capture the image. The orientation may include the yaw, pitch and roll angles of the traffic sign. In other words, the orientation may include information related to the orientation of the traffic sign in three dimensions. The information indicating the orientation may be the orientation. Alternatively, the information indicating the orientation may be any information from which the orientation can be determined. The information indicating the orientation may be determined using an on-board sensor of the vehicle that has captured the sample image (e.g., using a stereo camera). Alternatively or in combination, additional types of sensor data (in addition to image data), such as LIDAR, RADAR, may be used when determining the information indicating the orientation. Then, a point cloud generated by, for example, LIDAR can be mapped to the pixels of the sample image. Alternatively, the information indicating the orientation may be determined by computer vision techniques or by using a neural network. The information indicating the orientation may further include information related to the intrinsic parameters of the on-board sensor used to capture the sample image. Intrinsic parameters refer to internal characteristics related to sensor imaging characteristics. For example, for a camera, the intrinsic parameters may include one or more of focal length, image sensor size, lens distortion, lens type, aperture size, shutter speed, etc. The information indicating the orientation may further include information related to the position and orientation of the sensor used to capture the sample image relative to the vehicle coordinate system or the road.
[0041] It should be understood that the information indicating the orientation may be determined before the step of extracting (S106) the front face of the detected traffic sign. Alternatively, the information indicating the orientation may be determined as part of the step of extracting (S106) the front face.
[0042] The vehicle's surroundings are to be understood as the general area around the vehicle in which objects (such as other vehicles, landmarks, obstacles, etc.) can be detected and identified by the vehicle's sensors (RADAR, LIDAR, cameras, etc.), i.e., are within the vehicle's sensor range.
[0043] The method 100 further includes forming (S108) a 3D model of a traffic sign having the extracted front face of the detected traffic sign. The 3D model may include a traffic sign having a three-dimensional extracted front face, and the 3D model may be modified in one or more ways, including size and orientation. When the 3D model is placed in the image in a subsequent step, the 3D model including the traffic sign having the extracted front face may be placed on an existing traffic sign pole or signpost. Alternatively, the 3D model may further include a traffic sign pole or signpost. The traffic sign pole or signpost may be a pole or signpost extracted from an image of the real world. Alternatively, the traffic sign pole or signpost may be computer generated.
[0044] Forming (S108) the 3D model of the traffic sign may include stretching (S122) the 2D representation of the front of the traffic sign into a 3D representation of the traffic sign. Thus, the dimension of depth may be added to the 2D representation of the front of the traffic sign to generate a shape representation of the traffic sign. The 3D model of the traffic sign may be generated, for example, in 3D modeling software.
[0045] The method 100 further includes generating (S110) synthetic traffic sign data by placing a 3D model of the traffic sign in one or more background images. The 3D model is placed in the one or more background images based on the placement distribution of real-world signs in the dataset of real-world images. The 3D model can be placed "based on" the distribution in a sense that it is placed in a manner similar to the way signs are placed in the dataset of real-world images. For example, from the distribution, it can be determined that signs are usually placed on the right side of the road and at a certain height from the ground. Alternatively, the 3D model can be placed "based on" the distribution in a sense that it is placed in a manner that complements images that already exist in the dataset. For example, if the dataset includes few or no such examples, the 3D model can be placed in a new location, such as on the left side of the road.
[0046] In this article, the placement of a sign refers to the position (i.e., position and / or height in the ground plane), size and orientation of the sign. The placement (or more specifically, the position) can be given in a three-dimensional coordinate system for representing the depicted surrounding environment. Alternatively or in combination, the placement can be given in an image reference plane. Therefore, placing a 3D model of a traffic sign in one or more background images can include determining (S124) the position, orientation and size of the 3D model in one or more background images in which the 3D model is to be placed. Thus, determining (S124) the position, orientation and size of the 3D model is based on the placement distribution of the real sign in the image data set. Therefore, when the 3D model is placed in one or more images, the 3D model is placed in the determined position, has the determined size and is at the determined orientation. In general, the placement of the 3D model should be realistic in terms of the size of the sign (especially the front), the height of the sign (i.e., the distance from the front to the ground), the position of the sign (e.g., relative to the road) and the orientation (i.e., how the front of the sign is positioned toward the vehicle).
[0047] When placed in one or more background images, the 3D model of the traffic sign can be further adjusted by changing one or more visual characteristics of the sign, such as wear, dirt, lighting conditions, occlusion, etc. By changing different visual characteristics of the traffic sign, the traffic sign detection model trained using the generated synthetic traffic sign data can become more robust to such changes in the real world.
[0048] A dataset of real-world images is to be understood as a dataset of images depicting the vehicle's surroundings, including real-world traffic signs. The dataset of images may be, for example, a dataset used in training a traffic sign detection model. Based on this dataset of real-world images depicting traffic signs, a common distribution (or probability distribution) of traffic signs in images relative to vehicles may be determined. In other words, the placement distribution of real-world signs may describe the possible position, size, and orientation of a sign in an image captured by a vehicle. This information is used to place a 3D model in one or more background images, thereby achieving realistic placement of synthetic traffic signs.
[0049] It should be understood that the background images need not be part of the dataset on which the placement distribution of traffic signs is based. For example, one or more of the background images may be completely new surroundings or scenes that are not currently part of the dataset. However, in the case where the background images are captured by a vehicle having a different camera arrangement (e.g., position and orientation relative to the road) than the vehicle (which has captured the images in the image dataset), it may be necessary to adjust the placement distribution of the signs based on the different camera arrangement.
[0050] In some alternative embodiments, generating (S110) synthetic traffic sign data may include placing a 3D model of a traffic sign in a predetermined position and predetermined direction in one or more background images. In this article, the term "predetermined" means determined before placing the 3D model. The predetermined position and orientation may be set based on user input. Therefore, the 3D model of the traffic sign may be placed manually. In another example, the predetermined position and orientation may be the position and orientation of an existing traffic sign. In other words, an existing traffic sign may be replaced by a 3D model of a detected traffic sign.
[0051] Unlike synthetic images, the term "background" (e.g., "background image") should be understood as an image of the real world, such as an image captured by, for example, a vehicle equipped with a camera. The background image is therefore used as a background scene in which a 3D model of a traffic sign can be placed to form synthetic data. The background image may be free of any real-world traffic signs, allowing the 3D model of the traffic sign to be freely placed. Alternatively, the background image may depict one or more real-world traffic signs, which may be replaced by the generated 3D model of the traffic sign. In yet another example, the background image may depict one or more real-world traffic signs, and the 3D model of the traffic sign may be placed in the background image as a supplement to the real-world traffic signs that already exist. Therefore, the background image may be any image captured by the vehicle's on-board sensors.
[0052] As stated above, the 3D model can be placed in multiple background images. By placing the 3D model in more than one background image, more than one synthetic image can be generated. Therefore, the synthetic data can constitute multiple synthetic images. In this way, more than one example of a detected traffic sign can be generated for further improving the training of the traffic sign detection model.
[0053] The multiple background images may depict different surroundings of the vehicle to provide variation in the generated synthetic data of the detected traffic sign. In some embodiments, the multiple background images may depict substantially the same surroundings, but at different subsequent time frames. At that time, the placement of the traffic sign in each frame (e.g., orientation and distance of the camera) may be slightly adjusted to simulate a vehicle passing by the traffic sign. The generated synthetic traffic sign data may then be viewed as a synthetic video feed of the traffic sign.
[0054] In some embodiments, the multiple background images may be the same, but the 3D models of traffic signs are placed at different locations in the images (eg, different distances from the camera).
[0055] The method 100 may further include updating (S118) the image dataset by adding the generated synthetic traffic sign data to an image dataset used to determine the placement distribution of traffic signs or any other existing traffic sign dataset. Alternatively, a new dataset for training a traffic sign detection model may be formed, the new dataset including the generated synthetic traffic sign data.
[0056] Throughout this disclosure, the terms “real world” or “real” (eg, “real-world image” or “real-world traffic sign”) should be understood to refer to objects as seen in the real world or images used to depict the real world as they are, without modification.
[0057] The method 100 may further include capturing (S102) sample images depicting the vehicle's surroundings via one or more cameras of the vehicle. In conjunction with capturing (S102) the sample images, additional sensor data of the surroundings may be collected, such as LIDAR or RADAR data for providing depth information of the surroundings.
[0058] The method 100 may further include detecting (S104) traffic signs in a sample image depicting the vehicle's surroundings by a machine learning model. In other words, traffic signs may be detected (e.g., whether there is an object corresponding to a traffic sign in the image) by inputting the sample image into a machine learning model trained to detect traffic signs in the image. By detecting the traffic sign, a bounding box or segmentation of the traffic sign may be obtained. The machine learning model used to detect (S104) the traffic sign may be a detection model separate from the traffic sign detection model mentioned elsewhere herein. Alternatively, the traffic sign may be detected (S104) by a detection portion of the traffic sign detection model. In this case, the generated synthetic traffic sign data may be used to train the classification portion of the traffic sign detection model.
[0059] The method 100 may further include obtaining (S112) a label corresponding to the detected traffic sign. The label may specify what type of sign it is and, therefore, may be used as a basic fact when training a traffic sign detection model. The label may be obtained (S112) by user input. Alternatively, in the case where the traffic sign is known, the label may be obtained (S112) by matching the detected traffic sign with a labeled instance of the traffic sign. In other words, the label may be obtained as an existing label belonging to a labeled traffic sign. For example, if the traffic sign detects whether a new instance matches, the matching may be performed by a separate machine learning model trained on several instances. The method 100 may further include assigning (S114) a label to the generated synthetic traffic sign data. The label may be assigned to each background image on which a 3D model is placed. The steps of obtaining (S112) a label and assigning (S114) a label may be implemented independently of some of the steps of the method 100. For example, steps S112 and S114 may be performed at any time after it has been decided to extract ( S106 ) the front side of the detected traffic sign.
[0060] The above steps of method 100 may be performed locally in the vehicle. Method 100 may be performed, for example, by a computing device of the vehicle. Method 100 may further include transmitting (S116) the generated synthetic traffic sign data to a remote server and / or other vehicles. Therefore, the generated synthetic traffic sign data may be used to train traffic sign detection models of other vehicles.
[0061] The method 100 described above provides an improved way of generating synthetic traffic sign data. The principles of the method 100 can be used to generate synthetic data for any traffic sign. However, for the development of autonomous or semi-autonomous vehicles, the method 100 may be particularly advantageous when performed for certain traffic signs. Figure 5 Some embodiments of the presently disclosed technology are described herein, which are directed to generating synthetic traffic sign data for traffic signs belonging to defined traffic sign categories.
[0062] Generally, this can be viewed as a targeted approach to generating synthetic traffic sign data that focuses on those traffic signs that may be of relatively high value for training a traffic sign detection model.
[0063] In the above embodiment, extracting (S106) the front side of the detected traffic sign can be performed in response to the detected traffic sign being classified as a sign included in the defined traffic sign category. In other words, only signs classified as belonging to the defined traffic sign category can be extracted and further processed by method 100.
[0064] The defined traffic sign categories may be, for example, that the traffic sign is a rare traffic sign or an important traffic sign (e.g., from a training perspective). The word "rare" in this document means that the traffic sign is rare given what happens in the real world or given how many examples there are in the current dataset used to train the traffic sign detection model. The word "important" in this document means from a training perspective. If the detected traffic sign is valuable for improving the performance of the traffic sign detection model, then the traffic sign may be "important". As another way of saying it, if the traffic sign is determined or classified as being out-of-distribution (OOD) of the traffic sign detection model, then the traffic sign may be "rare" or "important". Therefore, the defined traffic sign category may be a rare traffic sign category or an important traffic sign category.
[0065] It should be understood that even though described as a "defined traffic sign category", the traffic sign need not be categorized as belonging to a particular category, but rather the traffic sign may be determined to be "rare", "important", or "out of distribution" in some way, and therefore, have greater value to the generated synthetic traffic sign data. In other words, the step of extracting (S106) the front face of the detected traffic sign may be performed in response to the detected traffic sign having an indication of belonging to a defined traffic sign category.
[0066] Determining whether a detected traffic sign belongs to a defined traffic sign class can be done in several ways. For example, it can be determined by using an integrated network to determine whether the detected traffic sign is OOD of the traffic sign detection model, and checking whether the outputs of different integrated networks diverge.
[0067] As another example, a low confidence score or other uncertain confidence score output by a traffic sign detection model for a detected traffic sign may be an indication that the traffic sign is rare because it may indicate that the traffic sign detection model may not have seen the detected traffic sign (or may have seen only to a small extent) during training.
[0068] In yet another example, a separate classification model for classifying signs as belonging to a defined traffic sign category may be used. For example, a separate classification model may be trained to recognize rare traffic signs by training on a few samples of rare traffic signs. In such an embodiment, as more and more examples of rare traffic signs are discovered, the separate classification model may be continually fine-tuned to improve its performance.
[0069] With the proposed method, the feedback cycle can be shortened by using an end-to-end approach, where, for example, each rare traffic sign detected can be directly used to generate new realistic training data. Figure 5 Related further content) can identify rare traffic signs, create more training data anytime and anywhere, retrain or fine-tune the traffic sign detection model, and share new updates with the fleet.
[0070] Optionally, executable instructions for performing these functions are included in a non-transitory computer-readable storage medium, or in other computer program products configured to be executed by one or more processors.
[0071] In general, a computer-accessible medium may include any tangible or non-transitory storage medium or memory medium, such as electronic, magnetic, or optical media (e.g., a disk or CD / DVD-ROM coupled to a computer system via a bus). As used herein, the terms "tangible" and "non-transitory" are intended to describe computer-readable storage media (or "memory"), excluding propagating electromagnetic signals, but are not intended to otherwise limit the types of physical computer-readable storage devices encompassed by the phrase "computer-readable medium or memory." For example, the terms "non-transitory computer-readable medium" or "tangible memory" are intended to encompass types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in a non-transitory form on a tangible computer-accessible storage medium may be further transmitted via a transmission medium or signals such as electronic, electromagnetic, or digital signals, which may be communicated via a communication medium such as a network and / or wireless link.
[0072] Figure 2 is a schematic flow chart representation of a method 200 for training a traffic sign detection model. In the following, the different steps will be described in more detail. Throughout this disclosure, a traffic sign detection model should be understood as a machine learning model for detecting and / or classifying traffic signs in an image.
[0073] In some embodiments, the machine learning model (which may also be referred to as a machine learning algorithm, neural network, etc.) is implemented in any appropriate manner known to a person of ordinary skill in the art, using publicly available appropriate software development machine learning code elements (e.g., such as those available in Python, Pytorch, TensorFlow, and Keras, or in any other appropriate software development platform).
[0074] The word "training" in this article refers to the process of teaching a machine learning model to recognize patterns and relationships in a dataset, or more specifically, to detect / classify / recognize traffic signs in an image. The training process can refer to training a new machine learning model from scratch. However, the process can also refer to retraining an existing machine learning model or fine-tuning an existing model.
[0075] Retraining a machine learning model refers to the process of updating an existing model with a new or updated dataset. Retraining a model can include training the model from scratch, where the training data includes new data (in this case, generated synthetic data). Alternatively, retraining a model can include training an already trained model only on new data. The purpose of retraining can be to improve the accuracy and generalization of the model for new, never-before-seen data or for rare data for which the model is not commonly seen.
[0076] Retraining may also be needed when the original training data is no longer representative of real-world data, or when the performance of the model has degraded due to changes in the data distribution. Retraining may also be needed when the model needs to be adapted to a new task or application.
[0077] The retraining process consists of several steps, similar to the initial training process. New data is added to the existing dataset, and the model is retrained using the updated data. Depending on how much the data has changed, the model may need to be modified or reconfigured before retraining.
[0078] Fine-tuning a machine learning model refers to the process of training a pre-trained model that has been trained for a different task or different data on new data. Fine-tuning can include training a portion of the pre-trained model (e.g., the last few layers), the entire model, or replacing or adding new layers and training only the new layers. The pre-trained model can be, for example, a general machine learning model that is fine-tuned to be suitable for a new task in a new domain.
[0079] The method 200 includes obtaining (S202) a training data set, wherein the training data set includes Figure 1 The method 100 described herein generates synthetic traffic sign data.
[0080] In this document, the term "obtain" is interpreted broadly and encompasses receiving, retrieving, collecting, and acquiring, etc., directly and / or indirectly, between two entities configured to communicate with each other or further with other external entities. However, in some embodiments, the term "obtain" is interpreted as determining, deriving, forming, calculating, etc. In other words, obtaining the training data set may include determining, calculating, or generating the training data set, such as by performing a combination of Figure 1The method 100 described herein. Alternatively, the training data set may be received, for example, from another entity, or from a memory or database storing the training data set. Thus, as used herein, "obtaining" may indicate receiving a parameter at a first entity / unit from a second entity / unit or determining a parameter at a first entity or unit, for example based on data received from another entity / unit.
[0081] The method 200 further includes training (S204) a traffic sign detection model on the obtained training data set. Training (S204) the traffic sign detection model can be done according to any suitable process that is readily understood by those skilled in the art. For example, the traffic sign detection model can be trained using supervised learning, where each instance of the training data set has an associated label that explains what traffic sign it is.
[0082] The step of training (S204) the traffic sign detection model may be performed on the server or in the vehicle, as shown below in combination with Figure 5 Further described.
[0083] The method 200 may further include transmitting the trained traffic sign detection model to the server if the trained traffic sign detection model is trained in the vehicle. Alternatively, the method 200 may further include transmitting the trained traffic sign detection model to the vehicle if the trained traffic sign detection model is trained in the server.
[0084] Optionally, executable instructions for performing these functions are included in a non-transitory computer-readable storage medium, or in other computer program products configured to be executed by one or more processors.
[0085] Figure 3 3 is a schematic diagram of an apparatus 300 for generating synthetic traffic sign data for training a traffic sign detection model. In particular, the apparatus is configured to perform the aforementioned Figure 1 The device 300 may be in a vehicle (e.g., in combination with Figure 4 The device 300 may be provided locally in the vehicle 400 described above. More specifically, the device 300 may be an integrated device of the vehicle's automated driving system ADS. Alternatively, the device 300 may be an external device of the ADS and communicatively connected to the ADS. Alternatively, the device 300 may be provided as an external device of the vehicle (e.g., a remote server), as described below in conjunction with Figure 5 As further described. The remote server may be communicatively connected to the vehicle. As will be readily appreciated by those skilled in the art, the functionality of the device 300 may be distributed across one or more devices. For example, some of the functions of the device 300 described herein may be implemented by a device in the vehicle, and some of the functions may be implemented by the remote server.
[0086] Device 300 includes control circuit 302. Control circuit 302 may physically include a single circuit device. Alternatively, control circuit 302 may be distributed across several circuit devices. As an example, in the case where device 300 is implemented in a vehicle, the device 300 may share its control circuit 302 with other parts of the vehicle.
[0087] like Figure 3 As shown in the example of , the device 300 may further include a transceiver 306 and a memory 308. The control circuit 302 is communicatively connected to the transceiver 306 and the memory 308. The control circuit 302 may include a data bus, and the control circuit 302 may communicate with the transceiver 306 and / or the memory 308 via the data bus.
[0088] The control circuit 302 may be configured to perform overall control of the various functions and operations of the device 300. The control circuit 302 may include a processor 304, such as a central processing unit (CPU), a microcontroller, or a microprocessor. The processor 304 may be configured to execute program code stored in the memory 308 to perform the various functions and operations of the device 300. The control circuit 302 is configured to perform the above-mentioned combination Figure 1 The steps of the method 100 described herein. These steps can be implemented in one or more functions stored in the memory 308. The control circuit 302 can be configured to perform a capture function 310, a detection function 312, an extraction function 314, a formation function 316, a generation function 318, an acquisition function 320, an allocation function 322, a transmission function 324 and / or an update function 326. It should be noted that the various functions and operations of the device 300 can be implemented in more functions than those described herein. In addition, one or more of the above functions can be implemented together in a common function. For example, the acquisition function 320 and the allocation function 322 can be implemented in a common marking function.
[0089] The transceiver 306 is configured to enable the device 300 to communicate with other devices. The transceiver 306 can transmit data to the device 300 and can receive data from the device 300. For example, the device 300 can transmit a sample image or generated synthetic traffic sign data to another entity (such as another vehicle or a remote server). In addition, the device 300 can receive a sample image from another entity. Although Figure 3 Although not explicitly illustrated in the drawings, device 300 may include user input devices, such as one or more of a keyboard, a mouse, and a touch screen.
[0090] The memory 308 may be a non-transitory computer-readable storage medium. The memory 308 may be one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile storage, a random access memory (RAM), or another suitable device. In a typical arrangement, the memory 308 may include a non-volatile memory for long-term data storage and a volatile memory used as a system memory for the device 300. The memory 308 may exchange data with the circuit 302 via a data bus. There may also be accompanying control lines and an address bus between the memory 308 and the circuit 302.
[0091] The functions and operations of the device 300 may be implemented in the form of executable logic routines (e.g., lines of code, software programs, etc.) that are stored on a non-transitory computer-readable recording medium (e.g., memory 308) of the device 300 and executed by the circuit 302 (e.g., using the processor 304). In other words, when it is described that the circuit 302 is configured to perform a specific function, the processor 304 of the circuit 302 may be configured to execute a program code portion stored on the memory 308, where the stored program code portion corresponds to the specific function. In addition, the functions and operations of the circuit 302 may be independent software applications, or form part of a software application for performing additional tasks associated with the circuit 302. The functions and operations described may be viewed as methods (e.g., the above combined Figure 1 The method 100 discussed above is described above, and the corresponding device is configured to perform the method. In addition, although the functions and operations described can be implemented in software, such functions can also be implemented via dedicated hardware or firmware or some combination of one or more of hardware, firmware and / or software. In the following, the functions and operations of the device 300 are described.
[0092] The control circuit 302 is configured to extract the front face of the detected traffic sign from the sample image depicting the vehicle surroundings including the detected traffic sign using the information indicating the orientation of the detected traffic sign. This can be performed, for example, by executing an extraction function 314. Extracting the front face of the detected traffic sign can be performed in response to the detected traffic sign being classified as a sign included in the defined traffic sign category. Extracting the front face of the detected traffic sign can include forming a 2D representation of the detected traffic sign by projecting the detected traffic sign into 2D based on the information indicating the orientation.
[0093] The control circuit 302 is further configured to form a 3D model of the traffic sign having the extracted front face of the detected traffic sign. This may be performed, for example, by executing a forming function 316. Forming the 3D model of the traffic sign may include stretching a 2D representation of the front face of the traffic sign into a 3D representation of the traffic sign.
[0094] The control circuit 302 is further configured to generate the synthetic traffic sign data by placing the 3D model of the traffic sign in one or more background images. The 3D model is placed in the one or more background images based on the placement distribution of the real-world traffic signs in the data set of real-world images. This can be performed, for example, by executing the generation function 318. Placing the 3D model of the traffic sign in the one or more background images can include determining the position, orientation, and size of the 3D model within the one or more background images in which the 3D model is to be placed based on the placement distribution of the real-world traffic signs in the data set of real-world images.
[0095] The control circuit 302 may be further configured to capture sample images depicting the vehicle's surroundings through one or more cameras of the vehicle. This may be performed, for example, by executing a capture function 310 .
[0096] The control circuit 302 may be further configured to detect traffic signs in sample images depicting the vehicle's surroundings using a machine learning model. This may be performed, for example, by executing a detection function 312 .
[0097] The control circuit 302 may be further configured to update the image dataset by adding the generated synthetic traffic sign data to the image dataset. This may be performed, for example, by executing the update function 326.
[0098] The control circuit 302 may be further configured to obtain a tag corresponding to the detected traffic sign. This may be performed, for example, by executing the obtain function 320.
[0099] The control circuit 302 may further be configured to assign a label to the generated synthetic traffic sign data. This may be performed, for example, by executing an assigning function 322 .
[0100] The control circuit 302 may be further configured to transmit the generated synthetic traffic sign data to a remote server and / or to other vehicles. This may be performed, for example, by executing a transmission function 324 .
[0101] It should be noted that, if the above is combined Figure 1 The principles, features, aspects and advantages of the described method 100 are also applicable to the apparatus 300 described herein. To avoid unnecessary repetition, reference is made to the above contents.
[0102] The device 300 may be further configured to perform the method 200 for training a traffic sign detection model (as described above in conjunction with Figure 2 Alternatively, other devices for performing method 200 may be provided according to the concept of the present invention. Other devices may have the same configuration as the device 300 described above.
[0103] Other devices (or the device 300 described above) may be configured to obtain a training data set including the synthetic traffic sign data generated by the method 100 described above. Other devices (or the device 300 described above) may be further configured to train a traffic sign detection model on the obtained training data set.
[0104] Figure 4 4 is a schematic illustration of an example of a vehicle 400 according to some embodiments. The vehicle 400 is equipped with an automated driving system (ADS) 410. As used herein, a "vehicle" is any form of motor vehicle. For example, the vehicle 400 may be any road vehicle, such as a car (as illustrated herein), a motorcycle, a (freight) truck, a bus, a smart bike, etc.
[0105] Vehicle 400 includes a number of elements that may be common in autonomous or semi-autonomous vehicles. It will be appreciated that vehicle 400 may have Figure 4 In addition, the vehicle 400 may include any combination of the various elements shown in FIG. Figure 4 Although various elements are shown herein as being located inside the vehicle 400, one or more elements may also be located outside the vehicle 400. In addition, even though various elements are depicted herein in a particular arrangement, various elements may also be implemented in different arrangements as will be readily appreciated by those skilled in the art. It should be further noted that various elements may be communicatively connected to each other in any suitable manner. Figure 4 The vehicle 400 should be considered as an illustrative example only, as the various elements of the vehicle 400 may be implemented in a number of different ways.
[0106] The vehicle 400 includes a control system 402. The control system 402 is configured to perform overall control of various functions and operations of the vehicle 400. The control system 402 includes a control circuit 404 and a memory 406. The control circuit 402 may physically include a single circuit device. Alternatively, the control circuit 402 may be distributed across several circuit devices. As an example, the control circuit 402 may share its control circuit 404 with other parts of the vehicle. The control circuit 402 may include one or more processors, such as a central processing unit (CPU), a microcontroller, or a microprocessor. One or more processors may be configured to execute program codes stored in the memory 406 to perform various functions and operations of the vehicle 400. The processor may be or may include any number of hardware components for performing data or signal processing or for executing computer codes stored in the memory 406. In some embodiments, the control circuit 404 or some of its functions may be implemented on one or more so-called systems on chips (SoCs). As an example, the ADS 410 may be implemented on the SoC. Memory 406 optionally includes high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 406 may include database components, object code components, script components, or any other type of information structure used to support various activities in this description.
[0107] In the illustrated example, the memory 406 further stores map data 408. The map data 408 may be used, for example, by the ADS 410 of the vehicle 400 in order to perform autonomous functions of the vehicle 400. The map data 408 may include high-definition (HD) map data. It is conceivable that the memory 408, even if illustrated as an element separated from the ADS 410, may be provided as an integrated element of the ADS 410. In other words, according to exemplary embodiments, any distributed memory device or local memory device may be used to implement the concept of the present invention. Similarly, the control circuit 404 may be distributed, for example, so that one or more processors of the control circuit 404 are provided as an integrated element of the ADS 410 or any other system of the vehicle 400. In other words, according to exemplary embodiments, any distributed control circuit device or local control circuit device may be used to implement the concept of the present invention.
[0108] The vehicle 400 further includes a sensor system 420. The sensor system 420 is configured to acquire sensory data related to the vehicle itself or its surroundings. The sensor system 420 may, for example, include a global navigation satellite system (GNSS) module 422 (e.g., GPS) configured to collect geographic location data of the vehicle 400. The sensor system 420 may further include one or more sensors 424. The one or more sensors 424 may be any type of on-board sensor, such as a camera, LIDAR and RADAR, an ultrasonic sensor, a gyroscope, an accelerometer, an odometer, etc. It should be understood that the sensor system 420 may also provide the possibility of acquiring sensory data directly or via a dedicated sensor control circuit in the vehicle 400.
[0109] The vehicle 400 further includes a communication system 426. The communication system 426 is configured to communicate with external units, such as other vehicles (i.e., via a vehicle-to-vehicle (V2V) communication protocol), remote servers (e.g., cloud servers, as described below in conjunction with Figure 5 Further explained), databases or other external devices (i.e., vehicle-to-infrastructure (V2I) or vehicle-to-everything (V2X) communication protocols). The communication system 426 can use one or more communication technologies to communicate. The communication system 426 may include one or more antennas. Cellular communication technology can be used for long-distance communication, such as to a remote server or cloud computing system. In addition, if the cellular communication technology used has low latency, it can also be used for V2V, V2I or V2X communication. Examples of cellular radio technologies include Global System for Mobile Communications (GSM), General Radio Packet Service (GPRS), Enhanced Data Rate for GSM Evolution (EDGE), Long Term Evolution (LTE), Fifth Generation Mobile Communication Technology (5G), 5G NR, etc., and also include future cellular solutions. However, in some solutions, medium- and short-range communication technologies, such as wireless local area networks (LANs), such as solutions based on IEEE 802.11, can be used to communicate with other vehicles near the vehicle 400 or with local infrastructure elements. The European Telecommunications Standards Institute (ETSI), which is developing cellular standards for vehicle communications, for example, considers 5G to be an appropriate solution due to its low latency and efficient handling of high bandwidth and communication channels.
[0110] The communication system 426 may further provide the possibility of sending the output to a remote location (e.g., a remote operator or a control center) via one or more antennas. In addition, the communication system 426 may be further configured to allow various elements of the vehicle 400 to communicate with each other. As an example, the communication system may provide a local network setup such as CAN bus, I2C, Ethernet, and fiber optics, etc. The local communication within the vehicle may also be of wireless type with protocols such as Wireless Fidelity (WiFi), Long Range Radio (LoRa), Zigbee, Bluetooth, or similar medium / short range technologies.
[0111] The vehicle 400 further includes a steering system 428. The steering system 428 is configured to control the steering of the vehicle 400. The steering system 428 includes a steering module 430 configured to control the driving direction of the vehicle 400. The steering system 428 further includes a throttle module 432 configured to control the actuation of the throttle of the vehicle 400. The steering system 428 further includes a brake module 434 configured to control the actuation of the brakes of the vehicle 400. The various modules of the steering system 428 can receive manual inputs from the driver of the vehicle 400 (i.e., from the steering wheel, the accelerator pedal, and the brake pedal, respectively). However, the steering system 428 can be communicatively connected to the ADS 410 of the vehicle to receive instructions related to how the various modules should act. Therefore, the ADS 410 can control the steering of the vehicle 400.
[0112] As described above, the vehicle 400 includes the ADS 410. The ADS 410 may be part of the vehicle's control system 402. The ADS 410 is configured to perform functions and operations of autonomous functions of the vehicle 400. The ADS 410 may include a plurality of modules, each of which is assigned a different function of the ADS 410.
[0113] The ADS 410 may include a positioning module 412 or positioning block / system. The positioning module 412 is configured to determine and / or monitor the geographic location and direction of travel of the vehicle 400 and may utilize data from a sensor system 420, such as data from a GNSS module 422. Alternatively or in combination, the positioning module 412 may utilize data from one or more sensors 424. However, the positioning system may alternatively be implemented as a real-time kinematic (RTK) GPS to improve accuracy.
[0114] The ADS 410 may further include a perception module 414 or perception block / system. The perception module 414 may refer to any known module and / or function, such as included in one or more electronic control modules and / or nodes of the vehicle 400, adapted and / or configured to interpret sensory data related to the driving of the vehicle 400 to identify, for example, obstacles, lanes, relevant landmarks, appropriate navigation paths, etc. Therefore, the perception module 414 may be adapted to rely on and obtain input from multiple data sources, such as automotive imaging, image processing, computer vision and / or in-vehicle networking, etc., and combined with sensory data from, for example, the sensor system 420. A traffic sign detection (or recognition) model for detecting and classifying traffic signs in the surrounding environment of the vehicle 400 may be part of the ADS 410, or more specifically, part of the perception module 414. The vehicle 400 is configured to perform the functions of the method 100 for generating synthetic traffic sign data. This can be achieved by the vehicle 400 including the device 300 for generating synthetic traffic sign data (as described above in conjunction with Figure 3 ). The device 300 may be provided as an integral part of the perception module 414. Alternatively, the device 300 may be distributed across several modules or elements of the vehicle 400, as will be readily appreciated by those skilled in the art.
[0115] The positioning module 412 and / or the perception module 414 may be communicatively connected to the sensor system 420 to receive the sensing data from the sensor system 420. The positioning module 412 and / or the perception module 414 may further transmit control instructions to the sensor system 420.
[0116] The ADS may further include a path planning module 416. The path planning module 416 is configured to determine a planned path for the vehicle 400 based on the perception and position of the vehicle determined by the perception module 414 and the positioning module 412, respectively. The planned path determined by the path planning module 416 may be sent to the maneuvering system 428 for execution.
[0117] The ADS may further include a decision and control module 418. The decision and control module 418 is configured to perform control of the ADS 410 and make decisions. For example, the decision and control module 418 may determine whether the planned path determined by the path planning module 416 should be executed.
[0118] It should be understood that parts of the described solutions may be implemented in the vehicle 400, in a system located external to the vehicle, or in a combination of the inside and outside of the vehicle; for example, in a server in communication with the vehicle, a so-called cloud solution. The different features and steps of the various embodiments may be combined in other combinations than those described. In addition, the elements (i.e., systems and modules) of the vehicle 400 may be implemented in combinations different from those described herein.
[0119] Figure 5 An example embodiment of a distributed system 500 for generating synthetic traffic sign data and / or for training a traffic sign detection model is shown. The system 500 may be viewed as a non-limiting example implementation of the various aspects of the inventive concept disclosed herein. For example, the system 500 is configured to perform the above-described Figure 1 The method 100 for generating synthetic traffic sign data described above. The system 500 may be further configured to perform the above-mentioned Figure 2 The method 200 for training a traffic sign detection model is described. Therefore, unless otherwise stated, the above Figures 1 to 4 Any features or principles described are also applicable to the system 500 described herein.
[0120] The system 500 includes a remote, central, back-end, fleet, or back-end server 502 (hereinafter referred to as remote server 502 or simply server 502). As illustrated, the remote server 502 may be provided in the cloud, i.e., as a cloud-implemented server. The system 500 further includes one or more vehicles 504a-504c, also referred to as a fleet of vehicles 504a-504c. The one or more vehicles 504a-504c may be as described above in combination with Figure 4 The vehicle 400 described herein is a vehicle. One or more vehicles 504a-504c are communicatively connected to a remote server 502 for transmitting and receiving data 506 between the two. Communication can be implemented by any appropriate wireless communication protocol. The wireless communication protocol can be, for example, a long-distance communication protocol, such as a cellular communication technology (e.g., GSM, GPRS, EDGE, LTE, 5G, 5G NR, etc.), or a short-to-medium-range communication protocol, such as a solution based on a wireless local area network (LAN) (e.g., IEEE 802.11). The server 502 includes appropriate memory and control circuitry, such as one or more processors or processing circuits and one or more other components (such as a data interface and a transceiver). The server 502 may also include a software module or other component so that the control circuit can be configured to execute machine-readable instructions loaded from the memory to implement the steps of the method to be performed.
[0121] Figure 5 The fleet shown in FIG. 5 includes three vehicles 504a-504c, which is only an example. The system 500 may include any number of vehicles 504a-504c. However, the system 500 will be described below with reference to one of these vehicles (denoted by 504a). Needless to say, these principles also apply to other vehicles in the fleet of vehicles 504a-504c.
[0122] In this example embodiment, the vehicle 504a (or, in other words, the computing device or its control circuit) is configured to perform the capture step S102 as explained above to capture the sample image. The sample image can be captured by one or more on-board cameras of the vehicle 504a. Therefore, the sample image depicts the surrounding environment of the vehicle 504a. When the vehicle 504a travels along the road, the sample image can be captured continuously or at defined time intervals. Therefore, the method 100 can be iteratively performed to capture any traffic sign of interest.
[0123] The vehicle 504a is further configured to perform a detection step S104, as further explained above. Thus, the vehicle 504a is configured to detect the sample image, i.e., whether a traffic sign is present (i.e., depicted) in the sample image. Detecting the traffic sign may be performed by a machine learning model.
[0124] The vehicle 504a may be further configured to determine, in response to detecting a traffic sign in the sample image, whether the detected traffic sign belongs to a defined traffic sign category. In other words, the vehicle 504a may classify the detected traffic sign. If the detected traffic sign is found to be of interest (e.g., by belonging to a defined traffic sign category), the vehicle 504a may transmit the sample image of the detected traffic sign to the remote server 502 for further processing.
[0125] In response to receiving the sample image from the vehicle 504a, the remote server 502 performs a step of extracting (S106) the front face of the traffic sign in the sample image. The remote server 502 is further configured to perform a step of forming (S108) a 3D model of the traffic sign having the extracted front face. The remote server 502 is further configured to generate (S110) synthetic traffic sign data by placing the 3D model of the traffic sign in one or more background images based on the placement distribution of the real-world traffic signs in the images in the dataset of real-world images. The dataset of images can be stored in the remote server 502. The image dataset can include images captured by the vehicles 504a-504c of the system 500 and used to depict the surrounding environment of the vehicles 504a-504c including the traffic signs.
[0126] After the new synthetic traffic sign data is generated, the stored image dataset can be updated by adding the generated traffic sign data to the dataset. Alternatively or in combination, the server 502 can transmit the generated synthetic traffic sign data (and / or the updated dataset) to one or more vehicles in the fleet of vehicles 504a-504c, for example, for use in training a traffic sign detection model of the vehicle.
[0127] In some embodiments, the step of obtaining (S112) a label corresponding to the detected traffic sign and the step of assigning (S114) the label to the generated synthetic traffic sign data are performed by the server 502. The label can be obtained, for example, from a defined traffic sign category to which the traffic sign belongs. In the case where the traffic sign for which the synthetic traffic sign data is generated is a new, i.e., never-before-seen, traffic sign, the label can be obtained from user input from a person responsible for labeling images of traffic signs. However, it should be understood that the obtaining (S112) and assigning (S114) steps can also be implemented in the vehicle 504a.
[0128] The illustrated system as described above provides an efficient way to collect images of, for example, new, rare, or important traffic signs and generate synthetic traffic sign data thereof. The system is time efficient because a fleet of multiple vehicles is used to collect and transmit sample images of traffic signs. This can increase the number of signs that can be detected within a particular time period.
[0129] Because only relevant traffic signs (e.g., traffic signs belonging to defined traffic sign categories) may need to be processed, the system 500 may be further efficient in terms of data management (e.g., transmission and storage). The current distribution of the steps of method 100 between the server 502 and the vehicles 504a-504c is further efficient in terms of computing resources. One reason for this may be that as many computing steps as possible can be performed by the server 502, thereby reducing the amount of computing resources required by the vehicles 504a-504c. However, it should be noted that the system 500 is described only by way of example. A person of ordinary skill in the art will understand that modifications to the system 500 may be accomplished depending on the specific implementation. For example, some steps may be performed by the vehicle 504a or by the server 502. For example, in some embodiments, the vehicle 504a performs all the steps of method 100, such as by combining the above. Figure 3The apparatus 300 described herein. Subsequently, the vehicle 504a may transmit the synthesized traffic sign data to the server 502. Subsequently, the server 502 may distribute the synthesized traffic sign data to other vehicles in the fleet of vehicles. In yet another example, the vehicle 504a may perform up to and including the step of extracting the front face of the sign in the sample image for which the synthesized traffic sign data is to be generated, and then, transmit the front face of the sign to the server 502. Subsequently, the server 502 may perform the remaining steps of the method 100. In yet another example, the vehicle 504a may perform up to and including the step of forming a 3D model of the sign. Subsequently, the vehicle 504a may transmit the 3D model to the server 502, which then generates the synthesized traffic sign data.
[0130] As described above, the system 500 may be further configured to perform the method 200 for training a traffic sign detection model. The training of the traffic sign detection model may be performed by so-called federated learning.
[0131] Federated learning is a scheme that integrates a "global" or "central" model from a "local" model trained at the edge. This is common in, for example, "voice assistants" and "typing models" on smartphones. In this context, the vehicle constitutes an edge device or node, and a remote server (or multiple remote servers) may constitute a central server that is responsible for aggregating or merging model updates from each of the edge devices. This will be referred to as a centralized federated learning scheme. However, in some embodiments, the need for a central entity may be omitted, and edge devices (e.g., a fleet of vehicles) may be configured to coordinate themselves to obtain a global model, such as a so-called decentralized federated learning scheme. In the following, a description is given mainly with reference to a centralized federated learning scheme. However, it is assumed that those skilled in the art will easily understand how to implement the teachings herein as a decentralized federated learning scheme, and that the scope conveyed by the invention disclosed herein covers such an implementation. It should be noted that in this context, the term "global" does not necessarily mean "full range", but should be interpreted as certain transactions being "shared" between multiple "local" entities.
[0132] In this example, training (or retraining) (S204) a traffic sign detection model on a training data set including synthetic traffic sign data generated by method 100 may be performed in vehicle 504a. The training data set may be obtained from a remote server 502 and generated from sample images captured by the vehicle itself or other vehicles in a fleet of vehicles. Optionally, vehicle 504a transmits the trained (or retrained) traffic sign detection model to remote server 502. Remote server 502 may in turn distribute the trained traffic sign detection model to other vehicles in the system. Alternatively, remote server 502 may form an aggregated traffic sign detection model from two or more traffic sign detection models and then publish it.
[0133] Alternatively, the training of the traffic sign detection model (S204) can be performed by the remote server 502. As explained above, the remote server 502 can receive the synthetic traffic sign data from the vehicle 504a, or generate the synthetic traffic sign data itself. Subsequently, the server 502 can distribute the trained traffic sign detection model to one or more vehicles 504a-504c in the system 500.
[0134] The present invention has been presented above with reference to specific embodiments. However, other embodiments in addition to the embodiments described above are also feasible and within the scope of the present invention. Within the scope of the present invention, method steps different from the method steps described above may be provided, and the method may be performed by hardware or software. Therefore, according to an exemplary embodiment, a non-transitory computer-readable storage medium is provided, which stores one or more programs, which are configured to be executed by one or more processors of a vehicle control system, and the one or more programs include instructions for performing a method according to any of the embodiments discussed above. Alternatively, according to another exemplary embodiment, a cloud computing system may be configured to perform any method presented herein. The cloud computing system may include distributed cloud computing resources that jointly perform the method presented herein under the control of one or more computer program products.
[0135] In general, a computer-accessible medium may include any tangible or non-transitory storage medium or memory medium, such as electronic, magnetic, or optical media (e.g., a disk or CD / DVD-ROM coupled to a computer system via a bus). As used herein, the terms "tangible" and "non-transitory" are intended to describe computer-readable storage media (or "memory"), excluding propagating electromagnetic signals, but are not intended to otherwise limit the types of physical computer-readable storage devices encompassed by the phrase "computer-readable medium or memory." For example, the terms "non-transitory computer-readable medium" or "tangible memory" are intended to encompass types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in a non-transitory form on a tangible computer-accessible storage medium may be further transmitted via a transmission medium or signals such as electronic, electromagnetic, or digital signals, which may be communicated via a communication medium such as a network and / or wireless link.
[0136] It should be noted that any reference signs do not limit the scope of the claims, that the present invention may be implemented at least partially by means of hardware and software, and that the same hardware item may represent multiple "means" or "units".
Claims
1. A computer-implemented method (100) for generating synthetic traffic sign data for training a traffic sign detection model, the method (100) comprising: extracting (S106) a front face of the detected traffic sign from a sample image depicting a vehicle surrounding environment including the detected traffic sign, using information indicating the orientation of the detected traffic sign; forming (S108) a 3D model of the traffic sign, the 3D model having the extracted detected front face of the traffic sign; as well as Synthetic traffic sign data is generated (S110) by placing the 3D model of the traffic sign in one or more background images, wherein the 3D model is placed in the one or more background images based on a placement distribution of real-world traffic signs in a dataset of real-world images.
2. The method (100) according to claim 1, further comprising: The sample image for depicting the surrounding environment of the vehicle is captured ( S102 ) by one or more cameras of the vehicle.
3. The method (100) according to claim 1, further comprising: The traffic sign in the sample image depicting the vehicle surroundings is detected (S104) by a machine learning model.
4. The method (100) according to claim 1, further comprising: The dataset of the image is updated ( S118 ) by adding the generated synthetic traffic sign data to the dataset of the image.
5. The method (100) of claim 1, wherein: The step of extracting ( S106 ) the front face of the detected traffic sign is performed in response to the detected traffic sign being classified as a sign included in the defined traffic sign category.
6. The method (100) of claim 1, wherein: The step of extracting (S106) the front face of the detected traffic sign comprises: forming (S120) a 2D representation of the detected traffic sign by projecting the detected traffic sign into 2D based on the information indicating the orientation, and The step of forming ( S108 ) the 3D model of the traffic sign comprises: stretching ( S122 ) the 2D representation of the front side of the traffic sign into a 3D representation of the traffic sign.
7. The method (100) of claim 1, wherein: Placing the 3D model of the traffic sign in one or more background images includes: determining (S124) the position, orientation and size of the 3D model within the one or more background images in which the 3D model is to be placed based on the placement distribution of real-world traffic signs in a dataset of real-world images.
8. The method (S100) according to claim 1, further comprising: Obtaining (S112) a label corresponding to the detected traffic sign; as well as The label is assigned ( S114 ) to the generated synthetic traffic sign data.
9. The method (100) of claim 1, wherein: The steps of the method (100) are performed locally in the vehicle, and wherein: The method (100) further comprises: The generated synthetic traffic sign data is transmitted (116) to a remote server and / or other vehicles.
10. A computer program product comprising instructions which, when executed by a computing device, cause the computing device to perform the method (100) according to any one of claims 1 to 9.
11. A method (200) for training a traffic sign detection model, the method (200) comprising: Obtaining (S202) a training data set, the training data set comprising synthetic traffic sign data generated by the method (100) according to any one of claims 1 to 9; as well as training (S204) the traffic sign detection model on the obtained training data set, The step of training (S204) the traffic sign detection model is performed in a server or in a vehicle.
12. A device (300) for generating synthetic traffic sign data for training a traffic sign detection model, the device (300) comprising a control circuit (302), the control circuit (302) being configured to: extracting a front face of the detected traffic sign from a sample image depicting a vehicle surrounding environment including the detected traffic sign using information indicating an orientation of the detected traffic sign; forming a 3D model of the traffic sign, the 3D model having the extracted detected front face of the traffic sign; as well as Synthetic traffic sign data is generated by placing the 3D model of the traffic sign in one or more background images, wherein the 3D model is placed in the one or more background images based on a placement distribution of real-world traffic signs in a dataset of real-world images.
13. The device (300) according to claim 12, wherein: The control circuit (302) is configured to extract the front side of the detected traffic sign in response to the detected traffic sign being classified as a sign included within a defined traffic sign class.
14. The device (300) according to claim 12 or 13, wherein: Extracting the front face of the detected traffic sign comprises: forming a 2D representation of the detected traffic sign by projecting the detected traffic sign into 2D based on the information indicating the orientation, and Wherein forming the 3D model of the traffic sign comprises: stretching the 2D representation of the front side of the traffic sign into a 3D representation of the traffic sign.
15. A vehicle (400), comprising: one or more sensors (424); as well as An apparatus (300) for generating synthetic traffic sign data for training a traffic sign detection model according to any one of claims 12 to 14.
