Teacher data generation device, teacher data generation system, teacher data generation method, and program
An automatic labeling system for radar images using marker detection in supplementary images addresses the inefficiencies of manual labeling, enhancing the performance of object detection models by reducing time and errors.
Patent Information
- Application Number
- JP2023575449
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-06-08
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-06-08
AI Technical Summary
Manual labeling of radar images for object detection is time-consuming and prone to errors due to low resolution, lack of color information, and small object size, which negatively impacts the performance of machine learning models.
An automatic labeling system that uses marker detection in supplementary images synchronized with radar signals to generate labels for radar images, associating them with the radar images to create training data efficiently.
The system significantly reduces labeling time and errors, improving the performance of object detection models by automating the labeling process and ensuring accurate class name and bounding box annotations.
Smart Images

Figure 0007586348000001 
Figure 0007586348000002 
Figure 0007586348000003
Abstract
Description
[Technical field]
[0001] FIELD OF THE DISCLOSURE Embodiments of the present invention relate generally to training data generation. [Background technology]
[0002] In radar image object detection systems, radio waves scattered by a target object are measured and then imaged to generate a radar image of the target under scan. The image is used by an object detection module to identify hidden objects. The object detection module is a machine learning network trained using a supervised approach, where a large amount of labeled dataset is required. Manual labeling is a time-consuming task and prone to human error. Needless to say, the quality of the labeling directly impacts the performance of the object detection module developed.
[0003] US Patent No. 5,399,633 discloses determining the area of an object in a map acquired by a radar using a calculated position of the object based on a range image acquired by a LIDAR, determining the type of object based on image data captured by a camera, and assigning ground truth data to the object. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2018-163096 A Summary of the Invention [Problem to be solved by the invention]
[0005] However, the technology of Patent Document 1 requires LIDAR measurements and complex processing.
[0006] It is an object of the present invention to provide a fast automatic radar image labeling (or auto-labeling) system. [Means for solving the problem]
[0007] The present invention relates to an acquisition unit for acquiring a radar image based on a scattering signal from an object; a marker detection unit for acquiring a supplementary image of an area including the object, the supplementary image being captured in synchronism with detection of the scattering signal, detecting a marker in the supplementary image, and extracting an identifier of the marker; a label generator that generates a label using the extracted identifier; an output unit that outputs output data in which the radar image and the generated label are associated with each other; The present invention provides a teacher data generation device comprising:
[0008] The present invention provides the above teacher data generation device, a measurement unit that detects a scattered signal by transmitting an electromagnetic wave to an object and detecting the electromagnetic wave scattered by the object; A supplementary imaging unit that captures a supplementary image; The present invention provides a teacher data generation system comprising:
[0009] The present invention provides a teacher data generation method performed by a computer, Obtaining a radar image based on a scattered signal from the object and a complementary image of an area including the object, the complementary image being captured synchronously with detection of the scattered signal; Detecting a marker in the captured image; Extracting an identifier of the marker; generating a label using the extracted identifier; outputting output data in which the radar image and the generated labels are associated with each other; The present invention provides a method for generating teacher data, including:
[0010] The present invention is a program for causing a computer to execute a teacher data generation method, the teacher data generation method comprising the steps of: Obtaining a radar image based on a scattered signal from the object and a complementary image of an area including the object, the complementary image being captured synchronously with detection of the scattered signal; Detecting a marker in the captured image; Extracting an identifier of the marker; generating a label using the extracted identifier; outputting output data in which the radar image and the generated labels are associated with each other; The Company provides programs including: Effect of the Invention
[0011] The objective of the present invention is to automate the labeling procedure while making it efficient and fast, thereby improving the performance of object detection modules as well as the speed of their development. In particular, the present disclosure relates to an efficient and fast automatic radar image labeling system. [Brief description of the drawings]
[0012] [Figure 1] FIG. 1 is a diagram illustrating class labels for training an image classifier. [Diagram 2] FIG. 2 is a diagram illustrating bounding box labels for training an image-based object detection network. [Diagram 3] FIG. 3 is a diagram illustrating the configuration of a radar image acquisition system, and also illustrates the relative positions of the target and camera with respect to the radar. [Figure 4] FIG. 4 is a block diagram illustrating a functional-based configuration of the teacher data generation device according to the first embodiment. [Diagram 5] FIG. 5 is a block diagram illustrating a hardware configuration of a computer that realizes the teacher data generation device according to the first embodiment. [Figure 6]FIG. 6 is a flowchart illustrating a process flow performed by the teacher data generation device of the first embodiment. [Figure 7] FIG. 7 is a block diagram illustrating a function-based configuration of the teacher data generation system according to the first embodiment. [Figure 8] FIG. 8 is a block diagram illustrating a function-based configuration of the teacher data generation system according to the first embodiment. [Figure 9] FIG. 9 is a block diagram illustrating a functional configuration of the marker detection unit according to the first embodiment. [Figure 10] FIG. 10 is an exemplary table showing the structure of the label information table according to the first embodiment. [Figure 11] FIG. 11 is a flowchart illustrating the operation of the teacher data generation system according to the first embodiment. [Figure 12] FIG. 12 is a block diagram illustrating a function-based configuration of the teacher data generation system according to the second embodiment. [Figure 13] FIG. 13 is an exemplary table showing the structure of the label information table according to the second embodiment. [Figure 14] FIG. 14 is a flowchart illustrating the operation of the teacher data generation system according to the second embodiment. [Figure 15] FIG. 15 is a block diagram illustrating a function-based configuration of the teacher data generation system according to the third embodiment. [Figure 16] FIG. 16 is an exemplary diagram illustrating the change in label size with object orientation. [Figure 17] FIG. 17(a) is an exemplary table showing the structure of a label information table in embodiment 3, and FIG. 17(b) is an exemplary table showing the structure of a registration class table, for the purpose of adjusting the label size according to the marker direction. [Figure 18]FIG. 18(a) is an exemplary table showing the structure of a label information table in embodiment 3, and FIG. 18(b) is an exemplary table showing the structure of a registered class table, for the purpose of adjusting class names according to marker direction. [Figure 19] FIG. 19 is a flowchart illustrating the operation of the teacher data generation system according to the third embodiment. [Figure 20] FIG. 20 is a block diagram illustrating a function-based configuration of the teacher data generation system according to the fourth embodiment. [Figure 21] FIG. 21 is a block diagram illustrating a function-based configuration of a QR code (registered trademark) generation system according to the fourth embodiment. [Figure 22] FIG. 22 is a flowchart illustrating the operation of the QR code generation system according to the fourth embodiment. [Figure 23] FIG. 23 is a flowchart illustrating the operation of the teacher data generation system according to the fourth embodiment. [Figure 24] FIG. 24 is a block diagram illustrating a function-based configuration of a teacher data generation system according to the fifth embodiment. [Diagram 25] FIG. 25 is a block diagram illustrating a function-based configuration of a QR code generation system according to the fifth embodiment. [Figure 26] FIG. 26 is a flowchart illustrating the operation of the QR code generation system according to the fifth embodiment. [Figure 27] FIG. 27 is a flowchart illustrating the operation of the teacher data generation system according to the fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present disclosure will now be described in detail with reference to the accompanying drawings, in which like reference numerals denote like elements throughout the drawings, and redundant description will be omitted where necessary for clarity.
[0014] EMBODIMENT 1
[0015] <Summary> With respect to labeling for machine learning purposes, some specific points relevant to this disclosure are described below: In a supervised learning approach, a machine learning model uses example input-output pairs to learn functions that map inputs to outputs, where the inputs can be images, lines of text, or snippets of audio. The outputs depend on the type of task being learned and are commonly referred to as "labels" for the inputs. The input-output pairs together are referred to as labeled data. The labels for the inputs may also be referred to as annotations (or metadata) for the inputs.
[0016] Specifically, we focus on images. Common types of labels for images are class name labels, bounding box labels, etc. The most common among them are class name labels used to train classifier networks or models, where the model learns to classify inputs into some pre-determined classes. For example, an input image needs to be classified into three classes depending on whether it contains a gun, a knife, or none of them. To learn such a task, the labeled data must consist of image-class name pairs. This means that each image must be associated with a class name (gun, knife, or nothing). We will illustrate this with the help of an example image shown in Figure 1. In this figure, the class names of the objects present in the image are shown superimposed on the image for ease of understanding. The figure contains a gun, and therefore the class name "Gun 1" (the number 1 designates the type of gun) is shown in the figure.
[0017] Another common form of image label is the bounding box label used to train object detection models. Here, the objective is to detect the presence of an object and locate it. In most cases, the network also classifies the objects into various classes as well. Thus, it can be said that the object detection network learns both the location and the class name of the object. Thus, the label in this case is both the location and the class name of the object. To ensure good localization performance, the bounding box label must be "tight" around the object. "Tight" means that the bounding box size must be the smallest size that completely covers the object of interest. The location is often specified as a rectangular bounding box with the parameters of center location (x, y), width, and height. Often, the rectangular bounding box and the class label are specified together in some predetermined standard format and are referred to as image annotations. In this disclosure, the aforementioned image annotations may also be referred to as rectangle label annotations. FIG. 2 shows an example image with a rectangle label annotation superimposed for ease of understanding. In this example image, the object is a gun with class name "Gun1" and its location is highlighted by a tight rectangular bounding box centered on the gun.
[0018] The creation of labels for supervised learning tasks is (almost always) done manually by humans. As the complexity of the images and / or labels increases, the time required as well as the possibility of labeling errors increases. Even for simple images and / or labels, the labeling time is still significant and the possibility of labeling errors is not zero. Needless to say, the large labeling time slows down the development rate of supervised models. Moreover, labeling errors adversely affect the performance of supervised models.
[0019] The configuration of the object detection system for radar images is described below with the aid of FIG. 3. The specific objective of the system is to ascertain whether a person (target 90) possesses any hidden dangerous object. When the target is walking in a fixed part (area 96), the system measures the target by a fixed antenna (radar 92) placed on a side panel 94, and generates a radar image (essentially 3D) based on the acquired measurement signal. If the person (target 90) possesses a hidden dangerous object, the object is visible in the radar image. The system also acquires a camera image using a camera (camera 98) simultaneously with the radar signal. To detect the presence of a dangerous object in the radar image, a machine learning-based object detection model is trained. This machine learning-based object detection model may acquire the radar image either in 2D form or in its original 3D form. The training of the object detection network requires at least class name label annotation along with the radar image.
[0020] For the purposes of this disclosure, the manual labeling procedure (of radar images) is considered background art. The problem for the background art is described. Before jumping to the problem description, the nature of radar images is described. Radar images are different from camera images in many aspects. First, there is no color information, which makes it difficult to distinguish objects. Furthermore, the resolution of radar images is almost always insufficient compared to camera images. This leads to the following: the hidden objects of interest, which are often small in size, making manual labeling of the dataset by a human labeler a difficult task. Needless to say, it is very time-consuming (much more so than camera images).
[0021] The first problem with manual labeling of radar images is that the low resolution, lack of color information, and small object size make it difficult to accurately identify class types. This problem becomes more pronounced when the number of classes increases and / or when the "visual" differences between classes are minor. This increases the time required for labeling.
[0022] A second problem with manual labeling of radar images is labeling errors: First, class name identification is prone to errors due to the nature of the images in question, which in turn adversely affect the performance of classifiers and / or object detection networks.
[0023] The teacher data generation device 10 of embodiment 1 provides an automatic labeling (or auto-labeling) system for radar images that can solve the first problem by generating class name labels automatically, i.e., with increased speed.
[0024] The teacher data generation device 10 of the first embodiment further provides an automatic labeling (or auto-labeling) system for radar images, which can solve the second problem by eliminating labeling errors through automation of the labeling process (negligible human intervention is required). It should also be noted that even though the labeling examples are shown on 2D images for ease of understanding, the present invention works equally well on 3D radar images.
[0025] <Example of feature-based configuration> FIG. 4 shows an exemplary diagram of a function-based configuration of the teacher data generating device 10 according to the first embodiment. The teacher data generating device 10 of the first embodiment includes an acquisition unit 12, a marker detection unit 14, a label generation unit 16, and an output unit 18. The acquisition unit 12 acquires a radar image based on a scattering signal from an object. The marker detection unit 14 acquires a supplementary image of an area including the object, detects a marker in the supplementary image, and extracts an identifier (ID) of the marker. The supplementary image is an image captured in synchronization with the detection of the scattering signal. The label generation unit 16 generates a label using the extracted identifier. The output unit 18 outputs output data in which the radar image and the generated label are associated with each other. The following is a detailed description.
[0026] The markers are visible in the supplemental imagery and not in the radar imagery. Note that it is assumed that: (1) the markers are visible only in the supplemental imagery (not in the radar imagery) and their IDs can be extracted using signal processing, and (2) unique classes are associated with unique marker IDs, and it is known a priori which marker IDs are associated with which classes.
[0027] The acquisition unit 12 may acquire a radar image by generating a two-dimensional image or a three-dimensional image from the scattered signal. The acquisition unit 12 may read and acquire a radar image that has been generated in advance and stored in a recording device accessible to the acquisition unit 12. The supplementary image may be a camera image, a depth image, or an IR image that is taken in synchronization with the detection of the scattered signal. The marker detection unit 14 may acquire a camera image from a camera or may read and acquire a camera image that has been captured in advance and stored in a recording device accessible to the acquisition unit 12.
[0028] <Example of hardware configuration> In some embodiments, each functional unit included in the teacher data generation device 10 may be implemented by at least one hardware component, and each hardware component may realize one or more of the functional units. In some embodiments, each functional unit may be implemented by at least one software component. In some embodiments, each functional unit may be implemented by a combination of hardware and software components.
[0029] The teacher data generating device 10 may be implemented on a dedicated computer manufactured for the purpose of implementing the teacher data generating device 10, or may be implemented on a general-purpose computer such as a personal computer (PC), a server machine, or a mobile device.
[0030] 5 is a block diagram illustrating an example of a hardware configuration of a computer 1000 that realizes the teacher data generation device 10 of embodiment 1. In FIG. 5, the computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a recording device 1080, an input-output (I / O) interface 1100, and a network interface 1120.
[0031] The bus 1020 is a data transmission channel for the processor 1040, the memory 1060, and the recording device 1080 to transmit and receive data to and from each other. The processor 1040 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 1060 is a primary recording device such as a random access memory (RAM). The recording medium 1080 is a secondary recording device such as a hard disk drive, a solid state drive (SSD), or a read only memory (ROM).
[0032] The I / O interface is an interface between the computer 1000 and peripheral devices such as a keyboard, a mouse, or a display device. The network interface is an interface between the computer 1000 and a communication line through which the computer 1000 communicates with another computer.
[0033] The recording device 1080 may store program modules, each of which is an implementation of a functional unit of the teacher data generation device 10. The CPU 1040 executes each program module, thereby realizing each functional unit of the teacher data generation device 10.
[0034] <Process flow> FIG. 6 is a flow chart illustrating a process sequence performed by the teacher data generating device 10 of the first embodiment. The acquisition unit 12 acquires a radar image based on a scattered signal from an object, and the marker detection unit 14 acquires a supplementary image of an area including the object (S12). The supplementary image is an image captured in synchronization with detection of the scattered signal. The marker detection unit 14 detects a marker in the supplementary image (S14). The marker detection unit 14 extracts an identifier of the marker (S16). The label generation unit 16 generates a label using the extracted identifier (S18). The output unit 18 outputs output data in which the radar image and the generated label are associated with each other (S20).
[0035] <Teacher data generation system> 7 shows an exemplary diagram of a function-based configuration of the teacher data generation system 20 according to the first embodiment. The teacher data generation system 20 of the first embodiment includes a teacher data generation device 10, a measurement unit 22, and a supplementary imaging unit 23. The measurement unit 22 detects a scattered signal by transmitting an electromagnetic wave to an object and detecting the electromagnetic wave scattered by the object. The supplementary imaging unit 23 captures a supplementary image.
[0036] <Example of operation> An example of the operation of the radar image teacher data generation system 20 will be described with reference to FIG. 3. It is assumed that a person (target 90) is present in a screening area (area 96) in front of a fixed radar antenna (radar 92) placed on a side panel 94. The radar 92 corresponds to the measurement unit 22. When in the screening area, the target is also in the detection field of a supplementary signal sensor (hereinafter referred to as a supplementary sensor), which in this case is a camera (camera 98). The camera 98 corresponds to the supplementary imaging unit 23. The measured scattered radar signal is used to generate a radar image, which is preprocessed and used to detect whether the target possesses any dangerous object. The camera captures an image (hereinafter referred to as a camera image) synchronized with the radar sensor. For example, a dangerous object hidden under clothing is not visible in the camera image, but information about it (type, location, etc.) can be identified using some marker (visible in the camera image but not in the radar image). The markers are printed or placed on materials that the radar signal can easily pass through, such as paper, wood, cloth, or plastic, so that the markers are not visible in the radar image. Information extracted from the markers is transferred from the camera image to the corresponding radar image. This information serves as label and / or annotation information for training machine learning classifiers and / or object detector models.
[0037] It should be noted that the markers can be placed anywhere within the detection range of the supplemental signal sensor. In the exemplary operation described above, the detection device is a camera, and therefore the markers should be visible in the camera image. One way to place the markers can be at the center of the object to be labeled.
[0038] <System configuration> An example of the configuration of the training data generation system 20 for radar images according to the first embodiment will be described with reference to the block diagram shown in FIG. 8. In this configuration of the training data generation system 20, the objective is to identify class labels of radar images to train a classifier. It is assumed that a marker with a unique ID is associated with each unique class. The supplementary signal is processed to detect the marker ID and retrieve the associated class label. Finally, the corresponding radar image is annotated with the retrieved class label. One example of such a marker that can be used is an AR marker combined with a supplementary sensor, an optical camera.
[0039] In the example of FIG. 8, the teacher data generation system includes a teacher data generation device 10, a receiver antenna 121, and a camera device 122. As shown in FIG. 8, one of the configurations of the teacher data generation device 10 according to the first embodiment may include a synchronization unit 101, a marker detection unit 103, a label generation unit 104, a label information storage 105, a radar measurement unit 106, an imaging unit 107, and an output unit 109. In this configuration, the imaging unit 107, the marker detection unit 103, the label generation unit 104, the output unit 109, and the camera 122 correspond to the acquisition unit 12, the marker detection unit 14, the label generation unit 16, the output unit 18, and the supplementary imaging unit 23, respectively. The receiver antenna 121 is included in the radar 92. The camera 122 corresponds to the camera 98.
[0040] The synchronization unit 101 transmits a synchronization signal to the radar measurement unit 106 and the marker detection unit 103. This function ensures that the radar signal and the supplementary signal are captured simultaneously. Note that the synchronization unit 101 may be installed separately from the teacher data generation device 10 as a controller or the like.
[0041] The marker detection unit 103 receives a synchronization signal from the synchronization unit 101 as an input and outputs a marker ID to the label generation unit 104. Upon receiving the synchronization signal, the marker detection unit 103 sends a signal to a supplementary sensor (camera 122) to detect a supplementary signal. The marker detection unit 103 then obtains the supplementary signal from the supplementary sensor 122 and detects the presence of a marker in the supplementary signal. If a marker is detected, the next step is to identify its ID (which may be represented by any alphanumeric value). The marker detection unit 103 then outputs the marker ID.
[0042] In one exemplary operation, the supplementary sensor may be a camera, in which case the supplementary signal is a camera image. In this case, the marker detection unit 103 is composed of sub-units as shown in FIG. 9. The sub-units are a camera measurement unit 103a and a detection unit 103b. When the camera measurement unit 103a receives a synchronization signal from the synchronization unit 101, it captures a camera image and provides it as an output to the detection unit 103b. The detection unit 103b detects the presence of a marker in the camera image and outputs its ID. The ID is an output of the marker detection unit 103 that is provided to the label generation unit 104. In this case, if an AR marker is used, an existing library of OpenCV can be used for marker detection.
[0043] The label generation unit 104 receives the marker ID from the marker detection unit 103 as input and outputs the label to the learning unit 123. When the label generation unit 104 receives the marker ID as input, it refers to a label information table retrieved from the label information storage 105. The structure of the label information table is explained using an example table given in FIG. 10. The table has two columns, namely, the marker ID (primary key) and the class name. The number of rows depends on the number of unique classes. The table is manually created in advance (using prior knowledge of the markers and class names) and stored in the storage 105. The label generation unit 104 uses the marker ID as a primary / search key to search for the class name from the label information table. The label generation unit 104 then uses the searched class name to generate annotations for the corresponding radar image. The annotation format can be in the form of a simple text file or in the form of tags, or any other format, depending on the learning model being used. The radar image annotations consisting of class names are the output of the label generator 104.
[0044] In some configurations, the class names may also be directly provided as output to the learning model without any post-processing.
[0045] The radar measurement unit 106 receives a synchronization signal as an input from the synchronization unit 101 and outputs a radar signal to the imaging unit 107. Upon receiving the synchronization signal from the synchronization unit 101, the radar measurement unit 106 transmits a signal to the receiver antenna 121 to receive the radar signal scattered by the target under the scan. The captured radar signal is received by the radar measurement unit 106 from the receiver antenna and transmitted to the imaging unit 107.
[0046] The imaging unit 107 receives radar signals as input from the radar measurement unit 106 and outputs a radar image to the training unit 123. The imaging unit generates a radar image of the scanned target. The imaging unit may use any conventional imaging technique. Furthermore, the output radar image may be either a 3D or 2D image.
[0047] The output unit 109 receives the radar image from the unit 107 and the associated label, i.e., class name, from the label generation unit 104. The output unit 109 outputs output data in which the radar image and the label are associated with each other.
[0048] A trainer 123, in this case a machine learning classifier model, receives output data from the output unit 109. The trainer 123 uses the output data to train a classifier model. The trainer can be either a support vector machine or a deep learning based CNN model, or anything else.
[0049] It should be noted that the output data from the output unit 109 may be stored in a memory device instead of being input to the learning unit 123 .
[0050] Next, an example of an operation performed by the radar image teacher data generation system 20 according to the first embodiment of the operation mode will be described with reference to the flowchart shown in FIG.
[0051] The label information table is pre-created and stored in the label information storage 105. Also, markers of various IDs are pre-created (and associated with the corresponding classes).
[0052] When the teacher data generation method is started, the synchronization unit 101 transmits a synchronization signal to the marker detection unit 103 and the radar measurement unit 106 (step S101). At the same time as receiving the synchronization signal, the radar measurement unit 106 performs radar measurement (step S106), and the marker detection unit 103 performs supplementary signal detection (step S103). In step S103, the marker detection unit 103 scans the supplementary signal for the presence of a marker, and if a marker is found, identifies its ID and provides it as an output. The label generation unit 104 uses the marker ID to identify a class name from the label information table stored in the label information storage 105 and outputs an annotation (step S104). The imaging unit 107 performs imaging on the radar signal received from the radar measurement unit 106 to generate a radar image (step S107). The process flow ends with the generation of the radar image in step S107 and the identification of the class name in step S104. The output of this process flow is a radar image with annotations consisting of class names. For ease of understanding, one may refer to Figure 1 for an example image with associated class labels shown superimposed on the image. Here, the object of interest is a gun with class name "Gun1".
[0053] As described above, the radar image training data generation system 20 according to the first embodiment of the present disclosure automatically labels radar images with class names, which dramatically reduces labeling time and eliminates labeling errors by automatically (no need to manually identify class names by looking at the image) inferring class labels from marker IDs.
[0054] The first effect of the teacher data generation device 10 of the first embodiment is to dramatically reduce the manual effort of labeling (which in turn leads to a reduction in the required time). This occurs because the class name (for classification tasks) is automatically inferred from the marker ID by referring to a pre-recorded database. Therefore, the automatic labeling process does not require any manual effort (e.g., visual confirmation of the image) and, in addition, reduces the required labeling time.
[0055] The second effect of the teacher data generation device 10 of the first embodiment is to reduce labeling errors, which in turn improves the performance of the classifier network. This is also possible by automating the labeling process, i.e., automatically inferring class names from marker IDs using a pre-recorded database. This automation eliminates errors in the teacher data generation process that occur when identifying classes manually (as long as the unit functions as intended).
[0056] EMBODIMENT 2 The teacher data generation device 10 and the teacher data generation system 20 of the second embodiment are the same as the teacher data generation device 10 and the teacher data generation system 20 of the first embodiment, respectively, except for the points described below.
[0057] As described in embodiment 1, the first problem with manual labeling of radar images is that the low resolution, lack of color information, and small object size make it difficult to accurately identify class types. The same challenge makes it difficult to manually place bounding box labels. A good bounding box label is object-centered, and the box is tight around the object. The nature of the problem image makes it difficult to precisely manually locate objects, which in turn makes it difficult to manually center bounding box labels on objects. In addition, it is difficult to manually place tight labels of appropriate size on each object. This also increases the time required for labeling.
[0058] A second problem with manual labeling of radar images is labeling errors. First, class name identification is prone to errors due to the nature of the images in question. The same challenge leads to mistakes when manually centering objects with tight bounding box labels. Such labeling errors then adversely affect the performance of classifiers and / or object detection networks.
[0059] The teacher data generation device 10 of embodiment 2 provides an automatic labeling (or auto-labeling) system for radar images that can solve the first problem by generating class names and bounding box labels automatically, i.e., with increased speed.
[0060] The teacher data generation device 10 of embodiment 2 further provides an automatic labeling (or auto-labeling) system for radar images that can solve the second problem by eliminating labeling errors through automation of the labeling process (negligible human intervention is required).
[0061] Next, the configuration of the teacher data generation system 20 according to the second embodiment will be described with reference to the block diagram shown in FIG. 12. In this configuration of the teacher data generation system 20, the objective is to determine the object location in addition to the class label in order to learn an object detection (and localization) model in the radar image. The object location is specified in the form of a rectangular bounding box to the learning model (label size and bounding box size refer to the same thing and will be used interchangeably from now on). Furthermore, assuming that the marker location is the same as the object (center) location, from now on any reference to the marker location will in fact refer to the object location. Also assume that it is possible to determine the marker location in the supplementary signal. Therefore, the marker location is determined in the supplementary signal and transferred to the radar image. The transfer of the location requires additional knowledge of the supplementary sensor and radar antenna location. The marker location in the radar image will be the center location of the bounding box. If the marker location is not the same as the object location, some corrections will be required when generating the label. Since object detection models are sensitive to bounding box size, the width and height are identified for each image using the detected marker ID information (unique for each class) instead of using a constant predefined size. Finally, the corresponding radar image is annotated with the identified bounding box in addition to the class label.
[0062] The radar image teacher data generation system 20 according to the second embodiment may include a synchronization unit 101, a marker detection unit 103, a label generation unit 104, a label information storage 105, a coordinate conversion unit 208, a position information storage 209, a radar measurement unit 106, an imaging unit 107, and an output unit 109. Next, the label generation unit 104 is composed of a label size specification unit 204a, a class information specification unit 204b, and a rectangular label generation unit 204c. Note that the configurations of the synchronization unit 101, the radar measurement unit 106, the imaging unit 107, the output unit 109, the camera 122, and the receiver antenna 121 are the same as those described in the first embodiment, and therefore the description will not be repeated.
[0063] As described in the first embodiment, the marker detection unit 103 outputs the marker IDs of the markers detected in the supplemental signal to the label generation unit 104. In addition to this output, the marker detection unit 103 also outputs the positions of the markers in world coordinates to the coordinate transformation unit 208. If the supplementary sensor is a camera, an AR marker can be used as described in the first embodiment, and the aruco module of OpenCV can be used to output the position of the AR marker in world coordinates.
[0064] Compared with the first embodiment, the teacher data generation system 20 according to the second embodiment may include a coordinate conversion unit 208. The coordinate conversion unit 208 receives the position of the marker in the world coordinate as an input from the marker detection unit 103, and outputs the position of the marker in the radar image coordinate to the label generation unit 104. As is clear, the coordinate conversion unit 208 is included to convert the marker position from the world coordinate to the radar image coordinate. The coordinate conversion unit 208 utilizes the camera and radar antenna position information to convert the marker position from the world coordinate to the radar image coordinate. The camera and radar antenna position information is stored in the position information storage 209. The coordinate conversion unit 208 then outputs the marker position in the radar image coordinate.
[0065] In this configuration, the label generator 104 is composed of several sub-units, namely, a label size specifier 204a, a class information specifier 204b, and a rectangle label generator 204c. Both the label size specifier 204a and the class information specifier 204b refer to a label information table in the label information storage 105. The structure of the label information table is slightly different from that of the first embodiment and is explained using the example table given in FIG. 13. The table has four columns, namely, marker ID (primary key), label width, label height, and class name. There are two additional columns, label width and label height, to specify the appropriate bounding box size. As in the first embodiment, the number of rows depends on the number of unique classes. The table is created manually in advance (using prior knowledge of markers, object sizes, and class names) and stored in the storage 105.
[0066] The label size determination unit 204a is responsible for determining the size of the bounding box label once a name is proposed and providing it as output to the rectangle label generation unit 204c. The label size determination unit 204a receives the input marker ID from the marker detection unit 103 and the label information table from the storage 105. The label size determination unit 204a then uses the marker ID to look up the label width and label height from the label information table.
[0067] The class information identification unit 204b is responsible for identifying class name information and providing it as output to the rectangle label generation unit 204c. Similar to the label size identification unit 204a, the class information identification unit 204b also receives a marker ID (from the marker detection unit 103) and a label information table (from the storage 105) as input. The class information identification unit 204b uses the marker ID to search for a class name in the label information table.
[0068] The rectangular label generator 204c is responsible for generating annotations of the radar image, including bounding box and class label information, and providing them as output to the learning unit 123 through the output unit 109. This rectangular label generator 204c receives the following inputs: marker position in the radar image coordinates from the coordinate conversion unit 208, label width and height from the label size specification unit 204a, and class name from the class information specification unit 204b. The rectangular label generator 204c then generates annotations of the radar image consisting of the bounding box center position, the bounding box size (width, height), and the class name. The annotation format can be in the form of a simple text file or in the form of tags, or any other format, depending on the learning model being used. The output of this rectangular label generator 204c is the annotation, which can also be considered as the output of the label generator 104.
[0069] In some configurations, the class names, bounding box centers, and bounding box sizes may also be directly fed to the learning model as output without any post-processing.
[0070] A trainer 123, i.e., a machine learning based object detection model, receives the radar images and associated annotations from the output unit 109. The trainer 123 uses them to train an object detection model. The trainer can be either a sliding window based object detector or a deep learning based CNN model, or any other.
[0071] Next, an example of an operation performed by the radar image teacher data generation system 20 according to the second embodiment of the operation mode will be described with reference to the flowchart shown in FIG.
[0072] The label information table is created in advance and stored in the label information storage 105. Also, markers of various IDs are created in advance (associated with corresponding classes and label sizes). Note that steps S201, S206, and S207 in Fig. 14 are the same as steps S101, S106, and S107 in Fig. 11, respectively, and therefore their description will not be repeated.
[0073] In step S203, similar to step S103 in the first embodiment, the marker detection unit 103 captures the supplementary signal when it receives the synchronization signal from the synchronization unit 101. Then, in this step S203, the marker detection unit 103 scans the supplementary signal for the presence of a marker, and if the marker is found, the marker ID is provided as an output to the label generation unit 104, and further, the marker position in the world coordinate is provided as an output to the coordinate conversion unit 208. Using the marker ID, the label size (the label size identification unit 204a) identifies the label size and the class information identification unit 204b identifies the class name from the label information table stored in the label information storage (step S204). The coordinate conversion unit 208 converts the marker position in the world coordinate to the radar image coordinate (step S208). The rectangular label generation unit 204c generates a rectangular label annotation using the label size, the class name, and the marker position in the radar image coordinate (step S205). The process flow ends with the rectangle label generation in step S205. The output of this process flow is a radar image with associated rectangle label annotations. Figure 2 shows an example image with rectangle label annotations superimposed for ease of understanding. In this example image, the object of interest is a gun with class name "Gun1" and its location is highlighted by a tight rectangular bounding box centered on the gun.
[0074] As described above, the radar image teacher data generation system 20 according to the second embodiment automatically generates class names and bounding box annotations for radar images, which dramatically reduces labeling time and eliminates labeling errors due to automatic (no need to manually identify class names or determine label sizes by looking at images) inference of class labels and label sizes from marker IDs.
[0075] The first effect of the teacher data generating device 10 of the second embodiment is to dramatically reduce the manual effort of labeling (which in turn leads to a reduction in the required time). This occurs because the class name (for classification tasks) and the bounding box label size (for object detection tasks) are automatically inferred from the marker ID by referring to a pre-recorded database. Also, the marker position is automatically identified. Therefore, the automatic labeling process does not require any manual effort (e.g., visual confirmation of the image) and, in addition, reduces the required labeling time.
[0076] The second effect of the teacher data generating device 10 of the second embodiment is to reduce labeling errors, which in turn improves the performance of the classifier / object detection network. This is also possible by automating the labeling process, i.e., automatically identifying the marker position using a pre-recorded database and inferring the class name or bounding box size from the marker ID. This automation eliminates errors in the teacher data generation process that occur when manually identifying the class or manually determining the bounding box size / position (as long as the unit functions as intended).
[0077] EMBODIMENT 3 The teacher data generation device 10 and the teacher data generation system 20 of the third embodiment are the same as the teacher data generation device 10 and the teacher data generation system 20 of the second embodiment, respectively, except for the points described below.
[0078] Next, the configuration of the teacher data generation system 20 according to the third embodiment of the present disclosure will be described with reference to the block diagram shown in FIG. 15. This configuration of the teacher data generation system 20 describes additional functions to the configuration described in the second embodiment. In this configuration of the teacher data generation system 20, the objective is to adjust the label size (and / or class name) according to the object direction in addition to identifying the object position and class name in order to learn an object detection (and localization) model for radar images. Here, it is further assumed that the marker direction corresponds to the object direction, and hereafter both terms will be used interchangeably.
[0079] One of the most commonly occurring situations is described with the help of Fig. 16, where it may be necessary to adjust the label size according to the object orientation. In this figure, two example images Fig. 16(a) and (b) are shown, where an object, a knife, is shown with a superimposed bounding box label. It can be seen that the two images differ in the orientation of the knife, and the change in orientation leads to the correct amount of change in the bounding box size dimension. Since the requirement is to generate a tight bounding box for the object, the object orientation also needs to be taken into account while specifying the label size.
[0080] The radar image teacher data generation system 20 according to the third embodiment may include a synchronization unit 101, a marker detection unit 103, a label generation unit 104, a label information storage 105, a coordinate conversion unit 208, a position information storage 209, a class registration unit 310, a radar measurement unit 106, and an imaging unit 107. Next, the label generation unit 104 is composed of a label size identification unit 204a, a class information identification unit 204b, and a rectangular label generation unit 204c. Note that the configurations of the synchronization unit 101, the marker detection unit 103, the camera 122, the radar measurement unit 106, the receiver antenna 121, the imaging unit 107, the coordinate conversion unit 208, the position information storage 209, the label size identification unit 204a, the class information identification unit 204b, and the rectangular label generation unit 204c are the same as those described in the second embodiment, and therefore will not be described repeatedly.
[0081] As described in embodiment 2, the marker detection unit 103 detects markers in the camera image and outputs the marker ID to the label generation unit 104 and its position in the world coordinate system to the coordinate transformation unit 208. In addition to these outputs, the marker detection unit 103 also outputs the 3D orientation of the marker (in the world coordinate system) in the form of Euler angles, i.e., yaw, pitch, and roll, to the class registration unit 310.
[0082] Compared with the second embodiment, the teacher data generation system 20 according to the third embodiment may include a class registration unit 310. The class registration unit 310 is included to correct the class name and / or label size information by utilizing the angle information of the marker direction. Before proceeding to the description of the class registration unit 310, the label information storage 105 will first be described.
[0083] The label information storage 105 of this configuration, which is different from the second embodiment, is composed of multiple label information tables. Each of the tables is designed for a predetermined range of marker orientation angles. For example, there is a table A for the marker orientation angle range of 0° to 30°, and there is a table B for 30° to 60°, etc. Each of the label information tables stored in the label information storage 105 has the same structure as that described in the second embodiment, and therefore the description will not be repeated. Several exemplary label information tables are shown in FIG. 17(a). Each of them is tagged with a name (table A or table B, or other). Furthermore, the column values (in this case, label width and label height) are different for the same marker ID between different tables.
[0084] As in the second embodiment, the label information storage unit 105 transmits only a single label information table to the label generation unit 104. However, in this configuration, the label information storage unit 105 requires an additional input to indicate which table (among the multiple tables) it should transmit to the label generation unit 104 (specifically, the label size specification unit 204a and the class information specification unit 204b). This additional input comes from the class registration unit 310, and will be described in detail next.
[0085] The class registration unit 310 receives the marker's 3D orientation angle from the marker detection unit 103 as input and outputs the label information table name to the label information storage 105. The class registration unit 310 selects the appropriate angle among yaw, roll, or pitch and adjusts it (if necessary) by using information about the radar and camera positions. The class registration unit 310 then uses the selected angle to select the appropriate label information table name by looking up another table called registration class table. The structure of the registration class table is shown using an example in FIG. 17(b), which has two columns: angle bin (primary key) and table name. The number of rows depends on the number of angle bins. Based on the angle value, the appropriate angle bin is selected and the corresponding table name is searched. The output of the class registration unit 310 is the label information table name.
[0086] The operation of the class registration unit 310 will be made clearer with the help of several examples. In the first exemplary operation, the objective is to adjust the label size according to the angle of the marker orientation, as introduced in FIG. 16. In this case, the class registration unit 310 should select an angle corresponding to the rotation of the marker in the x-plane. Considering the relative positions of the camera and the target (from the configuration shown in FIG. 3), the yaw captures the rotation of the marker in the x-plane, and therefore the yaw is selected. If the camera plane is different from the radar plane, an adjustment of the selected angle may be required. However, in this case, since the radar and the camera are in the same plane, no adjustment of the angle value is required. Now, the selected (adjusted) angle is used to select the label information table name by referring to the registration class table. If the yaw=25°, the yaw falls within the bin of 0°-30° in the registration class table in FIG. 17(b). The table name corresponding to this row is A, which is the output of the registration class unit. When the label information storage 105 receives the table name A from the class registration unit 310 , it transmits the label information table A to the label generation unit 104 .
[0087] In another exemplary operation, the objective is to change the class name according to the marker rotation around the y-axis. In the case of radar images, the object appearance changes according to the inclination of the object appearance with respect to the radar plane. Therefore, it makes sense to learn the same object as different classes depending on the angle of inclination of the object. An exemplary label information table for this purpose is shown in FIG. 18(a), where again each of the label information tables is tagged with a name (table A or table B, or other). As expected, the values of the class name column are different for the same marker ID between different tables. In this exemplary operation, the enrollment class unit selects the pitch angle since the pitch angle corresponds to the rotation around the y-axis. Again, no adjustment of the angle value is necessary since the camera and radar plane are the same. Next, the enrollment class table is referenced to look up the name of the label information table to be used. An exemplary enrollment class table is shown in FIG. 18(b), where pitch=20°, the pitch falls within the bin of 15°-30°, and the table name corresponding to this row is B. The output of the class registration unit 310, that is, table B, is sent to the label information storage 105. Next, the label information storage 105 sends the label information table B to the label generation unit 104.
[0088] Note that there may be scenarios where the camera and radar are not in the same plane. For example, the camera may be placed in a plane perpendicular to the radar plane. In this case, the class registration unit 310 needs to correct all angles by 90°.
[0089] Next, an example of an operation performed by the radar image teacher data generation system 20 according to the third embodiment of the operation mode will be described with reference to the flowchart shown in FIG.
[0090] The label information table is created in advance and stored in the label information storage 105. Also, markers of various IDs are created in advance (associated with corresponding classes and label sizes). Note that steps S301, S304, S305, S306, S307, and S308 in Fig. 19 are the same as steps S201, S204, S205, S206, S207, and S208 in Fig. 14, respectively, and therefore their description will not be repeated.
[0091] In step S303, similar to step S103 in the first embodiment, the marker detection unit 103 scans the captured camera image for the presence of a marker, and if a marker is found, in addition to the marker ID output (to the label generation unit 104) and the marker position output (to the coordinate transformation unit 208), the 3D orientation of the marker is also given as an output to the class registration unit 310. In step S309, the class registration unit 310 receives the 3D marker orientation as an input and selects an appropriate angle (adjusting it, if necessary). In this step, the selected angle is used to select the name of the label information table by referring to the registration class table. The name of the label information table is the output of this step. The selected label information table is used in step S304. The output of this process flow is a radar image with an associated rectangular label annotation.
[0092] As described above, the radar image teacher data generation system 20 according to the third embodiment of the present disclosure automatically generates class names and bounding box annotations for radar images, which dramatically reduces labeling time and eliminates labeling errors due to automatic (no need to manually identify class names or determine label sizes by looking at images) inference of class labels and label sizes from marker IDs.
[0093] EMBODIMENT 4 The teacher data generation device 10 and the teacher data generation system 20 of the fourth embodiment are the same as the teacher data generation device 10 and the teacher data generation system 20 of the first embodiment, respectively, except for the points described below.
[0094] Next, the configuration of the teacher data generation system 20 for radar images according to the fourth embodiment will be described with reference to the block diagram shown in FIG. 20. In this configuration of the teacher data generation system 20, the objective is to identify the class labels of the radar images to train a classifier, similar to the first embodiment. However, in this embodiment, it is assumed that the markers have some pattern that encodes the class label information. This means that when the camera detects the marker, the camera may use some post-processing to decode the class name from the marker pattern. Such a configuration eliminates the need to refer to any additional table that links the marker ID to the class name (the label information table mentioned in the previous embodiment). An example of such a marker is a QR code. Finally, the corresponding radar image is annotated with the decoded class label.
[0095] 20, one of the configurations of the teacher data generation system 20 according to the second embodiment may include a synchronization unit 101, a marker detection unit 103, a label generation unit 104, a radar measurement unit 106, and an imaging unit 107. The marker detection unit 103 includes a camera measurement unit 103a and a QR code detection unit 403b. Note that the configurations of the synchronization unit 101, the camera measurement unit 103a, the camera 122, the radar measurement unit 106, the receiver antenna 121, the imaging unit 107, and the learning unit 123 are the same as those described in the first embodiment, and therefore the description will not be repeated.
[0096] The QR code detector 403b receives the camera image as input from the camera measurement unit 103a and outputs a class name to the label generator 104. This unit is responsible for detecting the presence of a QR marker in the camera image and decoding its pattern to extract the class name (standard libraries known in the art may be used). This QR code detector 403b has prior knowledge of how the QR code is generated, i.e. how the class name is encoded in the pattern. It then outputs the class name.
[0097] The label generator 104 receives the class names from the marker detector 103 and outputs the class name annotations to the learning unit 123 through the output unit 109. This label generator 104 is responsible for generating annotations for the radar images. The annotation format can be in the form of a simple text file or made up of tags or any other format depending on the learning model used.
[0098] In some configurations, the class names may also be provided directly as output (to the learning model) without any post-processing.
[0099] The process of generating a QR code will be described with reference to the block diagram shown in Fig. 21. As shown in Fig. 21, one configuration of a QR code generation system 450 according to the fourth embodiment may include a QR code generation unit 451 and a label information storage 452. Note that the configuration of the label information storage 452 is the same as that of the label information storage 105 described in the first embodiment, and therefore the description will not be repeated.
[0100] The QR code generator 451 receives as input the label information table of the example shown in FIG. 10 and outputs a QR code marker. For each row in the table, the marker ID and class name are searched. Only the class name is encoded in the QR code (using a standard QR code library). The marker ID in this case is not encoded, but is used only for management purposes for the operator to identify (without decoding) which QR code corresponds to which class. The QR code generator system 450 may be included in the teacher data generation system 20 or may be provided separately from the teacher data generation system 20. The label information storage 452 may also function as the label information storage 105 or may be provided separately from the label information storage 105.
[0101] The teacher data generation system 20 of the fourth embodiment requires a QR code to function. Therefore, the QR code generation operation mode will be described first with reference to the flowchart shown in Fig. 22. The label information table is created in advance and stored in the label information storage 452 of the QR code generation system 450.
[0102] At the start, counter i is initialized to 0. Fetch the class name from the ith row of the label information table (step S451). Encode the fetched class name in a QR code and tag the generated QR code with i for management purposes (step S452). A check is made to ensure if QR codes have been generated for all classes (step S453). If yes, the process ends. If no, increment counter i by 1 (step S454) and repeat the process from step S451.
[0103] Next, an example of an operation performed by the radar image teacher data generation system 20 according to the fourth embodiment of the operation mode will be described with reference to the flowchart shown in Fig. 23. Note that steps S401, S406, and S407 in Fig. 23 are the same as steps S101, S106, and S107 in Fig. 11, respectively, and therefore their description will not be repeated.
[0104] Upon receiving the synchronization signal, simultaneously, the radar measurement unit 106 performs radar measurement (step S406) and the marker detection unit 103 performs camera image capture (step S403). In step S403, the QR code detection unit 403b scans the captured camera image for the presence of a QR code, and if a QR code is found, the QR code is decoded and a class name is extracted therefrom, and the extracted class name is provided as output to the label generation unit 104.
[0105] In step S404, the class names received from the marker detection unit 103 are used to generate annotations for the radar image that are provided as output to the learning unit 123 through the output unit 109. The output of this process flow is a radar image with annotations consisting of class names. For ease of understanding, one may refer to FIG. 1 for an exemplary image with associated class labels shown superimposed on the image. Here, the object of interest is a gun with class name "Gun1".
[0106] As described above, the radar image teacher data generation system 20 according to the fourth embodiment automatically generates class name labels for radar images, which dramatically reduces the labeling time and eliminates labeling errors by automatically (no need to manually identify class names by looking at the image) inferring class labels from marker IDs.
[0107] EMBODIMENT 5 The teacher data generation device 10 and the teacher data generation system 20 of the fifth embodiment are the same as the teacher data generation device 10 and the teacher data generation system 20 of the second embodiment, respectively, except for the points described below.
[0108] Next, the configuration of the teacher data generation system 20 for radar images according to the fifth embodiment will be described with reference to the block diagram shown in FIG. 24. In this configuration of the teacher data generation system 20, the purpose, which is the same as in the second embodiment, is to identify class labels and object positions to learn an object detection model for radar images. However, in this embodiment, it is assumed that the markers have some pattern that encodes the label size in addition to the class label information. This means that when the camera detects the marker, the camera may use some post-processing to decode the label size and class name from the marker pattern. Such a configuration eliminates the need to refer to any additional table (the label information table mentioned in the previous embodiment) that links the marker ID to the label size and class name. An example of such a marker is a QR code as introduced in the fourth embodiment of the present disclosure. Finally, the corresponding radar image is annotated with a rectangular bounding box of the label size decoded from the QR code and with the decoded class label.
[0109] The radar image teacher data generation system 20 according to the fifth embodiment may include a synchronization unit 101, a camera measurement unit 103a, a QR code detection unit 403b, a label generation unit 104, a coordinate conversion unit 208, a position information storage 209, a radar measurement unit 106, and an imaging unit 107. Note that the configurations of the synchronization unit 101, the camera measurement unit 103a, the camera 122, the coordinate conversion unit 208, the position information storage 209, the radar measurement unit 106, the receiver antenna 121, and the imaging unit 107 are the same as those described in the second embodiment, and therefore the description will not be repeated.
[0110] The QR code detector 403b receives the camera image as input from the camera measurement unit 103a and outputs the class name and label size to the label generator 104 and the marker position to the coordinate converter 208. This unit is responsible for detecting the presence of a QR marker in the camera image and decoding its pattern to extract the label width, label height and class name (standard libraries known in the art are used). This QR code detector 403b has prior knowledge of how the QR code is generated to encode said information. In addition to this, this QR code detector 403b is also responsible for extracting the position of the QR code marker (standard libraries known in the art are used). The marker position is in the world coordinate system. This QR code detector 403b then outputs the marker position, label width, label height and class name.
[0111] The label generator 104 receives the following inputs: label height, label width, and class name from the QR code detector 403b, and marker position in radar coordinates from the coordinate converter 208, and outputs bounding box annotations to the learning unit 123 through the output unit 109. The label generator 104 generates rectangular label annotations of the radar image consisting of the bounding box center position, the bounding box size (width, height), and the class name. The annotation format can be in the form of a simple text file or in the form consisting of tags, or any other format, depending on the learning model being used. The output of the label generator 104 is a rectangular label annotation of the corresponding radar image.
[0112] In some configurations, the class names, bounding box centers, and bounding box sizes may also be directly given as output (to the learning model) without any post-processing.
[0113] The process of generating a QR code will be described with reference to the block diagram shown in Fig. 25. As shown in Fig. 25, one configuration of a QR code generation system 550 according to the fifth embodiment may include a QR code generation unit 551 and a label information storage 552. Note that the configuration of the label information storage 552 is the same as that of the label information storage 105 described in the second embodiment, and therefore the description will not be repeated.
[0114] The QR code generator 551 receives as input the example label information table shown in FIG. 13 and outputs a QR code marker. For each row in the table, the marker ID, label width, label height, and class name are retrieved. The label width, label height, and class name are encoded in the QR code (using standard libraries known in the art). The marker ID is not encoded, but is used only for management purposes for an operator to identify (without decoding) which QR code corresponds to which class. The QR code generator system 550 may be included in the teacher data generation system 20 or may be provided separately from the teacher data generation system 20. The label information storage 552 may also function as the label information storage 105 or may be provided separately from the label information storage 105.
[0115] The teacher data generation system 20 of the fifth embodiment requires a QR code to function. Therefore, the QR code generation operation mode will be described first with reference to the flowchart shown in Fig. 26. The label information table is created in advance and stored in the label information storage 552 of the QR code generation system 550.
[0116] At the start, counter i is initialized to 0. Fetch the class name and label size (width, height) from the i-th row of the label information table (step S551). Encode the fetched class name and label size in a QR code and tag the generated QR code as i for management purposes (step S552). A check is made to ensure if QR codes have been generated for all classes (step S553). If yes, the process ends. If no, increment counter i by 1 (step S554) and repeat the process from step S551.
[0117] Next, an example of an operation performed by the radar image teacher data generation system 20 according to the fifth embodiment of the operation mode will be described with reference to the flowchart shown in Fig. 27. Note that steps S501, S506, S507, and S508 in Fig. 27 are the same as steps S201, S206, S207, and S208 in Fig. 14, respectively, and therefore their description will not be repeated.
[0118] Upon receiving the synchronization signal, at the same time, the radar measurement unit 106 performs radar measurement (step S506) and the marker detection unit 103 performs camera image capture (step S503). In step S503, the QR code detection unit 403b scans the captured camera image for the presence of a QR code, and if a QR code is found, it is decoded. The label size (width, height) and class name are extracted therefrom and given as output to the label generation unit 104. The QR code marker position is also extracted in world coordinates and sent to the coordinate transformation unit 208 in this step.
[0119] In step S504, the class name and label size received from the QR code detector 403b and the marker position received from the coordinate converter 208 are used to generate a rectangular label annotation for the radar image. The output of this process flow is a radar image with associated rectangular label annotations. Figure 2 shows an example image with rectangular label annotations superimposed for ease of understanding. In this example image, the object of interest is a gun with class name "Gun1" and its location is highlighted by a tight rectangular bounding box centered on the gun.
[0120] As described above, the radar image teacher data generation system 20 according to the fifth embodiment automatically generates class names and bounding box annotations for radar images. This dramatically reduces labeling time and eliminates labeling errors due to automatic (no need to manually identify class names or determine label sizes by looking at images) inference of class labels and label sizes from marker IDs.
[0121] As described above, the embodiments of the present invention have been described with reference to the attached drawings, but these embodiments are merely examples of the present invention, and various combinations of the above-mentioned embodiments and configurations other than those of the above-mentioned embodiments may also be adopted.
[0122] All or part of the embodiments disclosed above may be described as, but are not limited to, the following supplementary notes. 1-1. an acquisition unit for acquiring a radar image based on a scattering signal from an object; a marker detection unit that acquires a supplementary image of an area including the object, the supplementary image being captured in synchronization with detection of the scattering signal, detects a marker in the supplementary image, and extracts an identifier of the marker; a label generator that generates a label using the extracted identifier; an output unit that outputs output data in which the radar image and the generated label are associated with each other; A teacher data generating device comprising: 1-2. In the teacher data generation device according to 1-1, A teacher data generating device, wherein the supplementary image is a camera image, a depth image, or an IR image. 1-3. In the teacher data generation device according to 1-1. or 1-2., The teacher data generating device, wherein the marker is an AR marker or a QR code. 1-4. In the teacher data generation device according to any one of 1-1 to 1-3, A teacher data generation device, wherein the label generation unit generates the labels by further using a label information table in which the identifiers and the labels are associated with each other. 1-5. A teacher data generation device according to any one of 1-1. to 1-4., wherein the generated labels include a class name label indicating a type of the object. 1-6. In the teacher data generation device according to any one of 1-1 to 1-5, A training data generating device, wherein the generated labels include bounding box labels indicating the position and size of the object. 1-7. In the teacher data generation device according to 1-6, The marker detection unit detects the position of the marker, The teacher data generation device further includes a coordinate conversion unit that converts the detected position of the marker into coordinates in the radar image, The label generation unit generates the bounding box label based on the coordinates of the detected marker in the radar image. 1-8. In the teacher data generation device according to any one of 1-1 to 1-7, A teacher data generating device, wherein the marker is recognizable in the supplementary image but not in the radar image.
[0123] 2-1. A teacher data generation device according to any one of 1-1. to 1-7.; a measurement unit that detects the scattered signal by transmitting electromagnetic waves to the object and detecting the electromagnetic waves scattered by the object; A supplementary imaging unit that captures the supplementary image; A teacher data generation system comprising: 2-2. In the teacher data generation system according to 2-1, A teacher data generation system, wherein the teacher data generation device further comprises a synchronization unit for synchronizing the measurement unit and the supplementary imaging unit.
[0124] 3-1. A teacher data generation method performed by a computer, comprising the steps of: acquiring a radar image based on a scattered signal from an object, and acquiring a supplemental image of an area including the object, the supplemental image being captured synchronously with detection of the scattered signal; Detecting a marker in the captured image; extracting an identifier for the marker; generating a label using the extracted identifier; and outputting output data in which the radar image and the generated labels are associated with each other; A method for generating teacher data, comprising: 3-2. In the teacher data generation method according to 3-1, A teacher data generation method, wherein the supplementary image is a camera image, a depth image, or an IR image. 3-3. In the teacher data generation method according to 3-1. or 3-2., A teacher data generating method, wherein the marker is an AR marker or a QR code. 3-4. In the teacher data generation method according to any one of 3-1. to 3-3., A training data generating method, wherein the label is generated further using a label information table in which the identifier and the label are associated with each other. 3-5. In the teacher data generation method according to any one of 3-1. to 3-4., A training data generation method, wherein the generated labels include a class name label indicating a type of the object. 3-6. In the teacher data generation method according to any one of 3-1. to 3-5., A training data generation method, wherein the generated labels include bounding box labels indicating the position and size of the object. 3-7. In the teacher data generation method according to 3-6. Detecting the position of the marker; Transforming the detected positions of the markers into coordinates within the radar image; generating the bounding box label based on the coordinates of the detected markers in the radar image; The teacher data generating method further includes: 3-8. In the teacher data generation method according to any one of 3-1. to 3-7., A method for generating teacher data, wherein the marker is visible in the supplemental image and is not visible in the radar image.
[0125] 4-1. A program for causing a computer to execute a teacher data generation method, the teacher data generation method comprising: acquiring a radar image based on a scattered signal from an object, and acquiring a supplemental image of an area including the object, the supplemental image being captured synchronously with detection of the scattered signal; Detecting a marker in the captured image; extracting an identifier for the marker; generating a label using the extracted identifier; and outputting output data in which the radar image and the generated labels are associated with each other; Including, the program. 4-2. In the program described in 4-1., The program, wherein the supplemental image is a camera image, a depth image, or an IR image. 4-3. In the program described in 4-1. or 4-2., The program, wherein the marker is an AR marker or a QR code. 4-4. In the program according to any one of 4-1. to 4-3., The label is generated by further using a label information table in which the identifier and the label are associated with each other. 4-5. In the program according to any one of 4-1. to 4-4., The generated labels include a class name label indicating a type of the object. 4-6. In the program according to any one of 4-1. to 4-5., The generated labels include bounding box labels indicating a position and size of the object. 4-7. In the program described in 4-6., The teacher data generation method includes: Detecting the position of the marker; Transforming the detected positions of the markers into coordinates within the radar image; generating the bounding box label based on the coordinates of the detected markers in the radar image; The program further comprises: 4-8. In any one of the programs described in 4-1. to 4-7., The marker is visible in the supplemental image and not visible in the radar image.
Claims
1. an acquisition unit for acquiring a radar image based on a scattering signal from an object; a marker detection unit that acquires a supplementary image of an area including the object, the supplementary image being captured in synchronization with detection of the scattering signal, detects a marker in the supplementary image, and extracts an identifier of the marker; a label generator that generates a label using the extracted identifier; an output unit that outputs output data in which the radar image and the generated label are associated with each other; Equipped with the marker is visible in the supplemental image and not visible in the radar image; A teacher data generation device, wherein the object is not visible in the supplementary image.
2. 2. The teacher data generating device according to claim 1, A teacher data generating device, wherein the supplementary image is a camera image, a depth image, or an IR image.
3. 3. The teacher data generating device according to claim 1, A teacher data generating device, wherein the markers are printed on paper, wood, cloth, or plastic.
4. 4. The teacher data generating device according to claim 1, The teacher data generating device, wherein the marker is an AR marker or a QR code.
5. 5. The teacher data generating device according to claim 1, The label generation unit generates the labels by further using a label information table in which the identifiers and the labels are associated with each other.
6. 6. The teacher data generating device according to claim 1, A training data generating device, wherein the generated labels include a class name label indicating the type of the object.
7. 7. The teacher data generating device according to claim 1, A training data generating device, wherein the generated labels include bounding box labels indicating the position and size of the object.
8. The teacher data generating device according to any one of claims 1 to 7, a measurement unit that detects the scattered signal by transmitting electromagnetic waves to the object and detecting the electromagnetic waves scattered by the object; A supplementary imaging unit that captures the supplementary image; A teacher data generation system comprising:
9. A teacher data generation method performed by a computer, comprising: acquiring a radar image based on a scattered signal from an object, and acquiring a supplemental image of an area including the object, the supplemental image being captured synchronously with detection of the scattered signal; Detecting a marker in the captured image; extracting an identifier for the marker; generating a label using the extracted identifier; and outputting output data in which the radar image and the generated labels are associated with each other; Including, the marker is visible in the supplemental image and not visible in the radar image; A method for generating teacher data, wherein the object is not visible in the supplementary image.
10. A program for causing a computer to execute a teacher data generation method, the teacher data generation method comprising: acquiring a radar image based on a scattered signal from an object, and acquiring a supplemental image of an area including the object, the supplemental image being captured synchronously with detection of the scattered signal; Detecting a marker in the captured image; extracting an identifier for the marker; generating a label using the extracted identifier; and outputting output data in which the radar image and the generated labels are associated with each other; Including, the marker is visible in the supplemental image and not visible in the radar image; The object is not visible in the supplemental image, the program.
Citation Information
Patent Citations
Information processing method and information processing device
JP2018163096A
Learning data set preparing method, as well as object recognition and position attitude estimation method
JP2019029021A
Remote state following device
US20180014382A1
Radar-aided single image three-dimensional depth reconstruction
US20200286247A1
Learning dataset creation method and device
US20210012524A1
