Teacher data generation device, teacher data generation method, and program

Abstract

The teacher data generating device (10) includes a radar signal acquiring unit (108), an image generating unit (104), and an output unit (109). The radar signal acquiring unit (108) acquires radar signal data acquired by a radar. The image generating unit (104) generates a radar image to which motion blur has been added based on each of one or more velocity values. The image generating unit (104) generates the radar image by using the radar signal data. The output unit (109) outputs the radar image in association with the ground truth data. The ground truth data indicates at least whether a detection target has been detected in the radar signal data.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE Embodiments of the present invention relate generally to training data generation. [Background technology]

[0002] The present invention relates to a radar image object detection system in which radio waves scattered by a moving target object are measured and then image generation is performed to generate a radar image of the target under scan, said image is used by an object detection module to identify hidden objects, said object detection module being a machine learning network trained using supervised learning techniques on a training data set.

[0003] An example of a conventional object detection system for radar images is described in Non-Patent Document 1. This conventional object detection system includes a radar signal measuring means, an image generating means, and an object detecting means. Specifically, the measuring means includes a radar antenna that transmits radar waves and receives reflected scattered waves.

[0004] A conventional object detection system for radar imaging having such a structure operates as follows. First, the radar signals are measured. In the measurement step, the radar antenna transmits radar signals individually in a specific order, and the reflected waves are received by the antenna receiver. The measured radar signals are used by the image generation means to make a radar image by utilizing the radar antenna information. Finally, the generated radar image is used by the object detection means to detect the presence or absence of an object of interest in the radar image. Specifically, in Non-Patent Document 1, the object detection system for radar imaging is used to detect hidden weapons. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] L. Carrer, "Detecting hidden weapons: microwave imaging methods", Master's Scientific Thesis, Delft University of Technology, 2012 Summary of the Invention [Problem to be solved by the invention]

[0006] A supervised detection model performs well on a test dataset, particularly when the test dataset is similar to the training dataset in terms of data variation. In practice, it is difficult to predict all types of test scenarios that may be encountered, and one solution is to create a large and diverse training dataset. However, in many cases, collecting and / or labeling a large and diverse training dataset is impractical and time-consuming. In the case of radar images while training a machine learning-based object detection model, a similar problem occurs, i.e., mismatch between the training dataset and the test dataset leads to poor detection performance.

[0007] An example objective of the present invention is to improve radar image detection performance by reducing mismatch between training and test data sets. This is accomplished by augmenting the radar image training set in a realistic manner to cover a large number of test cases. In particular, the present disclosure relates to a data augmentation system for radar imagery that improves detection performance. [Means for solving the problem]

[0008] The present invention relates to a radar signal acquisition unit that acquires radar signal data acquired by a radar; an image generator that uses the radar signal data to generate a motion blurred radar image based on each of the one or more velocity values; an output unit that outputs a radar image in association with ground truth data that indicates at least whether a detection target has been detected by the radar signal data; The present invention provides a teacher data generation device comprising:

[0009] The present invention provides a teacher data generation method performed by a computer, acquiring radar signal data acquired by a radar; generating a motion blurred radar image based on each of the one or more velocity values ​​using the radar signal data; outputting the radar image in association with ground truth data indicating at least whether the detected target has been detected in the radar signal data; The present invention provides a method for generating teacher data, including:

[0010] The present invention is a program for causing a computer to execute a teacher data generation method, the teacher data generation method comprising the steps of: acquiring radar signal data acquired by a radar; generating a motion blurred radar image based on each of the one or more velocity values ​​using the radar signal data; outputting the radar image in association with ground truth data indicating at least whether the detected target has been detected in the radar signal data; The Company provides programs including: Effect of the Invention

[0011] An advantage of the present invention is that it improves the detection performance of an object detection system by reducing the mismatch between the training set rate and the test set rate. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating the configuration of a radar image measurement system and also illustrating the relative position of a target with respect to a radar antenna. [Diagram 2] FIG. 2 is a block diagram illustrating a functional configuration of the teacher data generation device according to the first embodiment. As shown in FIG. [Diagram 3] FIG. 3 is a block diagram illustrating a hardware configuration of a computer that realizes the teacher data generation device according to the first embodiment. [Figure 4] FIG. 4 is a flowchart illustrating a process flow performed by the teacher data generation device of the first embodiment. [Diagram 5] FIG. 5 is a block diagram illustrating a functional configuration of the teacher data generation device according to the first embodiment. As shown in FIG. [Figure 6] FIG. 6 is a block diagram illustrating a functional configuration of the teacher data generation device according to the first embodiment. As shown in FIG. [Figure 7] FIG. 7 illustrates an exemplary probability distribution of velocity values. [Figure 8] FIG. 8 is a flowchart illustrating the operation of the teacher data generation device according to the first embodiment. [Figure 9] FIG. 9 is a block diagram illustrating a function-based configuration of the teacher data generation device according to the second embodiment. As shown in FIG. [Figure 10] FIG. 10 is a flowchart illustrating the operation of the teacher data generation device according to the second embodiment. [Figure 11] FIG. 11 is a block diagram illustrating a functional configuration of a teacher data generation device according to the third embodiment. As shown in FIG. [Figure 12] FIG. 12 is a block diagram illustrating a function-based configuration of a teacher data generation device according to the third embodiment. [Figure 13] FIG. 13 is a flowchart illustrating the operation of the teacher data generation device according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present disclosure will now be described in detail with reference to the accompanying drawings, in which like reference numerals denote like elements throughout the drawings, and redundant description will be omitted where necessary for clarity.

[0014] First embodiment

[0015] <Summary> First, the configuration of the radar image object detection system is described with the aid of FIG. 1. The specific purpose of the system is to confirm whether a person (target 90) possesses any hidden dangerous object. When the target is walking in a fixed part (area 96), the system measures the target by a fixed antenna (radar 92) placed on a side panel 94. The transmitter in the antenna transmits a signal individually, and then the received scattered signal is acquired. The system may also acquire a camera image using a camera (camera 98) simultaneously with the radar signal. The radar signal is processed using the antenna information to generate a radar image that is essentially 3D. If the person (target 90) possesses a hidden dangerous object, the object will be visible in the radar image. Therefore, the radar image is used to detect the hidden dangerous object. A machine learning-based object detection model is trained to detect the presence of the dangerous object in the radar image. This machine learning-based object detection model may acquire the radar image either in a processed 2D form or in its original 3D form. In order to ensure good performance of the detection model in the testing phase, the training set must be similar to the test set in terms of data diversity. In short, there must be no mismatch between the training and test sets.

[0016] As mentioned above, during the measurement phase, the transmitters transmit individually, which makes the radar signal measurement procedure take some time. The movement of the target 90, i.e., walking during the measurement, causes each point of the target 90 to spread in the radar image. This occurs due to the target's position change during the scan and is called motion blur (hereafter simply called blur). The blur is proportional to the movement or walking speed value of the target 90, i.e., a larger speed value means a larger blur. This implies that if the speed value of the target 90 is different during the signal measurement of the two radar images, the two radar images will have different amounts of blur. This leads to problems during object detection, since the object shape changes with the amount of blur, i.e., different amounts of blur in different images make the object appear different. Thus, if the speed value of the target 90 during the training set measurement is different from the speed value of the target 90 during the test measurement, the object shape will be different in the training and test images. This mismatch between the walking speed of the training set and the walking speed of the test set leads to a decrease in the detection performance. This is because object detection models have difficulty detecting shapes that they have not learned during training. The obvious solution of covering the velocities of all targets during training set measurements is time-consuming and impractical.

[0017] The mismatch between the speed of the training data set and the speed of the test data set can be reduced in an effective and practical manner using the teacher data generation device 10 according to the first embodiment.

[0018] FIG. 2 shows an exemplary diagram of a function-based configuration of the teacher data generating device 10 according to the first embodiment. The teacher data generating device 10 of the first embodiment includes a radar signal acquiring unit 108, an image generating unit 104, and an output unit 109. The radar signal acquiring unit 108 acquires radar signal data acquired by a radar. The image generating unit 104 generates a radar image to which motion blur is added based on each of one or more speed values. The image generating unit 104 generates the radar image by using the radar signal data. The output unit 109 outputs the radar image in association with the ground truth data. The ground truth data indicates at least whether a detection target is detected in the radar signal data. Details will be described below.

[0019] Examples of detection targets include dangerous objects (such as knives and guns) possessed by the target 90. The radar signal acquisition unit 108 may acquire the radar signal data directly from the radar, or the radar signal acquisition unit 108 may acquire the radar signal data by reading the radar signal data acquired in advance by the radar and stored in a radar signal storage unit accessible from the radar signal acquisition unit 108. The radar signal storage unit may be included in the teacher data generation device 10 as described in FIG. 5, or the radar signal storage unit may be installed separately from the teacher data generation device 10. The correct answer data is previously associated with each of the radar signal data.

[0020] The output data output from the output unit 109 of the teacher data generating device 10 can be used as teacher data for training an object detection model. The output data includes a radar image based on the radar signal data. Here, the correct answer data associated with the radar signal data is associated with the radar image. The correct answer data may indicate at least one of the presence or absence, type (knife, gun, etc.), position, and direction of a detected target. The radar image included in the output data may be given a label indicating the correct answer data, or an identification code (e.g., a data number) that is individually determined for each of the radar signal data. In the latter case, reference data in which the identification code and the correct answer data are associated with each other is created separately. The radar image is associated with the correct answer data via the reference data.

[0021] It is preferable that the output data includes a plurality of radar images to which motion blur is added based on a plurality of velocity values ​​different from each other. These plurality of radar images may be generated from one radar signal data. Specifically, the image generating unit 104 may generate N images by using N velocity values. That is, the output data may include N radar images, where N is a positive integer. Using the teacher data generating device 10, a plurality of teacher data may be generated from one radar signal data. Furthermore, by training using a large number of teacher data, a highly accurate learned object detection model is provided.

[0022] It should be noted that the output data output from the output unit 109 may be stored so as to be accessible from the output unit 109, or may be directly input to a device that trains the object detection model.

[0023] A sample operation of the teacher data generating device 10 will be described below. First, the measurement stage of the radar signal will be described with reference to FIG. 1. Assume that a person (target 90) is walking in a screening area (area 96) in front of a fixed radar antenna (radar 92) placed on a side panel 94. When in the screening area, the target is also in the field of view of a camera (camera 98) that captures an image synchronized with the radar sensor. The measured scattered radar signal (radar signal data) is stored in the radar signal storage unit. During the test phase, this radar signal is processed to generate a radar image, which may be 2D or 3D, and is used as an input to a machine learning based model to detect whether the target possesses any dangerous object. The machine learning model needs to be trained before using it for detection. The teacher data generating device 10 operates in the training stage of the radar image object detection system to augment the measured image. Next, the operation of the teacher data generating device 10 will be described. During the operation, the radar signals are individually read from the radar signal storage unit and transmitted for image generation. The image generator 104, responsible for image generation, can generate one or more (N) radar images for each radar signal by adding a realistic motion blur to the original, thus augmenting it. The amount of motion blur is controlled or parameterized by a speed value, which in this case is a walking speed value. The output data, including the augmented images, can be used directly for training the model or can be stored in some image DB (storage unit) for later use in training the detection model.

[0024] It should be noted that during the testing phase (real-time operation) of the radar image object detection system, the radar signal can be directly processed to generate a radar image, which is then used by the trained machine learning model to detect whether the target possesses any dangerous objects.

[0025] Also, note that reference to the term velocity value implies both magnitude and direction.

[0026] <Example of hardware configuration> In some embodiments, each functional unit included in the teacher data generation device 10 may be implemented by at least one hardware component, and each hardware component may realize one or more of the functional units. In some embodiments, each functional unit may be implemented by at least one software component. In some embodiments, each functional unit may be implemented by a combination of hardware and software components.

[0027] The teacher data generating device 10 may be implemented on a dedicated computer manufactured for the purpose of implementing the teacher data generating device 10, or may be implemented on a general-purpose computer such as a personal computer (PC), a server machine, or a mobile device.

[0028] 3 is a block diagram illustrating an example of a hardware configuration of a computer 1000 that realizes the teacher data generation device 10 of the first embodiment. In FIG. 3, the computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a recording device 1080, an input-output (I / O) interface 1100, and a network interface 1120.

[0029] The bus 1020 is a data transmission channel for the processor 1040, the memory 1060, and the recording device 1080 to transmit and receive data to and from each other. The processor 1040 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 1060 is a primary recording device such as a random access memory (RAM). The recording medium 1080 is a secondary recording device such as a hard disk drive, a solid state drive (SSD), or a read only memory (ROM).

[0030] The I / O interface is an interface between the computer 1000 and peripheral devices such as a keyboard, a mouse, or a display device. The network interface is an interface between the computer 1000 and a communication line through which the computer 1000 communicates with another computer.

[0031] The recording device 1080 may store program modules, each of which is an implementation of a functional unit of the teacher data generation device 10. The CPU 1040 executes each program module, thereby realizing each functional unit of the teacher data generation device 10.

[0032] <Process flow> 4 is a flowchart showing a process sequence performed by the teacher data generating device 10 of the first embodiment. The radar signal acquiring unit 108 acquires radar signal data acquired by a radar (S12). The image generating unit 104 uses the radar signal data to generate a radar image to which motion blur has been added based on each of one or more speed values ​​(S14). The output unit 109 outputs the radar image in association with ground truth data indicating at least whether the detection target has been detected by the radar signal data (S16).

[0033] <Example of feature-based configuration> The configuration of a teacher data generation device 10 according to the first embodiment will be described with reference to the block diagram shown in FIG.

[0034] In the configuration of FIG. 5, the teacher data generating device 10 further includes a velocity generating unit 102. The velocity generating unit 102 generates one or more velocity values. The image generating unit 104 acquires the one or more velocity values ​​generated by the velocity generating unit 102 and generates a radar image by using each of the acquired one or more velocity values. The velocity generating unit 102 may generate the one or more velocity values ​​by, for example, randomly sampling one or more velocity values ​​from a velocity value distribution. In the configuration of FIG. 5, the velocity that parameterizes the motion blur is determined based on some prior knowledge, i.e., information from the radar signal is not directly used.

[0035] However, the teacher data generating device 10 does not have to include the velocity generating unit 102. In that case, the image generating unit 104 generates a radar image by using each of the one or more predetermined velocity values.

[0036] Note that the speeds are generated so that they are realistic and cover all possible variations in real test scenarios.

[0037] The details of the teacher data generating device 10 having the configuration shown in Fig. 5 are described below. The teacher data generating device 10 shown in Fig. 5 includes a radar signal storage unit 101, a radar signal acquisition unit 108, an antenna information storage unit 103, a velocity generation unit 102, an image generation unit 104, and an output unit 109. The radar signal storage unit 101 provides radar signals during data augmentation. The antenna information storage unit 103 provides antenna positions, orders, etc. for radar image generation. The velocity generation unit 102 generates one or more (N) velocity values. The image generation unit 104 generates N augmented radar images by adding realistic blur proportional to the generated velocities.

[0038] Further, it is assumed that a parameter N (>1) is predefined, which is used by the velocity generator 102 and the image generator 104. This parameter N denotes the number of augmented images to be generated for each radar signal.

[0039] The radar signal storage unit 101 contains radar signals stored during measurement time. It acts as a data source by providing the radar signals as input to the image generation unit 104.

[0040] The antenna information storage unit 103 includes information about antennas, such as the number of antennas, the order of transmission and reception, the position of antennas, etc. The antenna information is used by the image generation unit 104 during image generation. Note that the antenna information storage unit 103 may be installed separately from the teacher data generation device 10.

[0041] The radar signal acquisition unit 108 acquires radar signal data from the radar signal storage unit 101. The image generation unit 104 receives the radar signal data from the radar signal acquisition unit 108, receives antenna information from the antenna information storage unit 103, and generates N velocities [v1, v2, ..., v N ] from the velocity generator 102. The image generator 104 receives N extended radar images [I v1 , I v2 , ..., I vN]. The image generator 104 generates a radar image from the input radar signal by using any conventional imaging algorithm such as beamforming or MIMO-RMA or any other. For this purpose, antenna information input is also used. Image enhancement by blur addition is performed during image generation by using existing motion compensation techniques. The motion compensation techniques were invented for the purpose of motion blur compensation. However, in the teacher data generating device 10, the present technique is used instead to add controlled motion blur. The motion compensation technique is parameterized by a velocity value, and the resulting image is clear and has strong intensity when the velocity value used is the same or similar to the actual target velocity value. Following the same concept, if the velocity value of the parameters of the motion compensation is far from the actual target velocity value, the resulting image has motion blur instead. Therefore, it can be said that the motion blur is parameterized by a velocity value in the present technique. The velocity value is carefully selected by the velocity generator 102 to simulate a realistic blur in the radar image. A realistic blur implies that the augmented image is similar to the actual radar image. The image generator 104 receives the N velocities from the velocity generator 102 and thus generates N blurred images, i.e., N augmented images. It should be noted that one velocity among the N velocities may be set to be insignificant (e.g., set to 0) to ensure that the original unblurred image is also part of the N image output of the image generator 104.

[0042] The image generator 104 of the teacher data generating device 10 is further differentiated from existing motion compensation techniques, which generate a single radar image for each radar signal after compensating for target movement within the image. In the teacher data generating device 10, even if the image generator 104 utilizes the motion compensation technique, here the objective is to add blur rather than to compensate for blur. Furthermore, the image generator 104 may generate N(>1) radar images for each radar signal, each image having a different amount of blur.

[0043] The velocity generator 102 provides N velocities as input to the image generator 104 to generate the blur-added augmented image. The parameter N is pre-specified and known to the velocity generator 102. As mentioned above, the velocity generator 102 uses some prior knowledge to determine the velocities, and information from the radar signal is not directly used in the process.

[0044] One exemplary configuration of the velocity generator 102 will be described with reference to FIG. 6. Here, the velocity generator 102 is actually composed of two sub-units, namely a velocity identifier 102a and a velocity distribution storage unit 102b. Here, the velocity distribution storage unit 102b provides a velocity value distribution as an input to the velocity identifier 102a. The velocity identifier 102a receives the velocity value distribution as an input and provides N velocities as an output to the image generator 104. The velocity identifier 102a randomly samples these N velocities from the input velocity value distribution. The predetermined parameter N is known to the velocity identifier 102a.

[0045] In this exemplary configuration of the speed generation unit 102, it is assumed that a speed value distribution is pre-specified and stored in the speed distribution storage unit 102b. Said speed value distribution, more specifically, the walking speed value distribution, must be specified so as to be able to capture the realistic speeds encountered during the testing phase of the detection model. There may be many ways to specify a realistic speed value distribution, and one exemplary method is described below. Thus, the distribution is designed based on some prior knowledge. First, the statistical parameters of the distribution need to be specified. For known standard distributions, i.e., Gaussian distribution, uniform distribution, etc., it is sufficient that the mean value, standard deviation, minimum value, or maximum value is known. On the other hand, for customized distributions, the values ​​and their respective occurrence probabilities are required. In any case, the parameters can be calculated based on existing (walking) speed value statistics available on the Internet, i.e., average walking speed, standard deviation, etc., or using knowledge of the deployment scenario of the system, i.e., fast walking, slow walking, etc., or using data from some external speed value measurement device. Once the parameters are specified, a suitable distribution needs to be selected. Some of the example distributions, i.e. Gaussian, uniform, or some custom distribution, are shown in Figure 7. Finally, a speed value distribution can be designed using the above parameters and distributions. Examples of parameters for designing the speed value distribution are shown in Figure 7 as μ, σ, a, b, m, n, p1, p2, and p3.

[0046] The output unit 109 outputs N expanded radar images [I v1 , I v2 , ..., I vN ] from the image generation unit 104 to output an image, which may be used on the fly to train a detection model, or the image may also be stored in some radar image storage unit 111. The radar image storage unit 111 may be included in the teacher data generation device 10, or the radar image storage unit 111 may be located separately from the teacher data generation device 10 as shown in FIG.

[0047] Next, an example of an operation performed by the teacher data generation device 10 according to the first embodiment of the operation mode will be described with reference to the flowchart shown in Fig. 8. The radar signal has already been measured and stored in the radar signal storage unit 101. Furthermore, the antenna information has been stored in the antenna information storage unit 103. Furthermore, the parameter N has been determined in advance. In the case of the exemplary configuration shown in Fig. 6, the speed value distribution has been created in advance and stored in the speed distribution storage unit 102b.

[0048] When the operation of the teacher data generating device 10 is started, in step S101, radar signal data is fetched from the radar signal storage unit 101 and provided as an input to the image generating unit 104. At the same time, in step S102, the velocity generating unit 102 generates N velocities [v1, v2, ..., v N ] is generated and provided as an input to the image generation unit 104. Simultaneously with steps S101 and S102, in step S103, antenna information is fetched from the antenna information storage unit 103. In the next step S104, N augmented images are generated by the image generation unit 104. This indicates the end of one process flow cycle of the teacher data generation device 10 in this configuration. Here, the output of one process flow cycle is N augmented images [I v1 , I v2 , ..., I vN ]. N augmented images [I v1 , I v2 , ..., I vN ] is associated with the ground truth data.

[0049] As described above, the teacher data generating device 10 according to the first embodiment enhances the radar image of the training phase by adding realistic motion blur. The blur is parameterized, for example, by a speed sampled from a predefined speed value distribution. The speed value distribution is designed to capture the (walking) speeds encountered during the testing phase of the detection model. This results in a reduction of the speed (and blur) mismatch between the training and testing phases of the detection model, and therefore leads to improved detection performance.

[0050] Second embodiment 9 is a block diagram illustrating a function-based configuration of the teacher data generation device 10 according to the second embodiment. The teacher data generation device 10 of the second embodiment is the same as the teacher data generation device 10 of the first embodiment, except for the points described below.

[0051] The teacher data generating device 10 of the second embodiment further includes a foreground identifying unit 206 and a replacing unit 207. The foreground identifying unit 206 identifies a foreground region to which motion blur should be added in the radar image. The replacing unit 207 replaces a region, excluding the foreground region, in the radar image to which motion blur has been added with a region, excluding the foreground region, in the radar image to which motion blur has not been added, which is generated by using radar signal data. The output unit 109 outputs the replaced radar image in association with the ground truth data.

[0052] Foreground identification unit 206 may identify the foreground region by using, for example, a trained model that uses a radar image as input and outputs information indicating the foreground region, or a camera image captured simultaneously with the radar signal data. Details will be described below. Note that foreground identification unit 206 may identify the foreground region by other methods.

[0053] Next, the configuration of the teacher data generating device 10 according to the second embodiment will be described with reference to the block diagram shown in FIG. 9. In this configuration, there is an additional control of the data augmentation system by allowing the selection of where to add blur in the image. This implies that motion blur is added only to a part of the image and not to the whole image as described in the first embodiment. This can be useful to control the amount of noise in the image due to blur addition, and therefore to ensure the robustness of the detection model. Henceforth, the selected part of the image for blur addition will be referred to as the foreground, and the remaining part of the image excluding the foreground will be referred to as the background. One example of the foreground can be a dangerous object (detection target) to be detected. To implement this, there are two additional units added in this configuration. One of them locates the foreground, and the other unit replaces the background in the blur-added image with a version without blur addition.

[0054] As in the first embodiment, the rate at which the motion blur is parameterized is determined based on some a priori knowledge, i.e., no information from the radar signal is used directly, and the parameter N (>1), which denotes the number of augmented images generated per radar signal, is predetermined.

[0055] The radar image teacher data generating device 10 according to the second embodiment may include a radar signal storage unit 101, an antenna information storage unit 103, an image generating unit 104, a speed generating unit 102, a radar signal acquiring unit 108, an output unit 109, a foreground identifying unit 206, and a replacing unit 207. Note that the configurations of the radar signal storage unit 101, the antenna information storage unit 103, the speed generating unit 102, the radar signal acquiring unit 108, and the output unit 109 are the same as those described in the first embodiment, and therefore the description will not be repeated.

[0056] The inputs of the image generator 104 are the same as those described in the first embodiment, and therefore the description will not be repeated. The function of the image generator 104 is also very similar to that described in the first embodiment, and the image generator 104 generates N extended radar images [I v1 , I v2 , ..., I vN In addition, the image generator 104 also outputs the original unblurred radar image I0, which is provided as input to the foreground identifier 206.

[0057] The foreground identification unit 206 receives as input the original unblurred radar image I0 from the image generation unit 104 and outputs I0 with the foreground location in the image in the form of a rectangular bounding box, i.e., [x, y, w, h]. In this context, the foreground is the part of the image to which blurring is required to be added, where (x, y) is the center location of the box in Cartesian coordinates, w is the width, and h is the height of the box. The localization can be done by means of some pre-trained object detector (learned model) that locates the foreground, or by using information from the optical camera image or some other means. The optical camera image can be taken by a camera such as the camera 98 of FIG. 1.

[0058] The replacement unit 207 replaces the N expanded radar images [I v1 , I v2 , ..., I vN ] as input from the image generator 104. It also receives an additional input, namely the original unblurred image I0 from the foreground identification unit 206 together with the foreground location [x, y, w, h] in the image. The replacement unit 207 generates N augmented radar images [I' v1 , I' v2 , ..., I' vN As mentioned, the background is the image obtained after subtracting the foreground from the overall image. The subtraction can be done by well-known image processing techniques such as cropping selected pixels.

[0059] Next, an example of an operation performed by the teacher data generation device 10 according to the second embodiment of the operation mode will be described with reference to the flowchart shown in Fig. 10. The radar signal has already been measured and stored in the radar signal storage unit 101. The antenna information has been stored in the antenna information storage unit 103. Furthermore, the parameter N has been determined in advance. Note that steps S201, S202, and S203 in Fig. 10 are the same as steps S101, S102, and S103 in Fig. 8, respectively, and therefore their description will not be repeated.

[0060] In step S204, similar to step S104 in the first embodiment, N expanded images [I v1 , I v2 , ..., I vN ] are generated by the image generation unit 104. However, in this configuration, these images are provided as inputs to the substitution unit 207. Furthermore, in this step S204, the image generation unit 104 generates another output I0, which is the original image without blurring added, and provides it as input to the foreground identification unit 206. In step S206, the foreground identification unit 206 locates the part of the image to which blurring should be added, i.e., the foreground. The position [x, y, w, h] of the foreground is provided as output to the substitution unit 207 together with the image without blurring added I0. In step S207, the substitution unit 207 generates the N blurring-added extended images [I v1 , I v2 , ..., I vN ] is replaced with the background from the image I0 without blurring, and N modified augmented images are output. This indicates the end of one process flow cycle of the data augmentation system in this configuration. Here, the output of one process flow cycle is the N augmented images [I' v1 , I' v2 , ..., I' vN ], where blur is added only to part of the image, not the whole image.

[0061] As described above, the teacher data generating device 10 according to the second embodiment augments the radar image of the training phase by adding realistic motion blur only to selected parts of the image. This may be useful when adding blur to the entire image may increase noise, and therefore blur is added only to selected parts. The selected parts may be hazard objects to be detected. As in the first embodiment, this configuration results in a reduction of the (walking) speed and therefore the blur mismatch during the training and testing phases of the detection model. In addition to this, the configuration provides additional control of data augmentation by allowing the selection of where to add blur. This leads to improved detection performance while ensuring the robustness of the detection model, i.e., reduced false alarms.

[0062] Third embodiment 11 is a block diagram illustrating a function-based configuration of the teacher data generation device 10 according to the third embodiment. The teacher data generation device 10 of the third embodiment is the same as the teacher data generation device 10 of the first embodiment, except for the points described below.

[0063] The teacher data generation device 10 of the third embodiment further includes a speed estimation unit 305 that estimates a speed value of an object (e.g., the target 90 in FIG. 1) detected in the radar signal data. The speed generation unit 102 generates a speed value distribution by using the estimated speed value.

[0064] In the teacher data generating device 10 of the third embodiment, the speed estimating unit 305 generates a corrected radar image to which motion correction is applied based on each of a plurality of speed values, for example. The corrected radar image is generated by using radar signal data. In addition, the speed estimating unit 305 selects a speed value from the plurality of speed values ​​by comparing the generated plurality of corrected radar images with each other, and sets the selected speed value as an estimated speed value. Details will be described below.

[0065] Next, the configuration of the teacher data generating device 10 according to the third embodiment will be described with reference to the block diagram shown in FIG. 11. This configuration of the data augmentation system describes additional functions with respect to the configuration described in the first embodiment. In this configuration, instead of determining the velocity based on some predetermined velocity value distribution, the velocity is determined during the processing flow based on the radar signal. This is done by estimating the velocity value of the target (e.g., target 90 in FIG. 1) from the radar signal and generating the velocity based thereon. One exemplary method is to generate the velocity value distribution using the estimated velocity value as a parameter. It can be understood that this velocity value distribution is not constant, but is adjusted for each radar signal based on its estimated velocity value. It is expected that the generation of the motion blur velocity based on the estimated velocity value from the radar signal will make the added motion blur more realistic. Furthermore, in cases such as the exemplary scenario, the adjustment of the velocity value distribution for each radar signal handles the case of outliers (where the target velocity value is too large or too small). Again in this configuration, the parameter N (>1) means the number of augmented images generated for each radar signal and is predetermined.

[0066] The teacher data generation device 10 according to the third embodiment may include a radar signal storage unit 101, an antenna information storage unit 103, an image generation unit 104, a speed generation unit 102, a radar signal acquisition unit 108, an output unit 109, and a speed estimation unit 305. Note that the configurations of the radar signal storage unit 101, the antenna information storage unit 103, the image generation unit 104, the radar signal acquisition unit 108, and the output unit 109 are the same as those described in the first embodiment of the present disclosure, and therefore the description will not be repeated.

[0067] The velocity estimation unit 305 receives the radar signal as an input and estimates the target's estimated velocity value v eto the velocity generator 102. The velocity estimator 305 estimates the velocity value of the target using existing motion estimation and compensation techniques. The technique estimates the most appropriate velocity value among a given number of candidate velocities by comparing the motion compensated images using each of these velocities. The motion compensation technique is parameterized by the velocity values, and the resulting image has clear and strong intensity when the velocity values ​​used are the same or similar to the actual ones. Therefore, the most appropriate velocity value may be selected based on a comparison of the resulting image intensities. The most appropriate velocity value is the estimated velocity value v e It is.

[0068] The velocity generating unit 102 generates the estimated velocity value v e as input and compute N velocities [v1, v2, ..., v N ] as output to the image generator 104. The velocity generator 102 generates N velocities based on the input estimated velocity values. Thus, information from the radar signal is used directly to identify the velocities on which motion blur is generated for the augmented image.

[0069] One exemplary configuration of the velocity generator 102 will be described with reference to FIG. 12. Here, the velocity generator 102 is actually composed of two sub-units, namely a velocity identifier 102a and a distribution generator 102c. Here, the distribution generator 102c receives the estimated velocity value as an input and provides a velocity value distribution to the velocity identifier 102a as an output. The distribution generator 102c is a distribution of the estimated velocity values ​​v e Generate a distribution using v as parameters. One example could be a Gaussian distribution, where the mean is v e and the standard deviation is v e Factors of, for example, v e / 10. The velocity identification unit 102a receives the velocity value distribution as an input and outputs N velocities to the image generation unit 104. The velocity identification unit 102a randomly samples these N velocities from the input velocity value distribution. The predetermined parameter N is known to the velocity identification unit 102a.

[0070] Next, an example of an operation performed by the teacher data generation device 10 according to the third embodiment of the operation mode will be described with reference to the flowchart shown in Fig. 13. The radar signal has already been measured and stored in the radar signal storage unit 101. The antenna information has been stored in the antenna information storage unit 103. Furthermore, the parameter N has been determined in advance. Note that steps S301, S303, and S304 in Fig. 13 are the same as steps S101, S103, and S104 in Fig. 8, respectively, and therefore their description will not be repeated.

[0071] In step S305, the velocity estimation unit 305 estimates the velocity value of the target using the radar signal and outputs the estimated velocity value v e as an input to the velocity generator 102. In step S302, the velocity generator 102 uses the estimated velocity values ​​as parameters to generate N velocities. One exemplary operation is e is used as the mean value, and its factors, e.g., v e The first step is to generate a Gaussian distribution using n / 10 as the standard deviation, and then sample N velocities from this distribution. The generation of N augmented images in step S304 based on these N velocities marks the end of one process flow cycle of the data augmentation system in this configuration. Here, the output of one process flow cycle is the N augmented images [I v1 , I v2 , ..., I vN ], and the blur is the estimated velocity value v e are added based on

[0072] As described above, the teacher data generating device 10 according to the third embodiment enhances the radar image of the training phase by adding blur based on the estimated velocity value of the target for each radar signal. As in the first embodiment, the present configuration results in a reduction of the mismatch of (walking) speed, and therefore blur, during the training and testing phases of the detection model. In addition to this, in the present configuration, the generation of the motion blur speed based on the estimated velocity value from the radar signal is expected to make the added motion blur more realistic. Furthermore, the adjustment of the speed value distribution for each radar signal handles the case of outliers (where the target speed value is too large or too small). This leads to a further improvement of the detection performance.

[0073] As described above, the embodiments of the present invention have been described with reference to the attached drawings, but these embodiments are merely examples of the present invention, and various combinations of the above-mentioned embodiments and configurations other than those of the above-mentioned embodiments may also be adopted.

[0074] All or part of the embodiments disclosed above may be described as, but are not limited to, the following supplementary notes. 1-1. a radar signal acquisition unit that acquires radar signal data acquired by a radar; an image generator that uses the radar signal data to generate a motion blurred radar image based on each of the one or more velocity values; an output unit that outputs the radar image in association with ground truth data that indicates at least whether a detection target has been detected by the radar signal data; A teacher data generating device comprising: 1-2. A velocity generator configured to generate one or more velocity values, The teacher data generation device described in 1-1., wherein the image generation unit acquires the one or more velocity values ​​generated by the velocity generation unit and generates the radar image by using each of the acquired one or more velocity values. 1-3. The teacher data generation device described in 1-2., wherein the speed generation unit generates the one or more speed values ​​by randomly sampling the one or more speed values ​​from a speed value distribution. 1-4. A speed estimation unit that estimates a speed value of an object detected in the radar signal data, The teacher data generating device described in 1-3., wherein the speed generation unit generates the speed value distribution by using the estimated speed value. 1-5. The speed estimation unit using the radar signal data to generate a corrected radar image having motion correction applied based on each of a plurality of velocity values; selecting a velocity value from the plurality of velocity values ​​by comparing the generated plurality of corrected radar images to one another; The teacher data generation device described in 1-4., wherein the selected speed value is set as the estimated speed value. 1-6. a foreground identifier that identifies a foreground region in the radar image to which motion blur is to be added; a replacement unit that replaces a region, excluding the foreground region, in the radar image to which motion blur has been added with a region, excluding the foreground region, in a radar image to which motion blur has not been added, generated by using the radar signal data; Further comprising: The teacher data generation device according to any one of 1-1. to 1-5., wherein the output unit outputs the replaced radar image in association with the supervised data. 1-7. The teacher data generation device described in 1-6., wherein the foreground identification unit identifies the foreground area by using a trained model that inputs a radar image and outputs information indicating the foreground area, or a camera image captured simultaneously with the radar signal data.

[0075] 2-1. A teacher data generation method performed by a computer, comprising: acquiring radar signal data acquired by a radar; generating a motion blurred radar image based on each of the one or more velocity values ​​using the radar signal data; outputting the radar image in association with ground truth data indicating at least whether a detected target has been detected by the radar signal data; A method for generating teacher data, comprising: 2-2. generating one or more velocity values; The training data generation method described in 2-1., wherein the radar image with added motion blur is generated by using each of the generated one or more velocity values. 2-3. The teacher data generation method described in 2-2., wherein the one or more velocity values ​​are generated by randomly sampling the one or more velocity values ​​from a velocity value distribution. 2-4. estimating a velocity value of an object detected in the radar signal data; and generating the velocity value distribution by using the estimated velocity values; The teacher data generation method described in 2-3. further includes: 2-5. generating a corrected radar image using the radar signal data, the corrected radar image having motion correction applied based on each of a plurality of velocity values; selecting a velocity value from the plurality of velocity values ​​by comparing the generated plurality of corrected radar images to one another; setting the selected velocity value as the estimated velocity value; The teacher data generation method described in 2-4. further includes: 2-6. identifying a foreground region in the radar image to which motion blur is to be added; replacing the area, excluding the foreground area, in the radar image to which motion blur has been added with the area, excluding the foreground area, in a radar image, generated using the radar signal data, to which motion blur has not been added; Further comprising: The teacher data generating method according to any one of 2-1. to 2-5., wherein the output radar image is the replaced radar image. 2-7. The teacher data generation method described in 2-6., in which the foreground area is identified by using a trained model that inputs a radar image and outputs information indicating the foreground area, or a camera image captured simultaneously with the radar signal data.

[0076] 3-1. A program for causing a computer to execute a teacher data generation method, the teacher data generation method comprising: acquiring radar signal data acquired by a radar; generating a motion blurred radar image based on each of the one or more velocity values ​​using the radar signal data; outputting the radar image in association with ground truth data indicating at least whether a detected target has been detected by the radar signal data; Including, the program. 3-2. The teaching data generating method further includes generating one or more speed values; The program described in 3-1., wherein the radar image with added motion blur is generated by using each of the generated one or more velocity values. 3-3. The program described in 3-2., wherein the one or more velocity values ​​are generated by randomly sampling the one or more velocity values ​​from a velocity value distribution. 3-4. The teacher data generation method includes: estimating a velocity value of an object detected in the radar signal data; and generating the velocity value distribution by using the estimated velocity values; The program according to 3-3., further comprising: 3-5. The teacher data generation method includes: generating a corrected radar image using the radar signal data, the corrected radar image having motion correction applied based on each of a plurality of velocity values; selecting a velocity value from the plurality of velocity values ​​by comparing the generated plurality of corrected radar images to one another; setting the selected velocity value as the estimated velocity value; The program according to 3-4., further comprising: 3-6. The teacher data generation method includes: identifying a foreground region in the radar image to which motion blur is to be added; replacing the area, excluding the foreground area, in the radar image to which motion blur has been added with the area, excluding the foreground area, in a radar image, generated using the radar signal data, to which motion blur has not been added; Further comprising: The program according to any one of 3-1. to 3-5., wherein the output radar image is the replaced radar image. 3-7. The program described in 3-6., wherein the foreground area is identified by using a trained model that inputs a radar image and outputs information indicating the foreground area, or a camera image captured simultaneously with the radar signal data.

Claims

1. a radar signal acquisition unit that acquires radar signal data acquired by a radar; an image generator that uses the radar signal data to generate a motion blurred radar image based on each of the one or more velocity values; an output unit that outputs teacher data including the radar image associated with ground truth data indicating at least whether a hidden detection target is detected by the radar signal data; Equipped with The teacher data is data for training a detection model for detecting the presence of the detection target in an input radar image. Teacher data generation device.

2. A velocity generator configured to generate one or more velocity values, The teacher data generation device according to claim 1 , wherein the image generation unit acquires the one or more velocity values ​​generated by the velocity generation unit and generates the radar image by using each of the acquired one or more velocity values.

3. The teacher data generating device according to claim 2 , wherein the velocity generating unit generates the one or more velocity values ​​by randomly sampling the one or more velocity values ​​from a velocity value distribution.

4. A speed estimation unit that estimates a speed value of an object detected in the radar signal data, The teacher data generating device according to claim 3 , wherein the velocity generating unit generates the velocity value distribution by using the estimated velocity value.

5. The speed estimation unit using the radar signal data to generate a corrected radar image having motion correction applied based on each of a plurality of velocity values; selecting a velocity value from the plurality of velocity values ​​by comparing the generated plurality of corrected radar images to one another; The teacher data generating device according to claim 4 , wherein the selected velocity value is set as the estimated velocity value.

6. a foreground identifier that identifies a foreground region in the radar image to which motion blur is to be added; a replacement unit that replaces a region, excluding the foreground region, in the radar image to which motion blur has been added with a region, excluding the foreground region, in a radar image to which motion blur has not been added, generated by using the radar signal data; Further comprising: The teacher data generating device according to claim 1 , wherein the output unit outputs the replaced radar image in association with the correct answer data.

7. 7. The teacher data generation device according to claim 6, wherein the foreground identification unit identifies the foreground area by using a trained model that has a radar image as input and information indicating the foreground area as output, or a camera image captured simultaneously with the radar signal data.

8. A teacher data generation method performed by a computer, comprising: acquiring radar signal data acquired by a radar; generating a motion blurred radar image based on each of the one or more velocity values ​​using the radar signal data; outputting training data including the radar image associated with ground truth data indicating at least whether a hidden detection target has been detected in the radar signal data; Including, A teacher data generation method, wherein the teacher data is data for training a detection model that detects the presence of the detection target in an input radar image.

9. A program for causing a computer to execute a teacher data generation method, the teacher data generation method comprising: acquiring radar signal data acquired by a radar; generating a motion blurred radar image based on each of the one or more velocity values ​​using the radar signal data; outputting training data including the radar image associated with ground truth data indicating at least whether a hidden detection target has been detected in the radar signal data; Including, A program, wherein the teacher data is data for training a detection model that detects the presence of the detection target in an input radar image.

Images