Classification device, classification system, and classification method
The classification device addresses the challenge of aspect angle-induced inaccuracies in ISAR image classification by calculating feature ranges and adjusting positions, improving the accuracy of target identification through learning models and feature correlations.
Patent Information
- Application Number
- PCT/JP2023/020200
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-05
AI Technical Summary
Conventional ISAR image classification techniques face challenges in accurately determining the true length of targets due to aspect angle variations, leading to potential misclassification.
A classification device that identifies targets from ISAR images, calculates feature ranges, and adjusts characteristic positions to improve accuracy by using learning models and feature specification databases to select the most correlated target candidates.
Enhances target classification accuracy by determining and adjusting characteristic positions based on aspect angle calculations and feature correlations, allowing for precise identification even at varying angles.
Smart Images

Figure JP2023020200_05122024_PF_FP_ABST
Abstract
Description
Classification device, classification system and classification method
[0001] The present disclosure relates to a classification device, a classification system, and a classification method.
[0002] A surveillance system is known that monitors the ocean and coastlines using still or moving ISAR images (hereinafter referred to as ISAR video) generated by an inverse synthetic aperture radar (hereinafter referred to as ISAR video). For example, in a radar device described in Patent Document 1, a display processing unit displays two measurement lines for measuring the overall length (LOA; Length Over All) of a target such as a ship on a display unit displaying an ISAR image in response to a user's instruction. A brightness adjustment unit determines an area of the ISAR image outside the two measurement lines and increases the brightness of this area. The display processing unit updates the positions of the two measurement lines in response to a user's instruction. An LOA measurement unit calculates the distance between the two measurement lines and measures the overall length of the ship. An operator (user) can visually check the ISAR video to identify the type of target based on the LOA of the target in the ISAR video.
[0003] Japanese Patent Application Laid-Open No. 2021-25972
[0004] However, conventional techniques have had a problem in that they may not be able to accurately classify targets based on their characteristics in ISAR images. For example, the overall length, which is a characteristic of a target, varies depending on the aspect angle of the target relative to the direction of radio wave irradiation from the sensor. For this reason, the conventional technique described in Patent Document 1 may not be able to accurately classify targets because it is not possible to estimate the true overall length of a target based on the value of the aspect angle in an ISAR image.
[0005] The present disclosure is intended to solve the above-mentioned problems, and aims to provide a classification device that can improve the accuracy of target classification.
[0006] a feature range calculation unit that calculates a feature range, which is the range of a feature portion of the target, based on aspect angle information of the target included in the radar image, target identification result information by the first target identification unit, and known feature specifications for each target candidate; a first display processing unit that determines a feature position, which is the position of the feature portion of the target, using the feature range calculated by the feature range calculation unit and generates first display control information for overlaying and displaying the determined feature position on the radar image; a feature position editing unit that, upon acquiring editing operation information indicating an editing operation performed on the feature position of the target in the radar image displayed on the display unit based on the first display control information, calculates the feature specifications of the target using the feature position of the target for which the editing operation was performed, which is indicated by the editing operation information; a second target identification unit that identifies known feature specifications for each target candidate that correspond to the feature specifications of the target calculated by the feature position editing unit, and selects a target candidate that corresponds to the identified feature specifications; and a second display processing unit that generates second display control information for displaying the target candidate selected by the second target identification unit.
[0007] According to the present disclosure, the characteristic positions of a target identified from radar images are determined, and editing operation information indicating editing operations performed on the characteristic positions of the target in the radar images displayed on a display unit is acquired. Then, the characteristic parameters of the target are calculated using the characteristic positions where the editing operations were performed, and known characteristic parameters for each target candidate are identified that correspond to the calculated characteristic parameters, and the target candidate corresponding to the identified characteristic parameters is displayed. Even if the characteristic positions of the target identified from the radar images are far from the characteristic positions of the target in the radar images, the edited characteristic positions of the target are used to re-identify the target. This allows the classification device according to the present disclosure to improve the accuracy of target classification.
[0008] FIG. 1 is a block diagram showing an example of the configuration of a categorization system according to embodiment 1. FIG. 2 is a block diagram showing an example of the configuration of a categorization device according to embodiment 1. FIG. 3 is a schematic diagram showing an overview of the relationship between a sensor, a target, a target orientation, a radio wave irradiation direction, and an aspect angle. FIG. 4A and FIG. 4B are schematic diagrams showing an example (1) of a learning model and a learning database. FIG. 4 is a schematic diagram showing an overview of a feature specification database. FIG. 4 is a schematic diagram showing an overview of the relationship between a radar image, feature specifications, a feature range, and a radio wave irradiation direction. FIG. 5 is a schematic diagram showing an overview of a process for superimposing a feature position of a target on a radar image. FIG. 6 is a flowchart showing the operation of a radar device. FIG. 7 is a flowchart showing a categorization method according to embodiment 1. FIG. 8 is a schematic diagram showing an example (2) of a learning model and a learning database. FIG. 11A and FIG. 11B are block diagrams showing a hardware configuration for realizing the functions of a categorization device according to embodiment 1. FIG. 9 is a block diagram showing an example of the configuration of a categorization device according to embodiment 2. FIG. 10 is a flowchart showing a categorization method according to embodiment 2.
[0009] Embodiment 1. Fig. 1 is a block diagram showing an example configuration of a classification system 1 according to embodiment 1. In Fig. 1, the classification system 1 is a system for classifying targets present in external space, and includes a classification device 100 and a radar device 200. Targets to be classified are, for example, moving objects such as ships or aircraft. The classification device 100 classifies targets from radar images generated by the radar device 200. The radar images are, for example, radar still images or moving image information obtained by ISAR or synthetic aperture radar (hereinafter abbreviated as SAR). The radar device 200 is a device that transmits radio waves into external space and generates radar images of targets using echoes of the radio waves received from targets present in external space.
[0010] FIG. 2 is a block diagram showing an example configuration of the classifying device 100 according to the first embodiment. In FIG. 2, the classifying device 100 includes a storage unit 101 in which a feature specification database is stored, a display unit 102, an operation unit 103, and a calculation unit (not shown in FIG. 2). The storage unit 101 is a storage device included in a computer that functions as the classifying device 100, and includes storage such as a hard disk drive (HDD) or a solid state drive (SSD), or a memory 1004 (described later) shown in FIG. 11B. Note that the storage unit 101 may be external to the classifying device 100, as long as it is accessible from the classifying device 100.
[0011] The display unit 102 is a display device included in the computer functioning as the classifying device 100. For example, the display unit 102 is an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) display device. The operation unit 103 is an input device that accepts operations on the screen display of the display unit 102 included in the computer functioning as the classifying device 100. For example, the classifying device 100 is a touch panel that is integrated with the display screen of the display unit 102. The operation unit 103 may also be, for example, a mouse or a keyboard.
[0012] The calculation unit controls the overall operation of the classifying device 100. The calculation unit includes an aspect angle calculation unit 111, a radar image input unit 112, a first target identification unit 113, a feature range calculation unit 114, a first display processing unit 115, a feature position editing unit 116, a second target identification unit 117, and a second display processing unit 118. When the calculation unit executes the classification application, various functions of the aspect angle calculation unit 111, the radar image input unit 112, the first target identification unit 113, a feature range calculation unit 114, the first display processing unit 115, the feature position editing unit 116, the second target identification unit 117, and the second display processing unit 118 are realized.
[0013] The radar device 200 includes a transmitting / receiving unit 201 and a tracking processing unit 202. The transmitting / receiving unit 201 transmits and receives radio waves via a radar antenna. The tracking processing unit 202 performs processing to track a target moving in external space.
[0014] FIG. 3 is a schematic diagram showing an overview of the relationship between the sensor 13, the target Ta, the direction D1 of the target Ta, the radio wave irradiation direction D2, and the aspect angle θ. In FIG. 3, the sensor 13 is a radar antenna that emits radio waves into external space and receives echoes of the target Ta from the external space. The sensor 13, which is a radar antenna, is included in the transceiver unit 201. The target Ta is, for example, a ship shown by a dashed line in FIG. 3. The arrow through which the bow of the target Ta passes indicates the direction D1 of the target Ta. The radio wave irradiation direction D2 is the direction of the radio waves radiated from the sensor 13. The aspect angle θ is the angle between the radio wave irradiation direction D2 (the direction of the sensor 13) and the direction D1 of the target Ta.
[0015] The transmitter / receiver 201 performs signal processing to generate a radar image of the target based on the echo of the radio wave from the target received by the sensor 13. In the following description, it is assumed that the radar image is an ISAR image. In addition, the radar device 200 constantly acquires real-time information on the radio wave irradiation direction D2 in order to control the radio wave irradiation direction D2 shown in FIG. 3 .
[0016] The aspect angle calculation unit 111 acquires information indicating the direction D1 of the target Ta from the tracking processing unit 202, acquires information indicating the radio wave irradiation direction D2 acquired from the transmission / reception unit 201, and calculates the aspect angle θ using the direction D1 of the target Ta and the radio wave irradiation direction D2. For example, when the target Ta is disposed as shown in Fig. 3, the aspect angle calculation unit 111 calculates the aspect angle θ according to the following formula (1). In the following formula (1), the unit of angle is "degrees". Aspect angle θ = 180 - radio wave irradiation direction D2 + direction D1 of the target Ta (1)
[0017] The radar image input unit 112 acquires a radar image from the transmitting / receiving unit 201 included in the radar device 200. The radar image acquired by the radar image input unit 112 is output to the first target identification unit 113.
[0018] The first target identification unit 113 identifies the target Ta from the radar image. For example, in the radar image, the brightness value on the image is determined according to the intensity of the radio waves received as reflected waves. The first target identification unit 113 binarizes the image of the radar image and identifies the target by performing pattern recognition on the binarized image. Note that the function of the aspect angle calculation unit 111 may be provided in the feature range calculation unit 114, and the function of the radar image input unit 112 may be provided in the first target identification unit 113. In this case, the classifying device 100 does not need to include the aspect angle calculation unit 111 and the radar image input unit 112.
[0019] Furthermore, the first target identification unit 113 may identify a target using a first learning model that outputs target identification result information when a radar image is input. FIG. 4A is a diagram showing an overview of the learning database 14 for generating the first learning model 15. FIG. 4B is a diagram showing an overview of the first learning model 15. As shown in FIG. 4A , the learning data registered in the learning database 14 includes a radar image E and annotation information F corresponding to the radar image E, and is stored, for example, in the storage unit 101. The annotation information F includes candidates for target X included in the radar image X1. The first learning model 15 is generated by learning using the learning data registered in the learning database 14. Note that the radar image E and the corresponding annotation information F included in the learning data are not limited to those generated by the transceiver unit 201.
[0020] 4B , the first target identification unit 113 inputs the radar image E to the first learning model 15. As a result, the first learning model 15 infers the name or type of the target Ta in the radar image E and outputs the inferred name or type of the target Ta as identification result information. The first target identification unit 113 uses the first learning model 15, which enables the classification device 100 to accurately classify the target Ta using the radar image E.
[0021] The feature range calculation unit 114 calculates a feature range, which is the range of the feature part of the target Ta, based on the aspect angle information of the target Ta included in the radar image E, the identification result information of the target Ta by the first target identification unit 113, and the known feature specifications for each target candidate. For example, the feature range calculation unit 114 inputs the aspect angle θ and the identification result information of the target Ta, and calculates the feature range using the input information.
[0022] 5 is a schematic diagram showing an overview of the feature specification database, and shows the contents of the feature specification database stored in the storage unit 101. In the feature specification database, known feature specifications H are registered in association with annotation information G indicating the names or types of various target candidates. The feature specifications are parameters that represent the characteristic parts of a target.
[0023] For example, if the target is a ship, the characteristic parameters include the overall length (LOA) of the ship, which is determined for each type of ship, the length between the bow and the mast, the length between the bow and the container, etc. A single target candidate (e.g., target X) may be associated with multiple types of characteristic parameters (e.g., characteristic parameter X2, characteristic parameter X3, ...).
[0024] The feature range calculation unit 114 refers to the feature specification database and reads feature specifications corresponding to the target information indicated by the identification result information by the first target identification unit 113. For example, if the target information indicated by the identification result information is target X, the feature range calculation unit 114 reads feature specifications consisting of multiple types of feature specifications X2, feature specifications X3, ... corresponding to target X from the feature specification database.
[0025] The characteristic range calculation unit 114 converts the characteristic parameters read from the characteristic parameter database into a characteristic range. The characteristic range is information indicating the range of a characteristic part of a target, and for example, if the target is a ship, it is the slant range length of the target in the radar image.
[0026] 6 is a schematic diagram showing an outline of the relationship between the radar image E, the characteristic parameter H, the characteristic range I, and the radio wave irradiation direction D2. For example, if the target Ta shown in the radar image E is a ship, and the characteristic parameter read from the characteristic parameter database is the length between the bow and the mast, the length between the bow and the mast is expressed as the characteristic range I, which is the slant range length in the radar image E.
[0027] Since the slant range axis is parallel to the radio wave irradiation direction D2, as shown in Fig. 6, there is a proportional relationship depending on the aspect angle θ between the characteristic parameter H, which is the length between the bow and the mast, and the characteristic range I. When the aspect angle θ is defined as shown in Fig. 3, the characteristic range calculation unit 114 can calculate the characteristic range I using the following formula (2). The characteristic range calculation unit 114 outputs the radar image E and the calculated characteristic range I to the first display processing unit 115. Characteristic range I (m) = characteristic parameter H (m) × cos (aspect angle θ) (2)
[0028] The first display processing unit 115 determines the characteristic position PA of the target Ta using the characteristic range I calculated by the characteristic range calculation unit 114, and generates first display control information for displaying the determined characteristic position PA superimposed on the radar image E.
[0029] 6, the first display processing unit 115 converts the radar image E into an image in which one of the coordinate axes in the radar image E is a slant range axis parallel to the radio wave irradiation direction D2. The first display processing unit 115 uses the characteristic range I to generate first display control information for displaying a characteristic position PA of the target Ta superimposed on the radar image E. The characteristic position is the position of a characteristic part of the target Ta.
[0030] 7 is a schematic diagram showing an outline of the process of superimposing the characteristic position PA of the target Ta on the radar image E. In FIG. 7, the target Ta is a ship. The characteristic positions PA of the target Ta in the radar image E are a characteristic position J1 related to the bow, a characteristic position J2 related to the stern, and a characteristic position J3 related to the mast. The first display processing unit 115 calculates the center of gravity position G1 of the radar image E using pixel value information or complex signal information of the radar image E.
[0031] Of the characteristic positions PA of the target Ta, the characteristic position J1 related to the bow is a position shifted in the negative direction of the slant range axis by half the value of the characteristic range I1 from the center of gravity position G1 in the characteristic range I1 calculated by the characteristic range calculation unit 114 based on the LOA of the target Ta. Also, the characteristic position J2 related to the stern is a position shifted in the positive direction of the slant range axis by half the value of the characteristic range I1 from the center of gravity position G1 in the characteristic range I1. When the characteristic range I calculated according to the above equation (2) is a negative value, the positive and negative directions of the slant range axis are reversed.
[0032] Next, the first display processing unit 115 calculates, as a characteristic position J3 for the mast, a position shifted from the characteristic position J1 in the positive direction of the slant range axis by the length of the characteristic range I2, which is calculated by the characteristic range calculation unit 114 based on the length between the bow and the mast. Note that if the characteristic range I2 calculated according to the above equation (2) is a negative value, the characteristic position J3 is a position shifted from the characteristic position J1 in the negative direction of the slant range axis by the length of the characteristic range I2.
[0033] The first display processing unit 115 converts characteristic specifications such as the length between the bow and the container into a characteristic range I, and sets a position shifted from the bow in the characteristic range I as a characteristic position PA. For example, as shown in FIG. 7 , the first display processing unit 115 calculates a plurality of characteristic positions including characteristic positions J1, J2, and J3 as the characteristic position PA of the target Ta. The first display processing unit 115 generates first display control information for displaying the characteristic position PA of the target Ta superimposed on the radar image E, and outputs the first display control information to the display unit 102. The display unit 102 displays a screen such as that shown in FIG. 7 based on the first display control information.
[0034] The operator of the classification device 100 visually checks the radar image E displayed on the display unit 102 and compares the characteristic position PA of the target Ta superimposed on the radar image E with the characteristic position PB of the actual target Ta appearing in the radar image E. If the characteristic position PA and the characteristic position PB do not match, the operator uses the operation unit 103 to perform an editing operation to convert the characteristic position PA into a characteristic position PA1 that matches the characteristic position PB, and outputs editing operation information indicating this editing operation to the feature position editing unit 116. The feature position editing unit 116 generates a feature position PA1 obtained by performing the editing operation indicated by the editing operation information on the feature position PA of the target Ta.
[0035] Next, the characteristic position editing unit 116 uses the edited characteristic position PA1 of the target Ta to calculate the characteristic range R. Take an example where the target Ta is a ship, and the characteristic position PA1 includes a characteristic position J1 related to the bow, a characteristic position J2 related to the stern, and a characteristic position J3 related to the mast. In this case, the characteristic range R is, for example, the length from the characteristic position J1 related to the bow to another characteristic position, the characteristic position J2 related to the stern or the characteristic position J3 related to the mast.
[0036] Next, the feature position editing unit 116 converts the feature range R into a feature parameter CP. The feature parameter CP is a parameter representing the length of the actual target Ta that can be estimated using the feature range R. For example, when the aspect angle θ is defined according to the relationship shown in FIG. 3, the feature position editing unit 116 calculates the feature parameter CP using the following equation (3). In the following equation (3), |cos(X)| is the absolute value of cos(X). Feature parameter CP(m)=feature range R(m)÷|cos(aspect angle θ)| (3)
[0037] The second target identification unit 117 identifies, from the known characteristic parameters for each target candidate stored in the characteristic parameter database, those that correspond to the characteristic parameters CP of the target Ta calculated by the characteristic position editing unit 116, and selects a target candidate that corresponds to the identified characteristic parameters. For example, from the known characteristic parameters for each target candidate stored in the characteristic parameter database, the second target identification unit 117 identifies those that have the highest correlation with the characteristic parameters CP of the target Ta calculated by the characteristic position editing unit 116, and selects a target candidate that corresponds to the identified characteristic parameters.
[0038] Specifically, the second target identification unit 117 calculates a correlation function between the known feature parameter and the feature parameter CP to find a correlation coefficient that indicates the relationship between the two, and selects, from among the known feature parameters, the one with the largest correlation coefficient with the feature parameter CP as the one with the highest correlation. This enables the classification device 100 to select, from the known feature parameters, feature parameters that are closest to the feature parameter CP of the target Ta obtained by editing the feature positions. The target candidates selected by the second target identification unit 117 are output from the second target identification unit 117 to the second display processing unit 118 as the re-identification result of the target Ta.
[0039] The second display processing unit 118 generates second display control information for displaying the target candidates selected by the second target identification unit 117, and outputs the second display control information to the display unit 102. For example, the second display processing unit 118 generates second display control information for displaying the target candidates selected by the second target identification unit 117 together with the radar image E. The display unit 102 displays the target candidates that are the re-identification results of the target Ta based on the second display control information. The operator can confirm the re-identification results of the target Ta from the radar image E by referring to the identification results displayed on the display unit 102.
[0040] If the operator visually checks the radar image E displayed on the display unit 102 based on the first display control information and determines that the difference between the characteristic position PA and the characteristic position PB is within an allowable range, the operator performs an operation indicating that re-identification is not necessary, for example, using the operation unit 103. Operation information indicating this operation is output from the operation unit 103 to the characteristic position editing unit 116. Based on this operation information, the characteristic position editing unit 116 notifies the second target identification unit 117 that re-identification of the target Ta is not necessary.
[0041] When the second display processing unit 118 receives the notification indicating that re-identification is not required, it generates second display control information for displaying the characteristic position PA of the target Ta as is on the radar image E, and outputs the second display control information to the display unit 102. Based on the second display control information, the display unit 102 displays the target Ta, which is the identification result, together with the radar image E. The operator can confirm the identification result of the target Ta from the radar image E by referring to the identification result displayed on the display unit 102.
[0042] Furthermore, although the above description has been given of a case in which an operator who visually checks the radar image E displayed on the display unit 102 based on the first display control information performs an editing operation of the feature positions using the operation unit 103, the editing operation of the feature positions may be performed automatically. For example, the feature position editing unit 116 may compare the difference between the feature position PA and the feature position PB with an allowable threshold, and if the difference exceeds the allowable threshold, determine that the feature position PA is far from the feature position PB, and automatically convert (edit) the feature position PA to the feature position PA1.
[0043] In this case, the first display processing unit 115 may generate display control information for displaying on the display unit 102 the feature position editing operation automatically performed by the feature position editing unit 116, and output the generated display control information to the display unit 102. In this case, the display unit 102 displays the process of editing the feature position PA to the feature position PA1 based on this display control information. By referring to the editing operation displayed on the display unit 102, the operator can visually confirm the process of automatically editing the feature position PA to the feature position PA1.
[0044] Next, an operation of the radar device 200 will be described. Fig. 8 is a flowchart showing the operation of the radar device 200. The tracking processing unit 202 tracks a target Ta present in external space, thereby calculating a direction D1 of the target Ta based on the traveling direction of the target Ta, and outputs the calculated direction D1 to the aspect angle calculation unit 111 (step ST1-1).
[0045] The transmitter / receiver 201 generates beam pointing information indicating the beam pointing of the radar signal transmitted to the external space, and outputs the beam pointing information to the aspect angle calculation unit 111 (step ST1-2). The beam pointing information is information regarding the beam pointing at the time of transmission, and indicates the radio wave irradiation direction D2 of the transmitted radio wave.
[0046] Furthermore, the transmitter / receiver 201 generates a radar image E by processing the radar signal received from the external space, and outputs the radar image E to the radar image input unit 112 (step ST1-3). Note that the order of processing steps ST1-1, ST1-2, and ST1-3 does not matter, and they may be performed simultaneously.
[0047] 9 is a flowchart showing the classification method according to the first embodiment. The aspect angle calculation unit 111 calculates the aspect angle θ based on information indicating the direction D1 of the target Ta acquired from the radar device 200 (step ST1A-1). For example, the aspect angle calculation unit 111 calculates the aspect angle θ according to the above formula (1). The radar image input unit 112 inputs the radar image E from the radar device 200 (step ST1A-2). Note that the processing of steps ST1A-1 and ST1A-2 may be performed simultaneously, regardless of the order. Furthermore, these processing steps are pre-processing steps for the classification method according to the first embodiment, and it is assumed that these processing steps have already been performed in the classification method according to the first embodiment.
[0048] The first target identification unit 113 identifies the target Ta from the radar image E (step ST2A). Next, the feature range calculation unit 114 calculates a feature range I, which is the range of the feature portion of the target Ta, based on the aspect angle information of the target Ta included in the radar image E, the identification result information of the target Ta by the first target identification unit 113, and the known feature specifications for each target candidate (step ST3A). The first display processing unit 115 determines a feature position PA of the target Ta using the feature range I calculated by the feature range calculation unit 114, and generates first display control information for displaying the determined feature position PA superimposed on the radar image E (step ST4A). The display unit 102 displays the radar image E together with the feature position PA of the target Ta based on the first display control information.
[0049] When the feature position editing unit 116 acquires editing operation information indicating an editing operation performed on the feature position PA of the target in the radar image E displayed on the display unit 102 based on the first display control information, it calculates the feature parameters CP of the target Ta using the feature position PA1 of the target for which the editing operation indicated in the editing operation information was performed (step ST5A). The second target identification unit 117 identifies, from among the known feature parameters for each target candidate, those that correspond to the feature parameters CP of the target calculated by the feature position editing unit 116, and selects a target candidate corresponding to the identified feature parameters (step ST6A). The second display processing unit 118 generates second display control information for displaying the target candidate selected by the second target identification unit 117 (step ST7A). The display unit 102 displays the radar image E together with the re-identified target candidate based on the second display control information.
[0050] In step ST4A, the operator compares the characteristic position PA with the characteristic position PB in the radar image E. If the operator determines that the correspondence between the characteristic position PA and the characteristic position PB is valid, the operator can be convinced that the identification result obtained in step ST3A is not incorrect. On the other hand, if the characteristic position PA and the characteristic position PB are far apart, the operator can determine that the identification result obtained in step ST3A may be incorrect. In this case, by editing the characteristic position PA and re-identifying the target using the edited characteristic position PA1, the operator can be convinced of the re-identification result. In addition to the satisfaction, the classification accuracy of the classification device 100 actually improves as the identification information increases with the information input by the operator.
[0051] In addition, when the radar image E and the characteristic parameters CP of the target Ta calculated by the characteristic position editing unit 116 are input, the second target identification unit 117 may select a target candidate from among target candidates corresponding to the known characteristic parameters using a second learning model that outputs target candidates corresponding to the characteristic parameters CP of the target Ta.
[0052] FIG. 10A is a diagram showing an overview of the learning data for generating the second learning model 15A. FIG. 10B is a diagram showing an overview of the second learning model 15A. In FIG. 10A, the learning data includes a radar image E, annotation information F corresponding to the radar image E, and feature specifications H corresponding to the annotation information F. The annotation information F includes candidates for target X included in the radar image X1. The second learning model 15A is generated by learning using learning data registered in the learning database 14A. Note that the radar image E, the corresponding annotation information F, and the feature specifications H corresponding to the annotation information F included in the learning data are not limited to those generated by the transceiver unit 201.
[0053] 10B, the second target identification unit 117 inputs the radar image E and the feature specifications H1 to H4 to the second learning model 15A. As a result, the second learning model 15A infers target candidates corresponding to the radar image E and the feature specifications H1 to H4 and outputs the inferred target candidates as identification result information. By using the second learning model 15A in this way, the second target identification unit 117 can accurately select target candidates corresponding to the radar image E and the feature specifications.
[0054] Furthermore, the second learning model 15A may be a machine learning model that outputs target candidate identification information including the name and type of the target candidate as annotation information when the radar image E and the characteristic position of the target Ta determined based on the annotation information F of the target Ta are input. When the second target identification unit 117 uses this second learning model 15A, the classifying device 100 can accurately select target candidates including the name and type of the target candidate as annotation information.
[0055] Next, a hardware configuration for realizing the functions of the classifying device 100 will be described. The functions of the aspect angle calculation unit 111, radar image input unit 112, first target identification unit 113, feature range calculation unit 114, first display processing unit 115, feature position editing unit 116, second target identification unit 117, and second display processing unit 118 provided in the classifying device 100 are realized by processing circuits. That is, the classifying device 100 includes a processing circuit for executing the processes from step ST1A-1 to step ST7A shown in FIG. 9. The processing circuit may be dedicated hardware, or may be a CPU (Central Processing Unit) that executes a program stored in memory.
[0056] Fig. 11A is a block diagram showing a hardware configuration for realizing the functions of the classifying device 100. Fig. 11B is a block diagram showing a hardware configuration for executing software for realizing the functions of the classifying device 100. In Figs. 11A and 11B, the input interface 1000 is an interface that relays, for example, aspect angle information and radar image E output from the radar device 200 to the classifying device 100, and editing operation information output from the operation unit 103 to the classifying device 100. The output interface 1001 is an interface that relays, for example, identification result information output from the classifying device 100 to the display unit 102.
[0057] 11A , the processing circuit 1002 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. The functions of the aspect angle calculation unit 111, the radar image input unit 112, the first target identification unit 113, the feature range calculation unit 114, the first display processing unit 115, the feature position editing unit 116, the second target identification unit 117, and the second display processing unit 118 provided in the classification device 100 may be realized by separate processing circuits, or these functions may be realized together by a single processing circuit.
[0058] 11B , the functions of the aspect angle calculation unit 111, radar image input unit 112, first target identification unit 113, feature range calculation unit 114, first display processing unit 115, feature position editing unit 116, second target identification unit 117, and second display processing unit 118 provided in the classification device 100 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 1004. The memory 1004 is, for example, the storage unit 101 shown in FIG. 2 .
[0059] The processor 1003 reads and executes the classification application program stored in the memory 1004, thereby realizing the functions of the aspect angle calculation unit 111, the radar image input unit 112, the first target identification unit 113, the feature range calculation unit 114, the first display processing unit 115, the feature position editing unit 116, the second target identification unit 117, and the second display processing unit 118 included in the classification device 100. For example, the classification device 100 includes a memory 1004 for storing the above-mentioned program, which, when executed by the processor 1003, results in the execution of the processes of steps ST1A-1 to ST7A shown in FIG. 9. This program causes the computer to execute the procedures or methods of the processes performed by the aspect angle calculation unit 111, the radar image input unit 112, the first target identification unit 113, the feature range calculation unit 114, the first display processing unit 115, the feature position editing unit 116, the second target identification unit 117, and the second display processing unit 118. The memory 1004 may be a computer-readable storage medium storing a program for causing the computer to function as an aspect angle calculation unit 111, a radar image input unit 112, a first target identification unit 113, a feature range calculation unit 114, a first display processing unit 115, a feature position editing unit 116, a second target identification unit 117, and a second display processing unit 118.
[0060] The memory 1004 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically-EPROM), a magnetic disk, a flexible disk, an optical disk, a compact disk, a minidisk, or a DVD.
[0061] Some of the functions of the aspect angle calculation unit 111, radar image input unit 112, first target identification unit 113, feature range calculation unit 114, first display processing unit 115, feature position editing unit 116, second target identification unit 117, and second display processing unit 118 included in the classification device 100 may be realized by dedicated hardware, and the remaining functions may be realized by software or firmware. For example, the functions of the aspect angle calculation unit 111 and the radar image input unit 112 are realized by a processing circuit 1002, which is dedicated hardware, and the functions of the first target identification unit 113, feature range calculation unit 114, first display processing unit 115, feature position editing unit 116, second target identification unit 117, and second display processing unit 118 are realized by a processor 1003 reading and executing a program stored in a memory 1004. In this way, the processing circuit can realize the above functions by hardware, software, firmware, or a combination of these.
[0062] As described above, the classification device 100 according to the first embodiment includes a first target identification unit 113 that identifies the target Ta from the radar image E, a feature range calculation unit 114 that calculates the feature range I of the target Ta based on aspect angle information of the target Ta included in the radar image E, identification result information of the target Ta by the first target identification unit 113, and known feature specifications for each target candidate, a first display processing unit 115 that determines the feature position PA of the target using the feature range I and generates first display control information for displaying the feature position PA superimposed on the radar image E, and a display control unit 116 that controls the first display control information. The classification device 100 includes a feature position editing unit 116 that, upon acquiring editing operation information indicating an editing operation on a feature position PA of the target Ta in the radar image E displayed on the display unit 102 based on the radar image E, calculates feature parameters CP of the target Ta using the feature position PA1 of the target Ta that has been edited, a second target identification unit 117 that identifies feature parameters corresponding to the feature parameters CP from known feature parameters for each target candidate and selects a target candidate corresponding to the identified feature parameters, and a second display processing unit 118 that generates second display control information for displaying the target candidate selected by the second target identification unit 117. Even if the feature position PA of the target Ta identified from the radar image E is far from the feature position PB of the target Ta, the target Ta is re-identified using the edited feature position PA1 of the target Ta, and the classification device 100 can improve the classification accuracy of the target Ta.
[0063] In the classification device 100 according to the first embodiment, when a radar image E is input, the first target identification unit 113 identifies the target Ta using the first learning model 15, which outputs identification result information for the target Ta. The first target identification unit 113 uses the first learning model 15, which enables the classification device 100 to accurately classify the target Ta using the radar image E.
[0064] In the classifying device 100 according to the first embodiment, the second target identification unit 117 identifies, from the known characteristic parameters of each target candidate, those that have the highest correlation with the characteristic parameter CP of the target Ta calculated by the characteristic position editing unit 116, and selects the target candidate corresponding to the identified characteristic parameter. This enables the classifying device 100 to select, from the known characteristic parameters, characteristic parameters that are close to the characteristic parameter CP calculated based on the edited characteristic position PA1.
[0065] In the classifying device 100 according to the first embodiment, when the radar image E and the feature parameters CP of the target Ta calculated by the feature position editing unit 116 are input, the second target identification unit 117 selects a target candidate using the second learning model 15A that outputs a target candidate corresponding to the feature parameters CP of the target Ta. By using the second learning model 15A by the second target identification unit 117, the classifying device 100 can accurately select a target candidate corresponding to the feature parameters CP from target candidates corresponding to known feature parameters.
[0066] In the classification device 100 according to the first embodiment, the second learning model 15A outputs target candidate identification information including the name and type of the target candidate as annotation information when the radar image E and the characteristic position of the target Ta determined based on the annotation information F of the target Ta are input. The second target identification unit 117 uses this second learning model 15A, so that the classification device 100 can accurately select target candidates including the name and type of the target candidate as annotation information.
[0067] The classification system 1 according to the first embodiment includes a radar device 200 that generates and outputs a radar image E including the searched target Ta, a display unit 102 that displays the radar image E based on first display control information and second display control information, an operation unit 103 that accepts editing operations for a characteristic position PA of the target Ta in the radar image E displayed on the display unit 102, and a classifying device 100 that inputs the radar image E generated by the radar device 200 and the editing operation information. This makes it possible to provide a classification system 1 that has improved target classification accuracy.
[0068] The classification method according to the first embodiment includes step ST2A in which the first target identification unit 113 identifies the target Ta from the radar image E, step ST3A in which the feature range calculation unit 114 calculates a feature range I of the target Ta based on aspect angle information of the target Ta included in the radar image E, identification result information of the target Ta by the first target identification unit 113, and known feature specifications for each target candidate, step ST4A in which the first display processing unit 115 determines a feature position PA of the target Ta using the feature range I and generates first display control information for displaying the determined feature position PA superimposed on the radar image E, and step ST4A in which the feature position editing unit 116 generates a feature range I of the target Ta based on the first display control information. The method includes step ST5A in which, upon acquiring editing operation information indicating an editing operation performed on a feature position PA of the target in the radar image E displayed on the display unit 102 by the second target identification unit 117, the second target identification unit 117 calculates feature parameters CP of the target Ta using the feature position PA1 of the target Ta for which the editing operation indicated by the editing operation information has been performed, step ST6A in which the second target identification unit 117 identifies, from known feature parameters for each target candidate, those that correspond to the feature parameters CP of the target Ta calculated by the feature position editing unit 116 and selects a target candidate that corresponds to the feature parameters CP, and step ST7A in which the second display processing unit 118 generates second display control information for displaying the target candidate selected by the second target identification unit 117. By the classification device 100 performing this method, the accuracy of target classification can be improved.
[0069] Embodiment 2. Figure 12 is a block diagram showing an example of the configuration of a classification device 100A according to embodiment 2. In Figure 12, the same components as those in Figure 2 are assigned the same reference numerals, and redundant explanations will be omitted. The classification device 100A is provided in place of the classification device 100 included in the classification system 1 shown in Figure 1. Similar to the classification device 100, targets to be classified are moving objects such as ships or aircraft. The classification device 100A classifies targets from radar images generated by a radar device 200. The radar images are, for example, radar still images or moving image information obtained by ISAR or SAR.
[0070] The classifying device 100A includes a storage unit 101 in which a feature specification database is stored, a display unit 102, an operation unit 103, and a calculation unit not shown in Fig. 12. The storage unit 101 is a storage device included in a computer that functions as the classifying device 100A, and includes storage such as an HDD or SSD, or the memory 1004 shown in Fig. 11B. The storage unit 101 may be provided external to the classifying device 100A as long as it is accessible from the classifying device 100A.
[0071] The calculation unit controls the overall operation of the classification device 100A. The calculation unit includes an aspect angle calculation unit 111, a radar image input unit 112, a first target identification unit 113, a feature range calculation unit 114, a first display processing unit 115, a feature position editing unit 116, a second target identification unit 117A, a second display processing unit 118, an annotation processing unit 119, and a learning model update unit 120. When the calculation unit executes the classification application, the functions of the aspect angle calculation unit 111, the radar image input unit 112, the first target identification unit 113, a feature range calculation unit 114, the first display processing unit 115, the feature position editing unit 116, the second target identification unit 117A, the second display processing unit 118, the annotation processing unit 119, and the learning model update unit 120 are realized.
[0072] The function of the aspect angle calculation unit 111 may be provided by the feature range calculation unit 114, and the function of the radar image input unit 112 may be provided by the first target identification unit 113. In this case, the classification device 100A does not need to include the aspect angle calculation unit 111 and the radar image input unit 112.
[0073] The first target identification unit 113, when receiving the radar image E, identifies the target Ta using the first learning model 15, which outputs identification result information of the target Ta. The first learning model 15 is updated by the learning model update unit 120.
[0074] When the radar image E and the characteristic parameters CP of the target Ta calculated by the characteristic position editing unit 116 are input, the second target identification unit 117A selects a target candidate from among target candidates corresponding to the known characteristic parameters using a second learning model 15A that outputs target candidates corresponding to the characteristic parameters CP of the target Ta. The second learning model 15A is updated by the learning model update unit 120.
[0075] The annotation processing unit 119 generates annotation information F that links the identification result of the target Ta by the second target identification unit 117A to the radar image E and the characteristic parameters CP of the target Ta calculated by the characteristic position editing unit 116. For example, if the target Ta is a ship, the characteristic parameters CP may be the LOA of the ship, the length between the bow and the mast, or the length between the bow and the container, etc.
[0076] The learning model update unit 120 updates one or both of the first learning model 15 and the second learning model 15A using the annotation information F, the radar image E, and the feature parameters CP of the target Ta calculated by the feature position editing unit 116. For example, when updating the first learning model 15, the annotation processing unit 119 adds the radar image E and the corresponding annotation information F to the learning database 14 shown in FIG. 4A . Then, the learning model update unit 120 updates the first learning model 15 by re-learning the learning data including the added learning data. In this way, by updating the first learning model 15, the classification device 100A can identify targets using the first learning model 15 with improved inference accuracy.
[0077] Furthermore, when updating the second learning model 15A, the annotation processing unit 119 adds the radar image E, the annotation information F corresponding to this radar image E, and the feature specifications H corresponding to the annotation information F to the learning database 14A shown in FIG. 10A . Then, the learning model update unit 120 updates the second learning model 15A by re-learning the learning data including the added learning data. In this way, by updating the second learning model 15A, the classification device 100A can identify targets using the second learning model 15A with improved inference accuracy.
[0078] FIG. 13 is a flowchart showing a classification method according to the second embodiment, illustrating a case where the second learning model 15A is updated. The aspect angle calculation unit 111 calculates the aspect angle θ based on information indicating the orientation D1 of the target Ta acquired from the radar device 200 (step ST1B-1). For example, the aspect angle calculation unit 111 calculates the aspect angle θ according to the above formula (1). The radar image input unit 112 inputs the radar image E from the radar device 200 (step ST1B-2). Note that the processes of steps ST1B-1 and ST1B-2 may be performed simultaneously, regardless of order. Furthermore, these processes are preprocessing for the classification method according to the second embodiment, and it is assumed that these processes have already been performed in the classification method according to the second embodiment.
[0079] The first target identification unit 113 identifies the target Ta from the radar image E (step ST2B). Next, the feature range calculation unit 114 calculates a feature range I, which is the range of the feature portion of the target Ta, based on the aspect angle information of the target Ta included in the radar image E, the identification result information of the target Ta by the first target identification unit 113, and the known feature specifications for each target candidate (step ST3B). The first display processing unit 115 determines a feature position PA of the target Ta using the feature range I calculated by the feature range calculation unit 114, and generates first display control information for displaying the determined feature position PA superimposed on the radar image E (step ST4B). The display unit 102 displays the radar image E together with the feature position PA of the target Ta based on the first display control information.
[0080] When the feature position editing unit 116 acquires editing operation information indicating an editing operation performed on the feature position PA of the target in the radar image E displayed on the display unit 102 based on the first display control information, it calculates the feature parameters CP of the target Ta using the feature position PA1 of the target for which the editing operation was performed indicated by the editing operation information (step ST5B). The second target identification unit 117A identifies known feature parameters for each target candidate that correspond to the feature parameters CP calculated by the feature position editing unit 116 and selects a target candidate corresponding to the identified feature parameters (step ST6B). Furthermore, the second target identification unit 117A outputs the target identification result, the radar image E, and the feature parameters CP to the annotation processing unit 119. The second display processing unit 118 generates second display control information for displaying the target candidate selected by the second target identification unit 117A (step ST7B). The display unit 102 displays the radar image E together with the re-identified target candidate based on the second display control information.
[0081] The annotation processing unit 119 generates training data consisting of the target identification result, radar image E, and feature parameters CP acquired from the second target identification unit 117A, and adds the generated training data to the training database 14A (step ST8B). The training model update unit 120 updates the second training model 15A by re-training the training data including the training data added by the annotation processing unit 119 from the training database 14A (step ST9B). Thereafter, the second target identification unit 117A executes the processing of step ST6B using the updated second training model 15A.
[0082] In step ST4B, the operator compares the characteristic position PA with the characteristic position PB in the radar image E. If the operator determines that the correspondence between the characteristic position PA and the characteristic position PB is valid, the operator can be convinced that the identification result obtained in step ST3B is not incorrect. On the other hand, if the characteristic position PA and the characteristic position PB are far apart, the operator can determine that the identification result obtained in step ST3B may be incorrect. In this case, by editing the characteristic position PA and re-identifying the target using the edited characteristic position PA1, the operator can be convinced of the re-identification result. In addition to the satisfaction, the increase in identification information due to the information input by the operator actually improves the classification accuracy of the classification device 100A.
[0083] As described above, the classification device 100A according to the second embodiment includes the annotation processing unit 119 that generates annotation information linking the target identification result by the second target identification unit 117A to the radar image and the feature specifications of the target calculated by the feature position editing unit 116, and the learning model updating unit 120 that updates one or both of the first learning model and the second learning model using the annotation information, the radar image, and the feature specifications of the target calculated by the feature position editing unit 116. In this way, by updating one or both of the first learning model 15 and the second learning model 15A, the classification device 100A can identify targets using one or both of the first learning model 15 and the second learning model 15A with improved inference accuracy.
[0084] It is possible to combine the embodiments, modify any of the components of the embodiments, or omit any of the components of the embodiments.
[0085] The classification device according to the present disclosure can be used, for example, to classify ships or aircraft.
[0086] 1 Classification system, 13 Sensor, 14, 14A Learning database, 15 First learning model, 15A Second learning model, 100, 100A Classification device, 101 Memory unit, 102 Display unit, 103 Operation unit, 111 Aspect angle calculation unit, 112 Radar image input unit, 113 First target identification unit, 114 Feature range calculation unit, 115 First display processing unit, 116 Feature position editing unit, 117, 117A Second target identification unit, 118 Second display processing unit, 119 Annotation processing unit, 120 Learning model update unit, 200 Radar device, 201 Transmitting and receiving unit, 202 Tracking processing unit, 1000 Input interface, 1001 Output interface, 1002 Processing circuit, 1003 Processor, 1004 Memory.
Claims
a feature range calculation unit which calculates a feature range that is a range of a feature part of the target based on aspect angle information of the target included in the radar image, identification result information of the target by the first target identification unit, and known feature specifications for each target candidate; a first display processing unit which determines a feature position that is a position of the feature part of the target using the feature range calculated by the feature range calculation unit, and generates first display control information for displaying the determined feature position superimposed on the radar image; a feature position editing unit which, upon obtaining editing operation information indicating an editing operation on the feature position of the target in the radar image displayed on the display unit based on the first display control information, calculates the feature specifications of the target using the feature position of the target where the editing operation indicated by the editing operation information was performed; and a second target identification unit which identifies a feature corresponding to the feature specifications of the target calculated by the feature position editing unit from among the known feature specifications for each target candidate, and selects the target candidate corresponding to the identified feature specifications. a second display processing unit that generates second display control information for displaying the target candidate selected by the second target identification unit.
2. The classification device according to claim 1, characterized in that the first target identification unit, when the radar image is input, identifies the target using a first learning model that outputs identification result information of the target.
3. The classification device according to claim 2, characterized in that the second target identification unit identifies, from among the known characteristic parameters of each of the target candidates, those that are most highly correlated with the characteristic parameters of the target calculated by the characteristic position editing unit, and selects the target candidate that corresponds to the identified characteristic parameter.
4. The classification device according to claim 3, characterized in that the second target identification unit, when inputted with the radar image and the feature parameters of the target calculated by the feature position editing unit, selects the target candidates using a second learning model that outputs the target candidates corresponding to the feature parameters of the target.
5. The classification device described in claim 4, characterized in that when the second learning model is input with the radar image and the characteristic position of the target determined based on the annotation information of the target, it outputs identification information of the target candidate including the name and type of the target candidate as the annotation information.
6. The classification device according to claim 5, further comprising: an annotation processing unit that generates annotation information that links the target identification result by the second target identification unit to the radar image and the feature parameters of the target calculated by the feature position editing unit; and a learning model update unit that updates one or both of the first learning model and the second learning model using the annotation information, the radar image, and the feature parameters of the target calculated by the feature position editing unit.
7. A classification system comprising: a radar device that generates and outputs the radar image including the searched target; a display unit that displays the radar image based on the first display control information and the second display control information; an operation unit that accepts editing operations for characteristic positions of the target in the radar image displayed on the display unit; and a classification device described in any one of claims 1 to 6 that inputs the radar image generated by the radar device and the editing operation information.
8. A classification method using a classification device, comprising: a step in which a first target identification unit identifies a target from a radar image; a step in which a feature range calculation unit calculates a feature range that is a range of a feature part of the target based on aspect angle information of the target included in the radar image, identification result information of the target by the first target identification unit, and known feature specifications for each target candidate; a step in which a first display processing unit determines a feature position that is a position of the feature part of the target using the feature range calculated by the feature range calculation unit, and generates first display control information for displaying the determined feature position superimposed on the radar image; a step in which a feature position editing unit, when acquiring editing operation information indicating an editing operation on the feature position of the target in the radar image displayed on the display unit based on the first display control information, calculates the feature specifications of the target using the feature position of the target for which the editing operation indicated by the editing operation information has been performed; a step in which a second target identification unit identifies a known feature specification for each target candidate that corresponds to the feature specification of the target calculated by the feature position editing unit, and selects the target candidate corresponding to the identified feature specification; A categorization method comprising: a step of generating second display control information by a second display processing unit for displaying the target candidate selected by the second target identification unit.
Citation Information
Patent Citations
Radar device
JP2001264435A
Imaging radar apparatus
JP2004212187A
Radar device, target classification apparatus and target classification program
JP2012026976A
Radar device and signal processing method
JP2021025972A
System and Method for Determining a Change of Position and Orientation of an Object
US20200096629A1
