Training device, training method, and program

Abstract

A training device (2000) acquires training data (10) including training images (20), first angle information (30), and ground truth data (40). The first angle information (30) indicates a first angle of incidence (32) and a first azimuth angle (34). The training device (2000) inputs the training images (20) into a feature extraction model (52) to acquire a first feature set (80). The training device (2000) acquires second angle information (90), performs coordinate transformation on the first feature set (80) based on the first angle information (30) and the second angle information (90) to generate a second feature set (100), and updates the feature extraction model (52) based on the first feature set (80), the second feature set (100), and the ground truth data (40).

Description

[Technical Field]

[0001] The present disclosure generally relates to training devices, training methods, and non-transitory computer-readable storage media. [Background technology]

[0002] There are techniques for analyzing images using models that extract features from images, such as object classification using neural networks. Patent Document 1 discloses a system including a convolutional neural network (CNN) unit configured to receive images generated by a synthetic aperture radar as input and classify objects captured in the input images. The system includes a function for increasing the data used to train the CNN unit. Specifically, the system acquires training data including training images and ground truth data, and generates another image by changing the position, orientation, or both of the objects captured in the training image. Both the training image and the image generated by the system are then used to train the CNN unit. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-125203 Summary of the Invention [Problem to be solved by the invention]

[0004] The only method disclosed in Patent Document 1 is to generate another image based on a given image in order to increase the data used to train a model that handles images. The purpose of this disclosure is to provide a novel technique for training a model that handles images. [Means for solving the problem]

[0005] The present disclosure provides a training device comprising at least one memory configured to store instructions and at least one processor. At least one processor is configured to: acquire training data including training images, first angle information, and ground truth data, the training images being images of an object generated by a sensor, the first angle information indicating a first angle of incidence that is an angle of incidence of the sensor and a first azimuth angle that is an azimuth angle of the object imaged in the training image; input the training images to a feature extraction model to acquire a first feature set that is a set of features extracted from the training images; acquire second angle information indicating a second angle of incidence and a second azimuth angle, where the second angle of incidence, the second azimuth angle, or both are different from their corresponding angles in the first angle information; generate a second feature set by performing a coordinate transformation on the first feature set based on the first angle information and the second angle information; and update the feature extraction model based on the first feature set, the second feature set, and the ground truth data.

[0006] The present disclosure further provides a computer-implemented training method, the training method including: acquiring training data including training images, first angle information, and ground truth data; the training images are images of an object generated by a sensor, the first angle information indicating a first incident angle that is an incident angle of the sensor and a first azimuth angle that is an azimuth angle of the object imaged in the training images; inputting the training images into a feature extraction model to acquire a first feature set that is a set of features extracted from the training images; acquiring second angle information indicating a second incident angle and a second azimuth angle; the second incident angle, the second azimuth angle, or both are different from their corresponding angles in the first angle information; generating a second feature set by performing coordinate transformation on the first feature set based on the first angle information and the second angle information; and updating the feature extraction model based on the first feature set, the second feature set, and the ground truth data.

[0007] The present disclosure further provides a non-transitory computer-readable storage medium that stores the program. The program causes a computer to acquire training data including training images, first angle information, and ground truth data; the training images are images of an object generated by a sensor, the first angle information indicating a first angle of incidence that is an angle of incidence of the sensor and a first azimuth angle that is an azimuth angle of the object imaged in the training images; input the training images into a feature extraction model to acquire a first feature set that is a set of features extracted from the training images; acquire second angle information indicating a second angle of incidence and a second azimuth angle; the second angle of incidence, the second azimuth angle, or both are different from their corresponding angles in the first angle information; generate a second feature set by performing coordinate transformation on the first feature set based on the first angle information and the second angle information; and update the feature extraction model based on the first feature set, the second feature set, and the ground truth data. [Effects of the Invention]

[0008] According to the present disclosure, novel techniques are provided for training models to handle images. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram showing an overview of a training device according to a first embodiment. [Figure 2] FIG. 10 is a diagram illustrating an example of training data. [Figure 3] 1 is a block diagram showing an example of the functional configuration of a training apparatus according to a first embodiment. [Figure 4] FIG. 2 is a block diagram showing an example of the hardware configuration of a computer that realizes the training device of the first embodiment. [Figure 5] 1 shows a flowchart illustrating an exemplary flow of processing executed by the training device of embodiment 1. [Figure 6] 1 illustrates the feature extraction performed by the feature extraction model. [Figure 7]This shows how the angle of incidence affects the appearance of objects in a radar image. [Figure 8] This shows the coordinate transformation from the first coordinate system to the second coordinate system. [Figure 9] Another example of a transformation from a first feature set 80 to a second feature set 100 is shown. [Figure 10] 10 illustrates feature modification using first angle information and second angle information. [Figure 11] 1 illustrates an example of a method for extracting features from a set of first angle information and second angle information. DETAILED DESCRIPTION OF THE INVENTION

[0010] Embodiments according to the present disclosure will be described below with reference to the drawings. The same elements are assigned the same reference numerals throughout the drawings, and redundant description will be omitted as necessary. Furthermore, unless otherwise described, predetermined information (e.g., predetermined values ​​or predetermined threshold values) is pre-stored in a storage unit accessed by a computer that uses the information. In the present disclosure, the storage unit may be implemented by one or more storage devices such as a hard disk, a solid-state drive (SSD), or random-access memory (RAM).

[0011] Embodiment 1 <Summary> Fig. 1 shows an overview of a training device 2000 according to embodiment 1. It should be noted that Fig. 1 does not limit the operation of the training device 2000, but merely shows one example of possible operation of the training device 2000.

[0012] The training device 2000 is a device configured to acquire training data 10 and train a model set 50 using the training data 10. The model set 50 includes a feature extraction model 52 and a task performance model 54. The feature extraction model 52 and the task performance model 54 may be machine learning-based models such as neural networks.

[0013] The feature extraction model 52 is configured to take an image as input, extract features from the input image, and output the extracted features. The task execution model 54 is configured to take features as input, execute a task on the input features, and output the results of the task. Examples of tasks performed by the task execution model 54 include object detection, object classification, semantic segmentation, and image reconstruction.

[0014] The training data 10 includes training images 20, first angle information 30, and ground truth data 40. Fig. 2 shows an example of the training data 10. The training images 20 are images including an object 22 generated by a sensor 70. The training images 20 may be optical images or radar images.

[0015] If the training images 20 are optical images, the sensor 70 is an optical camera configured to receive light and generate an optical image based on the received light. If the training images 20 are radar images, the sensor 70 is a radar configured to transmit radio waves, receive reflections of the radio waves, and generate a radar image based on the reflections of the received radio waves. The sensor 70 may be installed on a satellite to capture objects such as the Earth, other planets, or satellites. An example of a radar is a synthetic aperture radar.

[0016] The first angle information 30 indicates a first angle of incidence 32 and a first azimuth angle 34. The first angle of incidence 32 represents the angle of incidence of the sensor 70 when the sensor 70 captured the object 22 to generate the training image 20. The first azimuth angle 34 is the azimuth angle of the object 22 when the sensor 70 captured the object 22 to generate the training image 20.

[0017] The ground truth data 40 is data that indicates a ground truth for training the model set 50. Assume that the model set 50 performs object classification on an image. In FIG. 2, the object 22 is a ship, so the ground truth data 40 indicates the class of "ship."

[0018] To train the model set 50, the training device 2000 may operate as follows: The training device 2000 acquires training data 10 and inputs training images 20 in the acquired training data 10 into the feature extraction model 52. As a result, the training device 2000 acquires a first feature set 80, which is a set of features extracted from the training images 20 by the feature extraction model 52.

[0019] The training device 2000 generates another feature set, called "second feature set 100," from the first feature set 80 in order to train the model set 50. To do so, the training device 2000 further acquires second angle information 90 indicating a second angle of incidence 92 and a second azimuth angle 94. The second angle of incidence 92 is not equal to the first angle of incidence 32, the second azimuth angle 94 is not equal to the first azimuth angle 34, or both.

[0020] The training device 2000 performs a coordinate transformation on the first feature set 80 based on the first angle information 30 and the second angle information 90, thereby converting the first feature set 80 into a second feature set 100. The second feature set 100 is thereby generated to represent features of an image having the second angle of incidence 92 and the second azimuth angle 94. Specifically, the second feature set 100 represents features of an image captured by the sensor 70 having an angle of incidence equal to the second angle of incidence 92 and depicting the object 22 having an azimuth angle equal to the second azimuth angle 94.

[0021] The training device 2000 trains the model set 50 using the first feature set 80, the second feature set 100, and the supervised answer data 40. The training of the model set 50 will be described in detail below.

[0022] <Examples of effects> It is preferable to use multiple images with various pairs of incidence angles and azimuth angles to train the model set 50. In particular, when images are generated by radar, due to the nature of radar imaging physics, as will be described in more detail below, the radar's incidence angle and the azimuth angle of the captured object can affect how the object appears in the image. However, there are some situations in which it is difficult to prepare a sufficient number of images for model training.

[0023] The training device 2000 provides a novel technique for training a model that handles images. Specifically, a first feature set 80 is extracted from training images 20, and a second feature set 100 is generated by performing a coordinate transformation on the first feature set 80. Then, a model set 50 is trained using both the first feature set 80 and the second feature set 100.

[0024] The second feature set 100 represents features of an image captured by the sensor 70 at a particular angle of incidence and depicting the object 22 at a particular azimuth angle. By generating the second feature set 100 from the first feature set 80, the training device 2000 can obtain the features of the image without actually obtaining another image. Thus, the training device 2000 can increase the number of sets of image features used to train the model set 50, thereby facilitating the collection of training data for training the model set 50. The training device 2000 can also facilitate improving the accuracy of the model set 50.

[0025] The training device 2000 is described in more detail below.

[0026] <Example of functional configuration> 3 is a block diagram showing an example of the functional configuration of the training device 2000 according to embodiment 1. The training device 2000 includes a training data acquiring unit 2020, an angle information acquiring unit 2040, a feature acquiring unit 2060, a converting unit 2080, and an updating unit 2100.

[0027] The training data acquisition unit 2020 acquires training data 10. The angle information acquisition unit 2040 acquires second angle information 90. The feature acquisition unit 2060 inputs training images 20 to the feature extraction model 52 and acquires a first feature set 80 extracted from the training images 20 by the feature extraction model 52. The transformation unit 2080 performs coordinate transformation on the first feature set 80 based on the first angle information 30 and the second angle information 90, thereby transforming the first feature set 80 into a second feature set 100. The update unit 2100 updates the model set 50 using the first feature set 80, the second feature set 100, and the correct answer data 40.

[0028] <Example of hardware configuration> Training device 2000 may be realized by one or more computers, each of which may be a dedicated computer manufactured for implementing training device 2000, or a general-purpose computer such as a personal computer (PC), a server machine, or a mobile device.

[0029] Training device 2000 may be realized by installing an application on a computer. The application is implemented by a program that causes a computer to function as training device 2000. In other words, the program is an implementation form of the functional parts of training device 2000. There are various methods for acquiring the program. For example, the program can be acquired from a storage medium (e.g., a DVD disk or a USB memory) on which the program is pre-stored. In another example, the program can be acquired by downloading it from a server machine that manages the storage medium on which the program is pre-stored.

[0030] Fig. 4 is a block diagram showing an example of the hardware configuration of a computer 1000 that realizes the training device 2000 of embodiment 1. In Fig. 4, the computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output (I / O) interface 1100, and a network interface 1120.

[0031] The bus 1020 is a data transmission channel through which the processor 1040, memory 1060, storage device 1080, input / output interface 1100, and network interface 1120 transmit and receive data to and from each other. The processor 1040 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), or a digital signal processor (DSP). The memory 1060 is a main storage element such as a random access memory (RAM) or a read-only memory (ROM). The storage device 1080 is an auxiliary storage element such as a hard disk, a solid-state drive (SSD), or a memory card. The input / output interface 1100 is an interface between the computer 1000 and peripheral devices such as a keyboard, a mouse, or a display device. The network interface 1120 is an interface between the computer 1000 and a network. The network may be a local area network (LAN) or a wide area network (WAN). The storage device 1080 may store the above-mentioned programs. The processor 1040 executes a program to realize each functional unit of the training device 2000.

[0032] The hardware configuration of computer 1000 is not limited to that shown in Figure 4. For example, as described above, training device 2000 may be realized by multiple computers, which may be connected to each other via a network.

[0033] <Processing flow> 5 is a flowchart illustrating an exemplary flow of processing performed by the training device 2000 of the first embodiment. The training data acquisition unit 2020 acquires training data 10 (S102). The angle information acquisition unit 2040 acquires second angle information 90 (S104). The feature acquisition unit 2060 inputs training images 20 into the feature extraction model 52 to acquire a first feature set 80 (S106). The transformation unit 2080 performs coordinate transformation on the first feature set 80 based on the first angle information 30 and the second angle information 90, thereby transforming the first feature set 80 into a second feature set 100 (S108). The update unit 2100 updates the model set 50 using the first feature set 80, the second feature set 100, and the supervised answer data 40 (S110).

[0034] Note that Figure 5 shows only an example of a possible processing flow that can be performed by the training device 2000, and the processing flow that can be performed by the training device 2000 is not limited to that shown in Figure 5. For example, the acquisition of the second angle information 90 (S104) may be performed at any timing before the coordinate transformation (S108).

[0035] <Acquisition of training data 10: S102> The training data acquisition unit 2020 acquires the training data 10 (S102). There are various methods for acquiring the training data 10. In some implementations, the training data acquisition unit 2020 can receive the training data 10 transmitted from another computer, such as a computer that generates the training data 10. In other implementations, the training data may be pre-stored in a storage unit that the training data acquisition unit 2020 accesses. In this case, the training data acquisition unit 2020 reads the training data 10 from the storage unit.

[0036] The training data acquiring unit 2020 may acquire two or more pieces of training data 10. In this case, the training device 2000 may train the model set 50 using each piece of data.

[0037] There may be various methods for determining the number of training data 10 to be acquired. For example, the number of training data 10 to be acquired may be predetermined, may be randomly determined by the training data acquiring unit 2020, or may be specified by a user of the training device 2000. In another example, the training data acquiring unit 2020 may acquire all of the prepared training data 10 (e.g., all of the training data 10 stored in a storage device).

[0038] <Acquisition of second angle information 90: S104> The angle information acquisition unit 2040 acquires the second angle information 90 (S104). Note that the number of pieces of second angle information 90 acquired by the angle information acquisition unit 2040 may not be limited to one. When the angle information acquisition unit 2040 acquires two or more pieces of second angle information 90, the conversion unit 2080 may generate a second feature set 100 for each piece of second angle information 90.

[0039] There may be various methods for determining the number of pieces of second angle information 90 to be acquired. For example, the number of pieces of second angle information 90 to be acquired may be predetermined, may be randomly determined by the angle information acquisition unit 2040, or may be specified by the user of the training apparatus 2000. In another example, the angle information acquisition unit 2040 may acquire all of the prepared pieces of second angle information 90 (e.g., all of the second angle information 90 stored in a storage device).

[0040] The second angle information 90 may be prepared in advance or may be dynamically generated by the angle information acquisition unit 2040. In the former case, various pairs of incident angles and azimuth angles may be stored in advance in a storage device accessed by the training device 2000 as candidates for the second angle information 90. The angle information acquisition unit 2040 may acquire the second angle information 90 from the storage device by selecting one of the candidates whose second incident angle 92, second azimuth angle 94, or both are not equal to the corresponding ones in the first angle information 30. The candidate for the second angle information 90 may be selected randomly or based on a specific rule.

[0041] When the second angle information 90 is dynamically generated, the angle information acquisition unit 2040 may randomly determine the second angle of incidence 92 and the second azimuth angle 94 to generate the second angle information 90. When the second angle of incidence 92 and the second azimuth angle 94 are equal to the first angle of incidence 32 and the first azimuth angle 34, respectively, the training device 2000 may again randomly determine the second angle of incidence 92, the second azimuth angle 94, or both, so that the second angle information 90 is not equal to the first angle information 30.

[0042] <Acquisition of first feature set 80: S106> The feature acquisition unit 2060 inputs the training images 20 into the feature extraction model 52 to acquire a first feature set 80 (S106). The feature extraction model 52 is configured to extract features of the images input thereto and output the extracted features. In this way, when the feature acquisition unit 2060 inputs the training images 20 into the feature extraction model 52, the feature extraction model 52 extracts features of the training images 20 and outputs the features extracted from the training images 20. The feature acquisition unit 2060 acquires the features of the training images 20 output from the feature extraction model 52 as the first feature set 80.

[0043] The feature extraction model 52 will now be described in more detail.

[0044] The feature extraction model 52 is configured to extract three-dimensional spatial features of the scene captured on the input image. Figure 6 illustrates the feature extraction performed by the feature extraction model 52. The feature extraction model 52 can be configured as a neural network, such as a convolutional neural network (CNN), having multiple filters for extracting multiple local spatial features for each sub-region 210 of the image 200 input thereto.

[0045] The feature extraction model 52 may be trained to generate a set of features of the input image 200, which may be represented by a set of cells having feature vectors (i.e., feature values) and coordinates in a particular coordinate system. In this disclosure, the term "cell" is used to describe a value and coordinate pair. A feature vector corresponding to particular coordinates represents the spatial features of the three-dimensional subregion of the scene corresponding to those coordinates. The set of cells may be represented by a cuboid of cells, each representing a feature vector corresponding to the cell's coordinates. Hereinafter, this cuboid of cells will be referred to as a "feature cuboid."

[0046] In the following, unless otherwise specified, the set of features extracted by the feature extraction model 52 will be described as a feature cuboid. However, the technology described in this disclosure is also applicable to cases where the set of features extracted by the feature extraction model 52 is represented in a form other than a cuboid (for example, a list of cells).

[0047] The feature cuboid 220 generated by the feature extraction model 52 is a cuboid in the first coordinate system 130 defined by a first orientation axis 132, a first range axis 134, and a first incidence axis 136. The first orientation axis 132 is on a horizontal plane and represents a reference orientation (e.g., east). The first range axis 134 is on a horizontal plane and is perpendicular to the first orientation axis 132. The first incidence axis 136 is an axis that defines the angle of incidence of the input image from a direction opposite to the direction of gravity (vertically upward). As shown in FIG. 6 , features of a subregion 210 of the input image 200 can be extracted as a sequence of cells of the feature cuboid 220 along the first incidence axis 136. Hereinafter, this sequence of cells will be referred to as a “cell sequence 230.”

[0048] When training images 20 are input to feature extraction model 52, the angle of incidence of the input images is first angle of incidence 32. Thus, a first coordinate system corresponding to first feature set 80 may be defined by first angle of incidence 32.

[0049] The first feature set 80 is further described below in terms of the nature of radar imaging physics. As noted above, radar imaging physics dictates that the radar's angle of incidence and the object's azimuth angle affect the appearance of an object in an image. Figure 7 illustrates how the angle of incidence affects the appearance of an object in a radar image.

[0050] The incidence angle of the radar 75 is different between the example on the left side of Fig. 7 and the example on the right side of Fig. 7. In the example on the left side, there is a line 160-1 that passes through the sub-region 210 and forms an incidence angle T1 from the horizontal plane. Therefore, the line 160-1 passes through the three-dimensional space of the real world that is projected onto the sub-region 210 on the image plane of the image 200.

[0051] In contrast, in the example on the right, there is a line 160-2 that passes through subregion 210 and forms an angle of incidence T2 from the horizontal plane. Thus, line 160-2 passes through real-world three-dimensional space that is projected onto subregion 210 on the image plane of image 200.

[0052] Due to the nature of radar imaging physics, the intensity of sub-region 210 can be calculated as the sum of the backscattering of points along line 160. Thus, in the example on the left, the intensity of sub-region 210 is the sum of the backscattering from p1 to pn. Similarly, in the example on the right, the intensity of sub-region 210 is the sum of the backscattering from q1 to qn. This means that the intensity of sub-region 210 on image 200 depends on the angle of incidence of radar 75.

[0053] Also, as the azimuth angle of object 22 changes, the point of object 22 along line 160 changes. Therefore, it can be said that the azimuth angle of object 22 affects the intensity of sub-region 210 on image 200 due to the nature of radar imaging physics.

[0054] 6, features of subregions 210 of image 200 may be extracted as sequences 230, which are sequences of cells along first axis of incidence 136. The direction represented by first axis of incidence 136 corresponds to the direction of line 160. Thus, feature extraction model 52 may be trained to generate, for each subregion 210, a feature cuboid 220 containing sequences 230 representing features of points along line 160 passing through that subregion 210.

[0055] <Coordinate conversion: S108> The conversion unit 2080 performs coordinate conversion on the first feature set 80 based on the first angle information 30 and the second angle information 90 to generate the second feature set 100 (S108). The coordinate conversion performed by the conversion unit 2080 is a coordinate conversion from the first coordinate system 130 defined by the first angle information 30 to the second coordinate system defined by the second angle information 90.

[0056] 8 shows a coordinate transformation from a first coordinate system 130 to a second coordinate system 150. This coordinate transformation can be decomposed into first to third coordinate transformations.

[0057] The first coordinate transformation M1 is a coordinate transformation from the first coordinate system 130 to the world coordinate system 140. The world coordinate system 140 is a real-world coordinate system defined by the first orientation axis 132, the first range axis 134, and the vertical-upward axis 146. The vertical-upward axis 146 is an axis that represents the direction opposite to the direction of gravity.

[0058] The second coordinate transformation M2 rotates the world coordinate system 140 by a rotation angle defined by the difference between the second azimuth angle 94 and the first azimuth angle 34. The second coordinate transformation rotates the first azimuth axis 132 and the first range axis 134 about a vertically upward axis to obtain the second azimuth axis 152 and the second range axis 154. Let the first azimuth angle 34 and the second azimuth angle 94 be S1 and S2, respectively. In this case, the second azimuth axis 152 and the second range axis 154 are obtained by rotating the first azimuth axis 132 and the first range axis 134 about a vertically upward axis by S2-S1.

[0059] The third coordinate transformation M3 is a coordinate transformation from the world coordinate system 140 rotated by the rotation angle to a second coordinate system 150. The second coordinate system 150 is a coordinate system defined by a second azimuth axis 152, a second range axis 154, and a second incidence axis 156. The second incidence axis 156 is an axis that forms a second incidence angle 92 from the vertical upaxis axis 146.

[0060] 8, the first coordinate transformation, the second coordinate transformation, and the third coordinate transformation are represented by transformation matrices M1, M2, and M3, respectively. Under this assumption, the coordinate transformation from the first coordinate system 130 to the second coordinate system 150 can be expressed as follows: formula 1

number

[0061] The transform unit 2080 determines the joint transform matrix Mc by determining the transformation matrices M1, M2, and M3. Note that there are known methods for calculating the transformation matrix between two coordinate systems, and one of these methods can be applied to the transform unit 2080 to determine the transformation matrices M1, M2, and M3.

[0062] As described above, the first feature set 80 can be represented by a rectangular parallelepiped of cells in the first coordinate system 130. The transformation unit 2080 transforms the rectangular parallelepiped of cells of the first feature set 80 using the transformation matrix Mc to obtain a rectangular parallelepiped of cells in the second coordinate system 150 as the second feature set 100.

[0063] Specifically, the transformation unit 2080 may use the transformation matrix Mc to transform the coordinates of each cell in the first feature set 80 in the first coordinate system 130 into coordinates in the second coordinate system 150. In this way, the transformation unit 2080 identifies cells in the second feature set 100 that correspond to cells in the first feature set 80. Next, the transformation unit 2080 sets the value of the cell in the first feature set 80 to the corresponding cell in the second feature set 100.

[0064] It is assumed that (x1, y1, z1) in the first coordinate system 130 is transformed into (x2, y2, z2) in the second coordinate system 150 by coordinate transformation using the combined transformation matrix Mc. In this case, the transformation unit 2080 may set the value of the cell (x1, y1, z1) in the first feature set 80 to the cell (x2, y2, z2) in the second feature set 100.

[0065] In another example, the transformation unit 2080 may calculate the inverse matrix of the connection matrix Mc, represented by Mc^-1, to calculate the second feature set 100. In this case, the transformation unit 2080 transforms the coordinates of each cell of the second feature set 100 in the second coordinate system 150 into coordinates in the first coordinate system 130. In this way, the transformation unit 2080 identifies cells in the first feature set 80 that correspond to cells in the second feature set 100. Next, the transformation unit 2080 sets the value of the cell in the first feature set 80 to the corresponding cell in the second feature set 100.

[0066] <<Feature Modification with Trainable Models>> The transformation unit 2080 may further perform feature modification using a trainable model called a "feature modification model" after the above-described coordinate transformation. FIG. 9 shows another example of the transformation from the first feature set 80 to the second feature set 100. The transformation unit 2080 first performs coordinate transformation on the feature cuboid 220 obtained as the feature set 80, thereby obtaining a feature cuboid 240. Next, the transformation unit 2080 inputs the feature cuboid 240 into the feature modification model 250.

[0067] The feature correction model 250 is configured to receive the feature cuboid 240 as input, correct the cell values ​​of the feature cuboid 240, and output a feature cuboid 260. The conversion unit 2080 outputs the feature cuboid 260 as the second feature set 100.

[0068] The feature correction model 250 can be implemented as a machine learning-based model, such as a neural network. There are various ways to correct the feature cuboid 240 to generate the second feature set 100. For example, the feature correction model 250 can be configured to calculate, for each cell of the feature cuboid 240, a weighted sum of the value of that cell and the values ​​of surrounding (e.g., adjacent) cells. The weighted sum calculated for a cell of the feature cuboid 240 is set for the corresponding cell of the feature cuboid 260. In this case, the weights are parameters to be trained.

[0069] In another example, the first angle information 30 and the second angle information 90 are also used for feature correction. FIG. 10 illustrates feature correction using the first angle information 30 and the second angle information 90. The transform unit 2060 may calculate a difference feature between the first angle information 30 and the second angle information 90 to generate a feature cuboid 270. The transform unit 2060 concatenates the feature cuboid 270 with the feature cuboid 240 to obtain a feature cuboid 280. As illustrated in FIG. 10 , the feature cuboid 270 is configured to have the same size as the feature cuboid 240 along the second orientation axis 152 and the second range axis 154 so that the feature cuboid 270 can be concatenated with the feature cuboid 240.

[0070] Next, the conversion unit 2060 inputs the feature cuboid 280 into the feature correction model 250 to obtain a feature cuboid 290 as the second feature set 100. In this case, the feature correction model 250 is configured to receive the feature cuboid 280 as input and output the feature cuboid 290. To convert the feature cuboid 280 into the feature cuboid 290, the feature correction model 250 is configured to correct the values ​​of the cells of the feature cuboid 240 using the values ​​of the cells of the feature cuboid 270.

[0071] There are various methods for extracting features from the set of first angle information 30 and second angle information 90. Fig. 11 shows an example of a method for extracting features from the set of first angle information 30 and second angle information 90. Briefly, the transform unit 2060 calculates the difference in incidence angle (hereinafter referred to as "incident angle difference") and the difference in azimuth angle (hereinafter referred to as "azimuth angle difference"). Then, the transform unit 2060 calculates the feature of the difference in incidence angle and the feature of the difference in azimuth angle, and concatenates them to obtain a feature cuboid 270.

[0072] An exemplary method for calculating the incidence angle difference feature is first described below: The converter 2060 calculates the incidence angle difference (i.e., the difference between the first incidence angle 32 and the second incidence angle 92) and quantizes the incidence angle difference to obtain one of predetermined integers.

[0073] In some embodiments, the entire range of incident angles (e.g., 360°) is divided into specific intervals to define predetermined integers. For example, if the entire range of incident angles is 360° and the division interval is 10°, the entire range of incident angles is divided into 36 bins. In this case, 1 to 36 are used as the predetermined integers. Assume that the incident angle difference is 35°. In this case, 35° belongs to the fourth bin, so the incident angle difference is quantized to 4.

[0074] The transform unit 2060 inputs the quantized incident angle difference into a transform model 300 to obtain a feature cuboid 330 that represents the feature of the incident angle difference and whose size is the same as that of the feature cuboid 240 along the second orientation axis 152 and the second range axis 154. The transform model 300 includes an embedding layer 310 and an encoding layer 320. The embedding layer 310 and the encoding layer 320 may be implemented as machine learning-based models, such as neural networks, and may therefore be trainable.

[0075] The embedding layer 310 is configured to receive the quantized incident angle difference as input and convert the input data into a random number that encodes the incident angle difference. The embedding layer 310 is trained to map each integer obtained by quantizing the incident angle difference to a specific random number. In other words, each bin of the quantized incident angle difference is associated with a specific random number by training the embedding layer 310.

[0076] The calculated random numbers are output as vectors and input to the encoding layer 320. The encoding layer 320 is configured to perform a transpose convolution on the input vectors to generate feature cuboids 330.

[0077] The azimuth angle difference feature is calculated in a manner similar to that for the incidence angle difference feature. The transform unit 2060 calculates the azimuth angle difference (i.e., the difference between the first azimuth angle 34 and the second azimuth angle 94) and quantizes the azimuth angle difference to obtain one of predetermined integers. The transform unit 2060 then inputs the quantized azimuth angle difference into the transformation model 340 to obtain a feature cuboid 370 that represents the azimuth angle difference feature and whose size is the same as that of the feature cuboid 240 along the second azimuth axis 152 and the second range axis 154. The transformation model 340 includes an embedding layer 350 and an encoding layer 360. The embedding layer 350 and the encoding layer 360 may be implemented as machine learning-based models, such as neural networks, and thus may be trainable.

[0078] The embedding layer 350 is configured to receive the quantized azimuth angle difference as input and convert the input data into a random number that encodes the azimuth angle difference. Each bin of the quantized azimuth angle difference is associated with a specific random number through training of the embedding layer 350. The calculated random number is output as a vector and input to the encoding layer 360. The encoding layer 360 is configured to perform a transposed convolution on the input vector to generate a feature cuboid 370.

[0079] After calculating the feature cuboid 330 and the feature cuboid 370 , the conversion unit 2060 concatenates them to obtain the feature cuboid 270 that represents the features of the set of first angle information 30 and second angle information 90 .

[0080] <Model Set 50 Update: S110> The update unit 2100 updates the model set 50 using the first feature set 80 and the second feature set 100 (S110). Specifically, the update unit 2100 inputs the first feature set 80 to the task execution model 54 and obtains a task result from the task execution model 54. Then, the update unit 2100 calculates a loss based on the supervised answer data 40 and the result of the task performed using the first feature set 80. Similarly, the update unit 2100 inputs the second feature set 100 to the task execution model 54 and obtains a task result from the task execution model 54. Then, the update unit 2100 calculates a loss based on the supervised answer data 40 and the result of the task performed using the second feature set 100. If two or more second feature sets 100 are generated by the conversion unit 2080, the update unit 2100 may calculate a loss for each of the second feature sets 100.

[0081] The calculated losses are used to train the model set 50. There are various methods for training a model based on losses, and one of these methods can be applied to the update unit 2100. For example, the update unit 2100 can calculate a batch loss with the calculated losses (e.g., calculate the average of the calculated losses) and use the batch loss to update the trainable parameters of the model set 50. In another example, the update unit 2100 can use each of the calculated losses separately to update the trainable parameters of the model set 50.

[0082] If the transform unit 2060 includes the feature correction model 250, the update unit 2100 may also use the calculated loss to update the trainable parameters (e.g., weights for calculating the weighted sum described above) of the feature correction model 250. Similarly, if the transform unit 2060 includes the transformation model 300 and the transformation model 340, the update unit 2100 may use the calculated loss to update the trainable parameters of the transformation model 300 and the trainable parameters of the transformation model 340. In another example, the feature correction model 250, the transformation model 300, and the transformation model 340 may be trained before training the model set 50.

[0083] <Output from Training Device 2000> The training device 2000 can output the results of training the model set 50. The results of training can be output in any manner. For example, the training device 2000 can store the trained parameters of the model set 50 (e.g., weights assigned to each connection of the neural network) in a memory unit. In another example, the training device 2000 can send the trained parameters to another device used to execute the model set 50. Note that in addition to the parameters, a program implementing the model set 50 may also be output.

[0084] During the operation phase of model set 50, if training device 2000 is also used to execute model set 50, training device 2000 may not output the training results. In this case, from the perspective of the user of training device 2000, it is preferable for training device 2000 to notify the user that training of model set 50 has finished.

[0085] The program can be stored and provided to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs, CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, programmable ROMs (PROMs), erasable PROMs (EPROMs), flash ROMs, and RAMs). The program may also be provided to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can provide the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0086] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that are understandable to those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present invention.

[0087] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes. <Additional Notes> (Appendix 1) 1. A training device comprising: at least one memory configured to store instructions; at least one processor executing the instructions to: acquiring training data including training images, first angle information, and ground truth data, the training images being images of an object captured by a sensor, the first angle information indicating a first incident angle that is an incident angle of the sensor and a first azimuth angle that is an azimuth angle of the object captured in the training images; inputting the training images into a feature extraction model to obtain a first feature set, the first feature set being a set of features extracted from the training images; acquiring second angle information indicative of a second angle of incidence and a second azimuth angle, wherein the second angle of incidence, the second azimuth angle, or both, differ from their corresponding angles in the first angle information; generating a second feature set by performing a coordinate transformation on the first feature set based on the first angle information and the second angle information; updating the feature extraction model based on the first feature set, the second feature set, and the ground truth data; and at least one processor configured to perform the steps of: (Appendix 2) the first feature set is represented by a set of cells, each cell having a feature value and a coordinate in a first coordinate system defined using the first angle of incidence; the second feature set is represented by a set of cells, each cell having a feature value and a coordinate in a second coordinate system defined using the second incidence angle, the first azimuth angle, and the second azimuth angle; Performing the coordinate transformation on the first feature set includes: For each cell of the first feature set, calculating a corresponding cell of the second feature set by performing a coordinate transformation from the first coordinate system to the second coordinate system on the coordinates of the cell of the first feature set; and setting the value of the cell of the first feature set to the corresponding cell of the second feature set. (Appendix 3) The coordinate transformation from the first coordinate system to the second coordinate system is a transformation from the first coordinate system to a world coordinate system defined by a horizontal plane and a vertically upward axis representing a direction opposite to the direction of gravity; rotating the world coordinate system about the vertically upward axis by a rotation angle, the rotation angle being a difference between the first azimuth angle and the second azimuth angle; 3. The training device of claim 2, comprising a transformation from the world coordinate system rotated by the rotation angle to the second coordinate system. (Appendix 4) the first feature set is represented by a first set of cells, each cell having a feature value and a coordinate in a first coordinate system defined using the first angle of incidence; the second feature set is represented by a second set of cells, each cell having a feature value and a coordinate in a second coordinate system defined using the second incidence angle, the first azimuth angle, and the second azimuth angle; Generating the second feature set includes: performing the coordinate transformation on the first set of features to transform the first set of features into a third set of cells in the second coordinate system; and modifying values ​​of one or more cells of the third set to generate the second feature set. (Appendix 5) Generating the second feature set includes: calculating a difference characteristic between the first angle information and the second angle information; and modifying the values ​​of one or more cells of the third set using a characteristic of the difference between the first angle information and the second angle information. (Appendix 6) 2. The training device of claim 1, wherein the training images are radar images generated by a radar. (Appendix 7) 7. The training device of claim 6, wherein the first feature set represents, for each subregion on the training image, backscattering characteristics at each of two or more points projected onto the subregion along a line forming the first angle of incidence from the image plane onto an image plane of the training image. (Appendix 8) Updating the feature extraction model includes: inputting the first feature set into a task execution model to obtain a first result of the task; inputting the second set of features into the task execution model to obtain a second result of the task; calculating one or more losses based on the first result of the task, the second result of the task, and the ground truth data; and updating trainable parameters of the feature extraction model and the task performance model based on the one or more losses. (Appendix 9) 1. A computer-implemented training method comprising: acquiring training data including training images, first angle information, and ground truth data, the training images being images of an object captured by a sensor, the first angle information indicating a first incident angle that is an incident angle of the sensor and a first azimuth angle that is an azimuth angle of the object captured in the training images; inputting the training images into a feature extraction model to obtain a first feature set, the first feature set being a set of features extracted from the training images; acquiring second angle information indicative of a second angle of incidence and a second azimuth angle, wherein the second angle of incidence, the second azimuth angle, or both, differ from their corresponding angles in the first angle information; generating a second feature set by performing a coordinate transformation on the first feature set based on the first angle information and the second angle information; updating the feature extraction model based on the first feature set, the second feature set, and the ground truth data. (Appendix 10) the first feature set is represented by a set of cells, each cell having a feature value and a coordinate in a first coordinate system defined using the first angle of incidence; the second feature set is represented by a set of cells, each cell having a feature value and a coordinate in a second coordinate system defined using the second incidence angle, the first azimuth angle, and the second azimuth angle; Performing the coordinate transformation on the first feature set includes: For each cell of the first feature set, calculating a corresponding cell of the second feature set by performing a coordinate transformation from the first coordinate system to the second coordinate system on the coordinates of the cell of the first feature set; and setting the value of the cell of the first feature set to the corresponding cell of the second feature set. (Appendix 11) The coordinate transformation from the first coordinate system to the second coordinate system is a transformation from the first coordinate system to a world coordinate system defined by a horizontal plane and a vertically upward axis representing a direction opposite to the direction of gravity; rotating the world coordinate system about the vertically upward axis by a rotation angle, the rotation angle being a difference between the first azimuth angle and the second azimuth angle; 11. The training method of claim 10, comprising transforming the world coordinate system rotated by the rotation angle to the second coordinate system. (Appendix 12) the first feature set is represented by a first set of cells, each cell having a feature value and a coordinate in a first coordinate system defined using the first angle of incidence; the second feature set is represented by a second set of cells, each cell having a feature value and a coordinate in a second coordinate system defined using the second incidence angle, the first azimuth angle, and the second azimuth angle; Generating the second feature set includes: performing the coordinate transformation on the first set of features to transform the first set of features into a third set of cells in the second coordinate system; and modifying values ​​of one or more cells of the third set to generate the second feature set. (Appendix 13) Generating the second feature set includes: calculating a difference characteristic between the first angle information and the second angle information; and modifying the value of one or more cells of the third set using the difference feature between the first angle information and the second angle information. (Appendix 14) 10. The training method of claim 9, wherein the training images are radar images generated by a radar. (Appendix 15) 15. The training method of claim 14, wherein, for each subregion on the training image, the first feature set represents backscattering features at each of two or more points projected onto the subregion along a line forming the first angle of incidence from the image plane onto an image plane of the training image. (Appendix 16) Updating the feature extraction model includes: inputting the first feature set into a task execution model to obtain a first result of the task; inputting the second set of features into the task execution model to obtain a second result of the task; calculating one or more losses based on the first result of the task, the second result of the task, and the ground truth data; and updating trainable parameters of the feature extraction model and the task performance model based on the one or more losses. (Appendix 17) A non-transitory computer-readable storage medium storing a program for a computer, the program including: acquiring training data including training images, first angle information, and ground truth data, the training images being images of an object captured by a sensor, the first angle information indicating a first incident angle that is an incident angle of the sensor and a first azimuth angle that is an azimuth angle of the object captured in the training images; inputting the training images into a feature extraction model to obtain a first feature set, the first feature set being a set of features extracted from the training images; acquiring second angle information indicative of a second angle of incidence and a second azimuth angle, wherein the second angle of incidence, the second azimuth angle, or both, differ from their corresponding angles in the first angle information; generating a second feature set by performing a coordinate transformation on the first feature set based on the first angle information and the second angle information; updating the feature extraction model based on the first feature set, the second feature set, and the ground truth data; A non-transitory computer-readable storage medium that causes the (Appendix 18) the first feature set is represented by a set of cells, each cell having a feature value and a coordinate in a first coordinate system defined using the first angle of incidence; the second feature set is represented by a set of cells, each cell having a feature value and a coordinate in a second coordinate system defined using the second incidence angle, the first azimuth angle, and the second azimuth angle; Performing the coordinate transformation on the first feature set includes: For each cell of the first feature set, calculating a corresponding cell of the second feature set by performing a coordinate transformation from the first coordinate system to the second coordinate system on the coordinates of the cell of the first feature set; and setting the value of the cell of the first feature set to the corresponding cell of the second feature set. (Appendix 19) The coordinate transformation from the first coordinate system to the second coordinate system is a transformation from the first coordinate system to a world coordinate system defined by a horizontal plane and a vertically upward axis representing a direction opposite to the direction of gravity; rotating the world coordinate system about the vertically upward axis by a rotation angle, the rotation angle being a difference between the first azimuth angle and the second azimuth angle; 19. The storage medium of claim 18, comprising a transformation from the world coordinate system rotated by the rotation angle to the second coordinate system. (Appendix 20) the first feature set is represented by a first set of cells, each cell having a feature value and a coordinate in a first coordinate system defined using the first angle of incidence; the second feature set is represented by a second set of cells, each cell having a feature value and a coordinate in a second coordinate system defined using the second incidence angle, the first azimuth angle, and the second azimuth angle; Generating the second feature set includes: performing the coordinate transformation on the first set of features to transform the first set of features into a third set of cells in the second coordinate system; and modifying values ​​of one or more cells of the third set to generate the second feature set. (Appendix 21) Generating the second feature set includes: calculating a difference characteristic between the first angle information and the second angle information; and modifying the values ​​of one or more cells of the third set using the difference characteristic between the first angle information and the second angle information. (Appendix 22) 18. The storage medium of claim 17, wherein the training images are radar images generated by a radar. (Appendix 23) 23. The storage medium of claim 22, wherein the first feature set represents, for each subregion on the training image, backscattering characteristics at each of two or more points projected onto the subregion along a line forming the first angle of incidence from the image plane onto an image plane of the training image. (Appendix 24) Updating the feature extraction model includes: inputting the first feature set into a task execution model to obtain a first result of the task; inputting the second set of features into the task execution model to obtain a second result of the task; calculating one or more losses based on the first result of the task, the second result of the task, and the ground truth data; updating trainable parameters of the feature extraction model and the task execution model based on the one or more losses. [Explanation of symbols]

[0088] 10 Training data 20 training images 22 Object 30 First angle information 32 1st angle of incidence 34 1st azimuth 40 Correct data 50 model sets 52 Feature Extraction Model 54 Task Execution Model 70 sensors 75 Radar 80 First feature set 90 Second angle information 92 2nd angle of incidence 94 2nd azimuth 100 Second feature set 130 First coordinate system 132 1st azimuth axis 134 First Range Axis 136 1st incident axis 140 World Coordinate System 146 Vertical Upward Axis 150 Second coordinate system 152 2nd azimuth axis 154 Second Range Axis 156 Second incident axis 160 lines 200 images 210 sub-areas 220 characteristic rectangular parallelepiped 230 Sequence 240 characteristic rectangular parallelepipeds 250 feature modified model 260 characteristic rectangular parallelepipeds 270 characteristic rectangular parallelepipeds 280 characteristic rectangular parallelepipeds 290 characteristic rectangular parallelepipeds 300 conversion model 310 Embedding Layer 320 coding layer 330 characteristic rectangular parallelepiped 340 conversion model 350 Embedding Layer 360 coding layer 370 characteristic cuboid 1000 computers 1020 Bus 1040 processor 1060 memory 1080 storage device 1100 Input / Output Interface 1120 Network Interface 2000 training equipment 2020 Training Data Acquisition Department 2040 Angle information acquisition unit 2060 Feature Acquisition Unit 2080 conversion unit 2100 Update Department

Claims

1. 1. A training device comprising: at least one memory configured to store instructions; At least one processor executing the instructions to: acquiring training data including a training image, first angle information, and ground truth data, the training image being an image of an object captured by a sensor, the first angle information indicating a first incident angle that is an incident angle of the sensor and a first azimuth angle that is an azimuth angle of the object captured in the training image; inputting the training images into a feature extraction model to obtain a first feature set, the first feature set being a set of features extracted from the training images; acquiring second angle information indicative of a second angle of incidence and a second azimuth angle, wherein the second angle of incidence, the second azimuth angle, or both, are different from their corresponding angles in the first angle information; generating a second feature set by performing a coordinate transformation on the first feature set based on the first angle information and the second angle information; updating the feature extraction model based on the first feature set, the second feature set, and the ground truth data; and at least one processor configured to perform the steps of:

2. the first feature set is represented by a set of cells, each cell having a feature value and a coordinate in a first coordinate system defined using the first angle of incidence; the second feature set is represented by a set of cells, each cell having a feature value and a coordinate in a second coordinate system defined using the second incidence angle, the first azimuth angle, and the second azimuth angle; Performing the coordinate transformation on the first feature set includes: For each cell of the first feature set, performing a coordinate transformation from the first coordinate system to the second coordinate system on the coordinates of the cell of the first feature set to calculate a corresponding cell of the second feature set; and setting the value of the cell of the first feature set to the corresponding cell of the second feature set.

3. The coordinate transformation from the first coordinate system to the second coordinate system is a transformation from the first coordinate system to a world coordinate system defined by a horizontal plane and a vertically upward axis representing a direction opposite to the direction of gravity; rotating the world coordinate system about the vertically upward axis by a rotation angle, the rotation angle being a difference between the first azimuth angle and the second azimuth angle; The training device of claim 2 , further comprising a transformation from the world coordinate system rotated by the rotation angle to the second coordinate system.

4. the first feature set is represented by a first set of cells, each cell having a feature value and a coordinate in a first coordinate system defined using the first angle of incidence; the second feature set is represented by a second set of cells, each cell having a feature value and a coordinate in a second coordinate system defined using the second incidence angle, the first azimuth angle, and the second azimuth angle; Generating the second feature set includes: performing the coordinate transformation on the first set of features to transform the first set of features into a third set of cells in the second coordinate system; and modifying values ​​of one or more cells of the third set to generate the second feature set.

5. Generating the second feature set includes: calculating a difference characteristic between the first angle information and the second angle information; and modifying the value of one or more cells of the third set using the difference characteristic between the first angle information and the second angle information.

6. The training device of claim 1 , wherein the training images are radar images generated by a radar.

7. 7. The training device of claim 6, wherein the first feature set represents, for each subregion on the training image, backscattering characteristics at each of two or more points projected onto the subregion along a line forming the first angle of incidence from the image plane of the training image.

8. Updating the feature extraction model includes: inputting the first feature set into a task execution model to obtain a first result of the task; inputting the second set of features into the task execution model to obtain a second result of the task; calculating one or more losses based on the first result of the task, the second result of the task, and the ground truth data; and updating trainable parameters of the feature extraction model and the task performance model based on the one or more losses.

9. 1. A computer-implemented training method comprising: acquiring training data including a training image, first angle information, and ground truth data, the training image being an image of an object captured by a sensor, the first angle information indicating a first incident angle that is an incident angle of the sensor and a first azimuth angle that is an azimuth angle of the object captured in the training image; inputting the training images into a feature extraction model to obtain a first feature set, the first feature set being a set of features extracted from the training images; acquiring second angle information indicative of a second angle of incidence and a second azimuth angle, wherein the second angle of incidence, the second azimuth angle, or both, are different from their corresponding angles in the first angle information; generating a second feature set by performing a coordinate transformation on the first feature set based on the first angle information and the second angle information; updating the feature extraction model based on the first feature set, the second feature set, and the ground truth data.

10. the first feature set is represented by a set of cells, each cell having a feature value and a coordinate in a first coordinate system defined using the first angle of incidence; the second feature set is represented by a set of cells, each cell having a feature value and a coordinate in a second coordinate system defined using the second incidence angle, the first azimuth angle, and the second azimuth angle; Performing the coordinate transformation on the first feature set includes: For each cell of the first feature set, performing a coordinate transformation from the first coordinate system to the second coordinate system on the coordinates of the cell of the first feature set to calculate a corresponding cell of the second feature set; and setting the value of the cell of the first feature set to the corresponding cell of the second feature set.

11. The coordinate transformation from the first coordinate system to the second coordinate system is a transformation from the first coordinate system to a world coordinate system defined by a horizontal plane and a vertically upward axis representing a direction opposite to the direction of gravity; rotating the world coordinate system about the vertically upward axis by a rotation angle, the rotation angle being a difference between the first azimuth angle and the second azimuth angle; The training method of claim 10 , comprising transforming the world coordinate system rotated by the rotation angle to the second coordinate system.

12. the first feature set is represented by a first set of cells, each cell having a feature value and a coordinate in a first coordinate system defined using the first angle of incidence; the second feature set is represented by a second set of cells, each cell having a feature value and a coordinate in a second coordinate system defined using the second incidence angle, the first azimuth angle, and the second azimuth angle; Generating the second feature set includes: performing the coordinate transformation on the first set of features to transform the first set of features into a third set of cells in the second coordinate system; and modifying values ​​of one or more cells of the third set to generate the second feature set.

13. Generating the second feature set includes: calculating a difference characteristic between the first angle information and the second angle information; and modifying the values ​​of one or more cells of the third set using the difference feature between the first angle information and the second angle information.

14. The training method of claim 9 , wherein the training images are radar images generated by a radar.

15. 15. The training method of claim 14, wherein the first feature set, for each subregion on the training image, represents backscattering characteristics at each of two or more points projected onto the subregion along a line forming the first angle of incidence from the image plane of the training image.

16. Updating the feature extraction model includes: inputting the first feature set into a task execution model to obtain a first result of the task; inputting the second set of features into the task execution model to obtain a second result of the task; calculating one or more losses based on the first result of the task, the second result of the task, and the ground truth data; and updating trainable parameters of the feature extraction model and the task performance model based on the one or more losses.

17. A non-transitory computer-readable storage medium storing a program for a computer, the program including: acquiring training data including a training image, first angle information, and ground truth data, the training image being an image of an object captured by a sensor, the first angle information indicating a first incident angle that is an incident angle of the sensor and a first azimuth angle that is an azimuth angle of the object captured in the training image; inputting the training images into a feature extraction model to obtain a first feature set, the first feature set being a set of features extracted from the training images; acquiring second angle information indicative of a second angle of incidence and a second azimuth angle, wherein the second angle of incidence, the second azimuth angle, or both, are different from their corresponding angles in the first angle information; generating a second feature set by performing a coordinate transformation on the first feature set based on the first angle information and the second angle information; updating the feature extraction model based on the first feature set, the second feature set, and the ground truth data; A non-transitory computer-readable storage medium that causes the

18. the first feature set is represented by a set of cells, each cell having a feature value and a coordinate in a first coordinate system defined using the first angle of incidence; the second feature set is represented by a set of cells, each cell having a feature value and a coordinate in a second coordinate system defined using the second incidence angle, the first azimuth angle, and the second azimuth angle; Performing the coordinate transformation on the first feature set includes: For each cell of the first feature set, performing a coordinate transformation from the first coordinate system to the second coordinate system on the coordinates of the cell of the first feature set to calculate a corresponding cell of the second feature set; and setting the value of the cell of the first feature set to the corresponding cell of the second feature set.

19. The coordinate transformation from the first coordinate system to the second coordinate system is a transformation from the first coordinate system to a world coordinate system defined by a horizontal plane and a vertically upward axis representing a direction opposite to the direction of gravity; rotating the world coordinate system about the vertically upward axis by a rotation angle, the rotation angle being a difference between the first azimuth angle and the second azimuth angle; 20. The storage medium of claim 18, comprising a transformation from the world coordinate system to the second coordinate system rotated by the rotation angle.

20. the first feature set is represented by a first set of cells, each cell having a feature value and a coordinate in a first coordinate system defined using the first angle of incidence; the second feature set is represented by a second set of cells, each cell having a feature value and a coordinate in a second coordinate system defined using the second incidence angle, the first azimuth angle, and the second azimuth angle; Generating the second feature set includes: performing the coordinate transformation on the first set of features to transform the first set of features into a third set of cells in the second coordinate system; and modifying values ​​of one or more cells of the third set to generate the second set of features.

21. Generating the second feature set includes: calculating a difference characteristic between the first angle information and the second angle information; and modifying the value of one or more cells of the third set using the difference characteristic between the first angle information and the second angle information.

22. The storage medium of claim 17 , wherein the training images are radar images generated by a radar.

23. 23. The storage medium of claim 22, wherein the first feature set represents, for each subregion on the training image, backscattering characteristics at each of two or more points projected onto the subregion along a line forming the first angle of incidence from the image plane of the training image.

24. Updating the feature extraction model includes: inputting the first feature set into a task execution model to obtain a first result of the task; inputting the second set of features into the task execution model to obtain a second result of the task; calculating one or more losses based on the first result of the task, the second result of the task, and the ground truth data; and updating trainable parameters of the feature extraction model and the task execution model based on the one or more losses.

Images