Training apparatus, training method, and program

Abstract

A training device (2000) acquires training data (10) including training images (20), first angle information (30), and ground truth data (40). The first angle information (30) indicates a first angle of incidence (32) and a first azimuth angle (34). The training device (2000) inputs the training images (20) into a feature extraction model (52) to acquire a first feature set (80). The training device (2000) acquires second angle information (90), performs coordinate transformation on the first feature set (80) based on the first angle information (30) and the second angle information (90) to generate a second feature set (100), and updates the feature extraction model (52) based on the first feature set (80), the second feature set (100), and the ground truth data (40).

Description

Technical Field

[0001] The present disclosure generally relates to a training device, a training method, and a non - transient computer - readable storage medium.

Background Art

[0002] There is a technology for analyzing an image using a model that extracts features from an image, for example, object classification using a neural network. Patent Document 1 discloses a system including a convolutional neural network (CNN) unit configured to input an image generated by a synthetic aperture radar and classify an object imaged in the input image. This system includes a function of increasing data used for training the CNN unit. Specifically, this system acquires training data including training images and correct - answer data, and generates another image by changing the position, orientation, or both of the object imaged on the training image. Then, both the training image and the image generated by the system are used to train the CNN unit.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Increasing data used for training a model that handles images by generating another image based on a given image is the only method disclosed by Patent Document 1. The object of the present disclosure is to provide a novel technique for training a model that handles images.

Means for Solving the Problems

[0005] This disclosure provides a training device comprising at least one memory configured to store instructions and at least one processor. At least one processor is configured to acquire training data including a training image, first angle information, and ground truth data, wherein the training image is an image of an object captured and generated by a sensor, the first angle information indicates a first incident angle which is the incident angle of the sensor and a first azimuth angle which is the azimuth angle of the object captured in the training image, input the training image into a feature extraction model to acquire a first feature set which is a set of features extracted from the training image, acquire second angle information indicating a second incident angle and a second azimuth angle, generate a second feature set by performing a coordinate transformation on the first feature set based on the first angle information and the second angle information if the second incident angle, the second azimuth angle, or both are different from their correspondings in the first angle information, and update the feature extraction model based on the first feature set, the second feature set, and the ground truth data.

[0006] This disclosure further provides a computer-based training method comprising: acquiring training data including a training image, first angle information, and ground truth data, wherein the training image is an image of an object captured and generated by a sensor, the first angle information indicates a first incident angle which is the incident angle of the sensor and a first azimuth angle which is the azimuth angle of the object captured in the training image, inputting the training image into a feature extraction model to acquire a first feature set which is a set of features extracted from the training image, acquiring second angle information indicating a second incident angle and a second azimuth angle, generating a second feature set by performing a coordinate transformation on the first feature set based on the first angle information and the second angle information, provided that the second incident angle, the second azimuth angle, or both are different from their correspondings in the first angle information, and updating the feature extraction model based on the first feature set, the second feature set, and the ground truth data.

[0007] This disclosure further provides a non-temporary computer-readable storage medium for storing programs. The program instructs the computer to acquire training data including a training image, first angle information, and ground truth data, wherein the training image is an image of an object captured by a sensor and the first angle information indicates the first incident angle, which is the incident angle of the sensor, and the first azimuth angle, which is the azimuth angle of the object captured in the training image, inputs the training image into a feature extraction model to acquire a first feature set, which is a set of features extracted from the training image, acquires second angle information indicating the second incident angle and the second azimuth angle, generates a second feature set by performing a coordinate transformation on the first feature set based on the first and second angle information, and updates the feature extraction model based on the first feature set, the second feature set, and the ground truth data, provided that the second incident angle, the second azimuth angle, or both are different from their correspondings in the first angle information. [Effects of the Invention]

[0008] This disclosure provides a novel technique for training models that handle images. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram shows an overview of the training device according to Embodiment 1. [Figure 2] This figure shows an example of training data. [Figure 3] This is a block diagram showing an example of the functional configuration of the training device of Embodiment 1. [Figure 4] This block diagram shows an example of a computer hardware configuration for realizing the training device of Embodiment 1. [Figure 5] A flowchart illustrating an exemplary flow of processing performed by the training device of Embodiment 1 is shown. [Figure 6] This shows the feature extraction performed by the feature extraction model. [Figure 7]This shows how the angle of incidence affects how objects appear in radar images. [Figure 8] This shows the coordinate transformation from the first coordinate system to the second coordinate system. [Figure 9] Here is another example of the conversion from the first feature set 80 to the second feature set 100. [Figure 10] This section demonstrates feature correction using the first and second angle information. [Figure 11] This document presents an example of a method for extracting features from a set of first and second angle information. [Modes for carrying out the invention]

[0010] Embodiments of the present disclosure will be described below with reference to the drawings. The same elements are assigned the same reference numerals throughout the drawings, and redundant descriptions will be omitted where necessary. In addition, predetermined information (e.g., predetermined values ​​or predetermined thresholds) is pre-stored in a memory unit accessed by the computer using that information, unless otherwise described. In the present disclosure, the memory unit may be implemented as one or more storage devices such as a hard disk, a solid-state drive (SSD), or random-access memory (RAM).

[0011] Embodiment 1 <Overview> Figure 1 shows an overview of the training device 2000 of Embodiment 1. Please note that Figure 1 does not limit the operation of the training device 2000, but merely shows an example of the possible operation of the training device 2000.

[0012] The training device 2000 is a device configured to acquire training data 10 and train a model set 50 using the training data 10. The model set 50 includes a feature extraction model 52 and a task execution model 54. The feature extraction model 52 and the task execution model 54 may be machine learning-based models such as neural networks.

[0013] The feature extraction model 52 is configured to take an image as input, extract features from the input image, and output the extracted features. The task execution model 54 is configured to take features as input, execute a task on the input features, and output the result of the task. Examples of tasks performed by the task execution model 54 include object detection, object classification, semantic segmentation, image reconstruction, and the like.

[0014] The training data 10 includes training images 20, first angle information 30, and correct answer data 40. FIG. 2 shows an example of the training data 10. The training image 20 is an image including an object 22 generated by a sensor 70. The training image 20 may be an optical image or a radar image.

[0015] When the training image 20 is an optical image, the sensor 70 is an optical camera configured to receive light and generate an optical image based on the received light. When the training image 20 is a radar image, the sensor 70 is a radar configured to transmit radio waves, receive reflections of the radio waves, and generate a radar image based on the received reflections of the radio waves. The sensor 70 may be installed on a satellite to capture objects such as the Earth, other planets, and satellites. An example of the radar is a Synthetic-Aperture Radar.

[0016] The first angle information 30 indicates a first incident angle 32 and a first azimuth angle 34. The first incident angle 32 represents the incident angle of the sensor 70 when the sensor 70 captures the object 22 to generate the training image 20. The first azimuth angle 34 is the azimuth angle of the object 22 when the sensor 70 captures the object 22 to generate the training image 20.

[0017] The correct answer data 40 is data indicating the correct answer for training the model set 50. Assume that the model set 50 performs object classification on an image. In FIG. 2, since the object 22 is a ship, the correct answer data 40 indicates the class of "ship".

[0018] To train the model set 50, the training device 2000 may operate as follows: The training device 2000 acquires training data 10 and inputs the training images 20 from the acquired training data 10 into the feature extraction model 52. As a result, the training device 2000 acquires a first feature set 80, which is a set of features extracted from the training images 20 by the feature extraction model 52.

[0019] The training device 2000 generates another feature set called the "second feature set 100" from the first feature set 80 in order to train the model set 50. To do this, the training device 2000 further acquires second angular information 90 indicating the second incidence angle 92 and the second azimuth angle 94. The second incidence angle 92 is not equal to the first incidence angle 32, or the second azimuth angle 94 is not equal to the first azimuth angle 34, or both.

[0020] The training device 2000 performs a coordinate transformation on the first feature set 80 based on the first angle information 30 and the second angle information 90, thereby transforming the first feature set 80 into the second feature set 100. As a result, the second feature set 100 is generated to represent the image features of the second incidence angle 92 and the second azimuth angle 94. Specifically, the second feature set 100 represents the features of an image in which an object 22 is captured by a sensor 70 having an incidence angle equal to the second incidence angle 92 and having an azimuth angle equal to the second azimuth angle 94.

[0021] The training device 2000 trains the model set 50 using the first feature set 80, the second feature set 100, and the ground truth data 40. Details of the training of the model set 50 will be described later.

[0022] <Examples of effects and benefits> To train the model set 50, it is preferable to use multiple images with various pairs of incidence and azimuth angles. In particular, when images are generated by radar, as will be detailed later, the incidence angle of the radar and the azimuth angle of the captured object can affect how the object appears in the image due to the nature of radar imaging physics. However, there are some situations in which it is difficult to prepare a sufficient number of images for training the model.

[0023] The training device 2000 provides a novel technique for training a model that handles images. Specifically, a first feature set 80 is extracted from the training image 20, and a second feature set 100 is generated by performing a coordinate transformation on the first feature set 80. Next, the model set 50 is trained using both the first feature set 80 and the second feature set 100.

[0024] The second feature set 100 represents the features of an image in which an object 22 is captured by the sensor 70 at a specific incidence angle and at a specific azimuth angle. By generating the second feature set 100 from the first feature set 80, the training device 2000 can obtain the features of an image without actually obtaining another image. Therefore, the training device 2000 can increase the number of image feature sets used to train the model set 50, thereby facilitating the collection of training data for training the model set 50. Furthermore, the training device 2000 can easily improve the accuracy of the model set 50.

[0025] The training device 2000 will be described in more detail below.

[0026] <Example of functional configuration> Figure 3 is a block diagram showing an example of the functional configuration of the training device 2000 of Embodiment 1. The training device 2000 includes a training data acquisition unit 2020, an angle information acquisition unit 2040, a feature acquisition unit 2060, a conversion unit 2080, and an update unit 2100.

[0027] The training data acquisition unit 2020 acquires training data 10. The angle information acquisition unit 2040 acquires second angle information 90. The feature acquisition unit 2060 inputs the training image 20 to the feature extraction model 52 and acquires the first feature set 80 extracted by the feature extraction model 52 from the training image 20. The transformation unit 2080 performs a coordinate transformation on the first feature set 80 based on the first angle information 30 and the second angle information 90, thereby transforming the first feature set 80 into the second feature set 100. The update unit 2100 updates the model set 50 using the first feature set 80, the second feature set 100, and the ground truth data 40.

[0028] <Example hardware configuration> The training device 2000 may be implemented by one or more computers. Each of the one or more computers may be a dedicated computer manufactured to implement the training device 2000, or it may be a general-purpose computer such as a personal computer (PC), server machine, or mobile device.

[0029] The training device 2000 may be implemented by installing an application on a computer. The application is implemented by a program that makes the computer function as the training device 2000. In other words, the program is an implementation of the functional part of the training device 2000. There are various ways to obtain the program. For example, the program can be obtained from a storage medium (e.g., a DVD disc or USB memory) on which the program is pre-stored. In another example, the program can be obtained by downloading it from a server machine that manages the storage medium on which the program is pre-stored.

[0030] Figure 4 is a block diagram showing an example of the hardware configuration of a computer 1000 that implements the training device 2000 of Embodiment 1. In Figure 4, the computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output (I / O) interface 1100, and a network interface 1120.

[0031] Bus 1020 is a data transmission channel for the processor 1040, memory 1060, storage device 1080, input / output interface 1100, and network interface 1120 to send and receive data to and from each other. The processor 1040 is a processor such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), or DSP (Digital Signal Processor). Memory 1060 is a main memory element such as RAM (Random Access Memory) or ROM (Read Only Memory). Storage device 1080 is an auxiliary storage element such as a hard disk, SSD (Solid State Drive), or memory card. The input / output interface 1100 is an interface between the computer 1000 and peripheral devices such as a keyboard, mouse, or display device. The network interface 1120 is an interface between the computer 1000 and a network. The network may be a LAN (Local Area Network) or a WAN (Wide Area Network). Storage device 1080 may store the aforementioned program. The processor 1040 executes a program to implement each functional part of the training device 2000.

[0032] The hardware configuration of computer 1000 is not limited to that shown in Figure 4. For example, as mentioned above, the training device 2000 may be implemented by multiple computers. In this case, these computers may be connected to each other via a network.

[0033] <Processing flow> Figure 5 shows a flowchart illustrating an exemplary flow of processing performed by the training device 2000 of Embodiment 1. The training data acquisition unit 2020 acquires training data 10 (S102). The angle information acquisition unit 2040 acquires second angle information 90 (S104). The feature acquisition unit 2060 inputs the training image 20 to the feature extraction model 52 and acquires the first feature set 80 (S106). The transformation unit 2080 performs a coordinate transformation on the first feature set 80 based on the first angle information 30 and the second angle information 90, and transforms the first feature set 80 into the second feature set 100 (S108). The update unit 2100 updates the model set 50 using the first feature set 80, the second feature set 100, and the ground truth data 40 (S110).

[0034] Note that Figure 5 is merely an example of a possible processing flow performed by the training device 2000, and the processing flow performed by the training device 2000 is not limited to that shown in Figure 5. For example, the acquisition of the second angle information 90 (S104) may be performed at any timing prior to the coordinate transformation (S108).

[0035] <Acquisition of training data 10: S102> The training data acquisition unit 2020 acquires the training data 10 (S102). There are various methods for acquiring the training data 10. In some implementations, the training data acquisition unit 2020 can receive the training data 10 transmitted from another computer, such as a computer that generates the training data 10. In other implementations, the training data may be stored in advance in a storage unit accessed by the training data acquisition unit 2020. In this case, the training data acquisition unit 2020 reads the training data 10 from this storage unit.

[0036] The training data acquisition unit 2020 may acquire two or more training data sets 10. In this case, the training device 2000 may use each of the data sets to train the model set 50.

[0037] There can be various methods for determining the number of training data 10 to be acquired. For example, the number of training data 10 to be acquired may be predetermined, determined randomly by the training data acquisition unit 2020, or specified by the user of the training device 2000. In another example, the training data acquisition unit 2020 may acquire all of the prepared training data 10 (for example, all of the training data 10 stored in the memory device).

[0038] <Acquisition of second angle information 90: S104> The angle information acquisition unit 2040 acquires the second angle information 90 (S104). Note that the number of second angle information 90 acquired by the angle information acquisition unit 2040 is not limited to one. If the angle information acquisition unit 2040 acquires two or more second angle information 90, the conversion unit 2080 may generate a second feature set 100 for each second angle information 90.

[0039] There can be various methods for determining the number of second angle information 90 to be acquired. For example, the number of second angle information 90 to be acquired may be predetermined, determined randomly by the angle information acquisition unit 2040, or specified by the user of the training device 2000. In another example, the angle information acquisition unit 2040 may acquire all of the prepared second angle information 90 (for example, all of the second angle information 90 stored in the memory device).

[0040] The second angle information 90 may be prepared in advance, or it may be dynamically generated by the angle information acquisition unit 2040. In the former case, various pairs of incident angle and azimuth angle may be pre-stored as candidates for the second angle information 90 in a memory device accessed by the training device 2000. The angle information acquisition unit 2040 may acquire the second angle information 90 from this memory device by selecting one of the candidates for the second angle information 90, which is a second incident angle 92, a second azimuth angle 94, or both of which are not equivalent to the corresponding ones in the first angle information 30. The candidates for the second angle information 90 may be selected randomly or based on a specific rule.

[0041] If the second angle information 90 is generated dynamically, the angle information acquisition unit 2040 may generate the second angle information 90 by randomly determining the second incidence angle 92 and the second azimuth angle 94. If the second incidence angle 92 and the second azimuth angle 94 are equal to the first incidence angle 32 and the first azimuth angle 34, respectively, the training device 2000 may randomly determine the second incidence angle 92, the second azimuth angle 94, or both again, so that the second angle information 90 is not equal to the first angle information 30.

[0042] <Get Feature Set 80: S106> The feature acquisition unit 2060 inputs the training image 20 to the feature extraction model 52 to acquire the first feature set 80 (S106). The feature extraction model 52 is configured to extract features from the image input thereto and output the extracted features. Thus, when the feature acquisition unit 2060 inputs the training image 20 to the feature extraction model 52, the feature extraction model 52 extracts features from the training image 20 and outputs the features extracted from the training image 20. The feature acquisition unit 2060 acquires the features of the training image 20 output by the feature extraction model 52 as the first feature set 80.

[0043] The following provides a more detailed explanation of the feature extraction model 52.

[0044] The feature extraction model 52 is configured to extract three-dimensional spatial features of a scene captured on an input image. Figure 6 shows the feature extraction performed by the feature extraction model 52. The feature extraction model 52 can be configured as a neural network such as a convolutional neural network (CNN) having multiple filters for extracting multiple local spatial features for each sub-region 210 of the input image 200.

[0045] A feature extraction model 52 can be trained to generate a set of features of an input image 200, which may be represented by a set of cells having feature vectors (i.e., feature values) and coordinates in a particular coordinate system. In this disclosure, the term “cell” is used to describe a pair of values ​​and coordinates. Feature vectors corresponding to particular coordinates represent spatial features of a three-dimensional sub-region of the scene corresponding to those coordinates. A set of cells may be represented by a cuboid of cells, each showing a feature vector corresponding to the coordinates of the cell. Hereinafter, this cuboid of cells will be referred to as a “feature cuboid”.

[0046] In the following, unless otherwise specified, the set of features extracted by the feature extraction model 52 will be described as a feature cuboid. However, the techniques described herein are also applicable when the set of features extracted by the feature extraction model 52 is represented in a form other than a cuboid (e.g., a list of cells).

[0047] The feature cuboid 220 generated by the feature extraction model 52 is a cuboid in a first coordinate system 130 defined by a first azimuthal axis 132, a first range axis 134, and a first incidence axis 136. The first azimuthal axis 132 lies on the horizontal plane and represents a reference direction (e.g., east). The first range axis 134 lies on the horizontal plane and is perpendicular to the first azimuthal axis 132. The first incidence axis 136 is the axis that forms the incidence angle of the input image from the direction opposite to the direction of gravity (vertically upward). As shown in Figure 6, features of a subregion 210 of the input image 200 can be extracted as a sequence of cells in the feature cuboid 220 along the first incidence axis 136. Hereinafter, this sequence of cells will be referred to as the "cell sequence 230".

[0048] When the training image 20 is input to the feature extraction model 52, the angle of incidence of the input image becomes the first angle of incidence 32. Therefore, the first coordinate system corresponding to the first feature set 80 can be defined by the first angle of incidence 32.

[0049] The first feature set 80 will be further explained below from the perspective of the properties of radar imaging physics. As mentioned above, according to the properties of radar imaging physics, the radar incidence angle and the object's azimuth angle affect how the object appears in the image. Figure 7 shows how the incidence angle affects how an object appears in a radar image.

[0050] The incident angle of radar 75 differs between the example on the left and the example on the right of Figure 7. In the example on the left, there is a line 160-1 that passes through sub-region 210 and forms an incident angle T1 from the horizontal plane. Therefore, line 160-1 passes through the three-dimensional space of the real world projected onto sub-region 210 on the image plane of image 200.

[0051] On the other hand, in the example on the right, there is a line 160-2 that passes through sub-region 210 and forms an incident angle T2 from the horizontal plane. Therefore, line 160-2 passes through the real-world three-dimensional space projected onto sub-region 210 on the image plane of image 200.

[0052] Due to the properties of radar imaging physics, the intensity of sub-region 210 can be calculated as the sum of backscatter from points along line 160. Therefore, in the example on the left, the intensity of sub-region 210 is the sum of backscatter from p1 to pn. Similarly, in the example on the right, the intensity of sub-region 210 is the sum of backscatter from q1 to qn. This means that the intensity of sub-region 210 on image 200 depends on the incident angle of radar 75.

[0053] Furthermore, when the azimuth angle of object 22 changes, the point of object 22 along line 160 changes. Therefore, it can be said that the azimuth angle of object 22 also affects the intensity of sub-region 210 on image 200 due to the properties of radar imaging physics.

[0054] As described with reference to Figure 6, the feature of subregion 210 of image 200 is the sequence of cells along the first incidence axis 136. cell This can be extracted as sequence 230. The direction represented by the first incidence axis 136 corresponds to the direction of line 160. Therefore, the feature extraction model 52 represents the features of points along line 160 passing through each sub-region 210. cell The model can be trained to generate a feature cuboid 220 containing sequence 230.

[0055] <Coordinate transformation: S108> The transformation unit 2080 performs a coordinate transformation on the first feature set 80 based on the first angle information 30 and the second angle information 90 to generate the second feature set 100 (S108). The coordinate transformation performed by the transformation unit 2080 is a coordinate transformation from the first coordinate system 130 defined by the first angle information 30 to the second coordinate system defined by the second angle information 90.

[0056] Figure 8 shows the coordinate transformation from the first coordinate system 130 to the second coordinate system 150. This coordinate transformation can be decomposed into the first to third coordinate transformations.

[0057] The first coordinate transformation M1 is a coordinate transformation from the first coordinate system 130 to the world coordinate system 140. The world coordinate system 140 is a real-world coordinate system defined by the first azimuthal axis 132, the first range axis 134, and the vertical upward axis 146. The vertical upward axis 146 is the axis that represents the direction opposite to the direction of gravity.

[0058] The second coordinate transformation M2 involves rotating the world coordinate system 140 by an angle defined by the difference between the second azimuth angle 94 and the first azimuth angle 34. The second coordinate transformation rotates the first azimuth axis 132 and the first range axis 134 around the vertically upward axis, thereby obtaining the second azimuth axis 152 and the second range axis 154. Let the first azimuth angle 34 and the second azimuth angle 94 be S1 and S2, respectively. In this case, the second azimuth axis 152 and the second range axis 154 are obtained by rotating the first azimuth axis 132 and the first range axis 134 by S2-S1 around the vertically upward axis.

[0059] The third coordinate transformation M3 is a coordinate transformation from the world coordinate system 140, which has been rotated by the rotation angle, to the second coordinate system 150. The second coordinate system 150 is a coordinate system defined by the second azimuthal axis 152, the second range axis 154, and the second incidence axis 156. The second incidence axis 156 is the axis that forms the second incidence angle 92 from the vertical upward axis 146.

[0060] In Figure 8, the first coordinate transformation, the second coordinate transformation, and the third coordinate transformation are represented by transformation matrices M1, M2, and M3, respectively. Under this assumption, the coordinate transformation from the first coordinate system 130 to the second coordinate system 150 can be expressed as follows. formula 1

number

[0061] The transformation unit 2080 determines the combined transformation matrix Mc by determining the transformation matrices M1, M2, and M3. Note that there are known methods for calculating transformation matrices between two coordinate systems, and one of these methods can be applied to the transformation unit 2080 to determine the transformation matrices M1, M2, and M3.

[0062] As described above, the first feature set 80 can be represented by a rectangular prism of cells in the first coordinate system 130. The transformation unit 2080 uses the transformation matrix Mc to transform the rectangular prism of cells of the first feature set 80, thereby obtaining the rectangular prism of cells in the second coordinate system 150 as the second feature set 100.

[0063] Specifically, the transformation unit 2080 may use the transformation matrix Mc to transform the coordinates of each cell in the first feature set 80 in the first coordinate system 130 to the coordinates in the second coordinate system 150. This allows the transformation unit 2080 to identify the cells in the second feature set 100 that correspond to the cells in the first feature set 80. The transformation unit 2080 then sets the values ​​of the cells in the first feature set 80 to the corresponding cells in the second feature set 100.

[0064] The coordinate transformation by the combined transformation matrix Mc transforms (x1, y1, z1) in the first coordinate system 130 to (x2, y2, z2) in the second coordinate system 150. In this case, the transformation unit 2080 may set the value of the (x1, y1, z1) cell in the first feature set 80 to the (x2, y2, z2) cell in the second feature set 100.

[0065] In another example, the transformation unit 2080 may calculate the inverse of the concatenation matrix Mc, represented by Mc⁻¹, in order to compute the second feature set 100. In this case, the transformation unit 2080 converts the coordinates of each cell in the second feature set 100 in the second coordinate system 150 to the coordinates in the first coordinate system 130. This allows the transformation unit 2080 to identify the cells in the first feature set 80 that correspond to the cells in the second feature set 100. The transformation unit 2080 then sets the values ​​of the cells in the first feature set 80 to the corresponding cells in the second feature set 100.

[0066] <<Feature modification using trainable models>> The transformation unit 2080 may further perform feature modification using a trainable model called a "feature modification model" after the coordinate transformation described above. Figure 9 shows another example of the transformation from the first feature set 80 to the second feature set 100. The transformation unit 2080 first, 1 A coordinate transformation is performed on the feature cuboid 220 obtained as feature set 80, thereby obtaining feature cuboid 240. Next, the transformation unit 2080 inputs feature cuboid 240 to the feature modification model 250.

[0067] The feature modification model 250 is configured to take the feature cuboid 240 as input, modify the values ​​of the cells in the feature cuboid 240, and output the feature cuboid 260. The conversion unit 2080 outputs the feature cuboid 260 as the second feature set 100.

[0068] The feature modification model 250 can be implemented as a machine learning-based model such as a neural network. Various methods exist for modifying the feature cuboid 240 to generate the second feature set 100. For example, the feature modification model 250 can be configured to calculate a weighted sum for each cell in the feature cuboid 240, based on the value of that cell and the values ​​of surrounding (e.g., adjacent) cells. The weighted sums calculated for the cells of the feature cuboid 240 are then set for the corresponding cells in the feature cuboid 260. In this case, the weights are the parameters to be trained.

[0069] In another example, the first angle information 30 and the second angle information 90 are also used for feature modification. Figure 10 shows feature modification using the first angle information 30 and the second angle information 90. Conversion unit 20 8 0 can generate a feature cuboid 270 by calculating the difference feature between the first angle information 30 and the second angle information 90. (Transformation unit 20) 8 0 concatenates feature cuboid 270 with feature cuboid 240 to obtain feature cuboid 280. As shown in Figure 10, feature cuboid 270 is configured to have the same size as feature cuboid 240 along the second azimuthal axis 152 and the second range axis 154 so that feature cuboid 270 can be concatenated with feature cuboid 240.

[0070] Next, the conversion unit 20 8 0 obtains feature cuboid 290 as the second feature set 100 by inputting feature cuboid 280 into feature modification model 250. In this case, feature modification model 250 is configured to take feature cuboid 280 as input and output feature cuboid 290. In order to convert feature cuboid 280 to feature cuboid 290, feature modification model 250 is configured to use the cell values ​​of feature cuboid 270 to modify the cell values ​​of feature cuboid 240.

[0071] Various methods exist for extracting features from the set of first angle information 30 and second angle information 90. Figure 11 shows an example of a method for extracting features from the set of first angle information 30 and second angle information 90. Briefly explained, the conversion unit 20 8 0 calculates the difference in the angle of incidence (hereinafter referred to as the "angle of incidence difference") and the difference in the azimuth angle (hereinafter referred to as the "azimuth angle difference"). Next, the conversion unit 20 8 0 calculates the characteristics of the incident angle difference and the azimuth angle difference, and concatenates them to obtain the characteristic rectangular parallelepiped 270.

[0072] Below, we will first describe an example method for calculating the characteristics of the incident angle difference. Conversion unit 20 80 is obtained by calculating the incident angle difference (i.e., the difference between the first incident angle of 32 and the second incident angle of 92), quantizing the incident angle difference, and obtaining one of the predetermined integers.

[0073] In some embodiments, the entire range of incident angles (e.g., 360°) is divided into specific intervals to define predetermined integers. For example, if the entire range of incident angles is 360° and the division interval is 10°, the entire range of incident angles is divided into 36 bins. In this case, 1 to 36 are used as predetermined integers. Suppose the incident angle difference is 35°. In this case, since 35° belongs to the fourth bin, the incident angle difference is quantized to 4.

[0074] Conversion unit 20 8 0 inputs the quantized incident angle difference into the transformation model 300 to obtain a feature cuboid 330 that represents the features of the incident angle difference and whose size is the same as the feature cuboid 240 along the second azimuthal axis 152 and the second range axis 154. The transformation model 300 includes an embedding layer 310 and an encoding layer 320. The embedding layer 310 and the encoding layer 320 may be implemented as machine learning-based models such as neural networks and may therefore be trainable.

[0075] The embedding layer 310 takes the quantized incident angle difference as input and is configured to convert the input data into a random number that encodes the incident angle difference. The embedding layer 310 is trained to map each integer obtained by quantizing the incident angle difference to a specific random number. In other words, each bin of the quantized incident angle difference is associated with a specific random number through the training of the embedding layer 310.

[0076] The calculated random numbers are output as vectors and input to the coding layer 320. The coding layer 320 is configured to perform transpose convolution on the input vectors to generate the feature cuboid 330.

[0077] The characteristics of the azimuth angle difference are calculated using the same method as the characteristics of the incident angle difference. (Conversion unit 20) 80 calculates the azimuth angle difference (i.e., the difference between the first azimuth angle 34 and the second azimuth angle 94), and quantizes the azimuth angle difference to obtain one of a predetermined integer. Then the conversion unit 20 8 0 inputs the quantized azimuthal angle difference into the transformation model 340 to obtain a feature cuboid 370 that represents the features of the azimuthal angle difference and whose size is the same as that of the feature cuboid 240 along the second azimuthal axis 152 and the second range axis 154. The transformation model 340 includes an embedding layer 350 and an encoding layer 360. The embedding layer 350 and the encoding layer 360 may be implemented as machine learning-based models such as neural networks and may therefore be trainable.

[0078] The embedding layer 350 takes the quantized azimuth difference as input and is configured to convert the input data into random numbers that encode the azimuth difference. Each bin of the quantized azimuth difference is associated with a specific random number through training of the embedding layer 350. The computed random numbers are output as a vector and input to the encoding layer 360. The encoding layer 360 is configured to perform transpose convolution on the input vector to generate the feature cuboid 370.

[0079] After calculating the feature cuboid 330 and feature cuboid 370, the conversion unit 20 8 0 concatenates them to obtain a feature cuboid 270 that represents the features of the set of first angle information 30 and second angle information 90.

[0080] <Model Set 50 Update: S110> The update unit 2100 updates the model set 50 using the first feature set 80 and the second feature set 100 (S110). Specifically, the update unit 2100 inputs the first feature set 80 into the task execution model 54 and obtains the task results from the task execution model 54. The update unit 2100 then calculates the loss based on the ground truth data 40 and the results of the task performed using the first feature set 80. Similarly, the update unit 2100 inputs the second feature set 100 into the task execution model 54 and obtains the task results from the task execution model 54. The update unit 2100 then calculates the loss based on the ground truth data 40 and the results of the task performed using the second feature set 100. If the transformation unit 2080 generates two or more second feature sets 100, the update unit 2100 may calculate the loss for each of the second feature sets 100.

[0081] The calculated loss is used to train the model set 50. There are various methods for training the model based on the loss, and one of these methods can be applied to the update unit 2100. For example, the update unit 2100 can calculate a batch loss with the calculated loss (e.g., calculate the average of the calculated losses) and use the batch loss to update the trainable parameters of the model set 50. In another example, the update unit 2100 can use each of the calculated losses separately to update the trainable parameters of the model set 50.

[0082] Conversion unit 20 8 If 0 includes the feature modification model 250, the update unit 2100 can also use the calculated loss to update the trainable parameters of the feature modification model 250 (e.g., the weights for calculating the weighted sum described above). Similarly, the transformation unit 20 8 If 0 includes the transformation model 300 and the transformation model 340, the update unit 2100 may use the calculated loss to update the trainable parameters of the transformation model 300 and the trainable parameters of the transformation model 340. In another example, the feature modification model 250, the transformation model 300, and the transformation model 340 may be trained before the model set 50 is trained.

[0083] <Output from training device 2000> The training device 2000 can output the training results of the model set 50. The training results can be output in any way. For example, the training device 2000 can store the trained parameters of the model set 50 (e.g., the weights assigned to each connection in the neural network) in its memory. In another example, the training device 2000 can send the trained parameters to another device used to run the model set 50. In addition to parameters, the program that implements the model set 50 may also be output.

[0084] In the operational phase of model set 50, if the training device 2000 is also used to run model set 50, the training device 2000 does not need to output the training results. In this case, from the user's perspective, it is preferable for the training device 2000 to notify the user that training of model set 50 has finished.

[0085] Programs can be stored and provided to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs, CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, RAMs). Programs may also be provided to a computer using various types of transient computer-readable media. Examples of transient computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable media can be supplied to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0086] While the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be made within the scope of the present invention, as will be understood by those skilled in the art.

[0087] Some or all of the above embodiments may also be described as follows, but are not limited to the following: <Note> (Note 1) A training device, At least one memory configured to store instructions, At least one processor, which executes the instruction, Training data is acquired that includes a training image, first angle information, and ground truth data. The training image is an image of an object captured by a sensor and is generated by the sensor. The first angle information indicates the first incident angle, which is the incident angle of the sensor, and the first azimuth angle, which is the azimuth angle of the object captured in the training image. The aforementioned training images are input into the feature extraction model to obtain a first feature set, which is a set of features extracted from the training images. Second angular information indicating a second incidence angle and a second azimuth angle is obtained, and the second incidence angle, the second azimuth angle, or both are different from their corresponding ones in the first angular information. Based on the first and second angle information, a coordinate transformation is performed on the first feature set to generate a second feature set. Updating the feature extraction model based on the first feature set, the second feature set, and the ground truth data. A training device having at least one processor configured to perform the following. (Note 2) The first feature set is represented by a set of cells, each cell having a feature value and coordinates in a first coordinate system defined using the first angle of incidence. The second feature set is represented by a set of cells, each cell having a feature value and coordinate in a second coordinate system defined using the second incidence angle, the first azimuth angle, and the second azimuth angle. Performing the coordinate transformation on the first set of features means For each cell in the first feature set, a coordinate transformation is performed on the coordinates of the cell in the first feature set from the first coordinate system to the second coordinate system to calculate the corresponding cell in the second feature set. The training apparatus according to Appendix 1, comprising setting the value of the cell in the first feature set to the corresponding cell in the second feature set. (Note 3) The coordinate transformation from the first coordinate system to the second coordinate system is: This includes a transformation from the aforementioned first coordinate system to a world coordinate system defined by a horizontal plane and a vertically upward axis representing the direction opposite to the direction of gravity, This includes rotating the world coordinate system by a rotation angle around the vertically upward axis, wherein the rotation angle is the difference between the first azimuth angle and the second azimuth angle. The training apparatus according to Appendix 2, which includes a transformation from the world coordinate system rotated by the aforementioned rotation angle to the second coordinate system. (Note 4) The first feature set is represented by a first set of cells, each cell having feature values ​​and coordinates in a first coordinate system defined using the first incidence angle, The second set of features is represented by a second set of cells, each cell having feature values ​​and coordinates in a second coordinate system defined using the second angle of incidence, the first azimuth angle, and the second azimuth angle. Generating the aforementioned second set of features means Performing the coordinate transformation on the first feature set to convert the first feature set into a third set of cells in the second coordinate system, The training apparatus according to Appendix 1, comprising: modifying the values ​​of one or more cells in the third set to generate the second feature set. (Note 5) Generating the aforementioned second set of features means To calculate the characteristics of the difference between the first angle information and the second angle information, The training apparatus according to Appendix 4, comprising modifying the value of one or more cells of the third set using the difference characteristics between the first angle information and the second angle information. (Note 6) The training image is a radar image generated by radar, as described in Appendix 1 of the training apparatus. (Note 7) The training apparatus according to Appendix 6, wherein the first feature set represents, for each sub-region on the training image, the backscatter features at each of two or more points projected onto the sub-region along a line forming the first incidence angle from the image plane of the training image. (Note 8) Updating the aforementioned feature extraction model means The aforementioned first feature set is input into the task execution model to obtain the first result of the task, The second set of features is input into the task execution model to obtain the second result of the task, Calculating one or more losses based on the first result of the task, the second result of the task, and the ground truth data, The training apparatus according to Appendix 1, comprising updating the trainable parameters of the feature extraction model and the task execution model based on the one or more losses. (Note 9) A training method performed by a computer, Training data is acquired that includes a training image, first angle information, and ground truth data. The training image is an image of an object captured by a sensor and is generated by the sensor. The first angle information indicates the first incident angle, which is the incident angle of the sensor, and the first azimuth angle, which is the azimuth angle of the object captured in the training image. The aforementioned training images are input into the feature extraction model to obtain a first feature set, which is a set of features extracted from the training images. Second angular information indicating a second incidence angle and a second azimuth angle is obtained, and the second incidence angle, the second azimuth angle, or both are different from their corresponding ones in the first angular information. Based on the first and second angle information, a coordinate transformation is performed on the first feature set to generate a second feature set. A training method comprising updating the feature extraction model based on the first feature set, the second feature set, and the ground truth data. (Note 10) The first feature set is represented by a set of cells, each cell having a feature value and coordinates in a first coordinate system defined using the first angle of incidence. The second feature set is represented by a set of cells, each cell having a feature value and coordinate in a second coordinate system defined using the second incidence angle, the first azimuth angle, and the second azimuth angle. Performing the coordinate transformation on the first set of features means For each cell in the first feature set, a coordinate transformation is performed on the coordinates of the cell in the first feature set from the first coordinate system to the second coordinate system to calculate the corresponding cell in the second feature set. The training method according to Appendix 9, comprising setting the value of the cell in the first feature set to the corresponding cell in the second feature set. (Note 11) The coordinate transformation from the first coordinate system to the second coordinate system is: This includes a transformation from the aforementioned first coordinate system to a world coordinate system defined by a horizontal plane and a vertically upward axis representing the direction opposite to the direction of gravity, This includes rotating the world coordinate system by a rotation angle around the vertically upward axis, wherein the rotation angle is the difference between the first azimuth angle and the second azimuth angle. The training method described in Appendix 10, which includes a transformation from the world coordinate system rotated by the aforementioned rotation angle to the second coordinate system. (Note 12) The first feature set is represented by a first set of cells, each cell having feature values ​​and coordinates in a first coordinate system defined using the first incidence angle, The second set of features is represented by a second set of cells, each cell having feature values ​​and coordinates in a second coordinate system defined using the second angle of incidence, the first azimuth angle, and the second azimuth angle. Generating the aforementioned second set of features means Performing the coordinate transformation on the first feature set to convert the first feature set into a third set of cells in the second coordinate system, The training method according to Appendix 9, comprising: modifying the values ​​of one or more cells in the third set to generate the second feature set. (Note 13) Generating the aforementioned second set of features means To calculate the characteristics of the difference between the first angle information and the second angle information, The training method according to Appendix 12, comprising modifying the values ​​of one or more cells in the third set using the difference characteristics between the first angle information and the second angle information. (Note 14) The training method described in Appendix 9, wherein the training image is a radar image generated by radar. (Note 15) The training method according to Appendix 14, wherein the first feature set represents, for each sub-region on the training image, the backscatter features at each of two or more points projected onto the sub-region along a line forming the first incidence angle from the image plane of the training image. (Note 16) Updating the aforementioned feature extraction model means The aforementioned first feature set is input into the task execution model to obtain the first result of the task, The second set of features is input into the task execution model to obtain the second result of the task, Calculating one or more losses based on the first result of the task, the second result of the task, and the ground truth data, The training method according to Appendix 9, comprising updating the trainable parameters of the feature extraction model and the task execution model based on the one or more losses. (Note 17) A non-temporary computer-readable storage medium for storing computer data, wherein the computer can access the computer, Training data is acquired that includes a training image, first angle information, and ground truth data. The training image is an image of an object captured by a sensor and is generated by the sensor. The first angle information indicates the first incident angle, which is the incident angle of the sensor, and the first azimuth angle, which is the azimuth angle of the object captured in the training image. The aforementioned training images are input into the feature extraction model to obtain a first feature set, which is a set of features extracted from the training images. Second angular information indicating a second incidence angle and a second azimuth angle is obtained, and the second incidence angle, the second azimuth angle, or both are different from their corresponding ones in the first angular information. Based on the first and second angle information, a coordinate transformation is performed on the first feature set to generate a second feature set. Updating the feature extraction model based on the first feature set, the second feature set, and the ground truth data. A non-temporary computer-readable storage medium that enables execution of [a certain action]. (Note 18) The first feature set is represented by a set of cells, each cell having a feature value and coordinates in a first coordinate system defined using the first angle of incidence. The second feature set is represented by a set of cells, each cell having a feature value and coordinate in a second coordinate system defined using the second incidence angle, the first azimuth angle, and the second azimuth angle. Performing the coordinate transformation on the first set of features means For each cell in the first feature set, a coordinate transformation is performed on the coordinates of the cell in the first feature set from the first coordinate system to the second coordinate system to calculate the corresponding cell in the second feature set. The storage medium according to Appendix 17, which includes setting the value of the cell in the first feature set to the corresponding cell in the second feature set. (Note 19) The coordinate transformation from the first coordinate system to the second coordinate system is: This includes a transformation from the aforementioned first coordinate system to a world coordinate system defined by a horizontal plane and a vertically upward axis representing the direction opposite to the direction of gravity, This includes rotating the world coordinate system by a rotation angle around the vertically upward axis, wherein the rotation angle is the difference between the first azimuth angle and the second azimuth angle. The storage medium described in Appendix 18, which includes a transformation from the world coordinate system rotated by the aforementioned rotation angle to the second coordinate system. (Note 20) The first feature set is represented by a first set of cells, each cell having feature values ​​and coordinates in a first coordinate system defined using the first incidence angle, The second set of features is represented by a second set of cells, each cell having feature values ​​and coordinates in a second coordinate system defined using the second angle of incidence, the first azimuth angle, and the second azimuth angle. Generating the aforementioned second set of features means Performing the coordinate transformation on the first feature set to convert the first feature set into a third set of cells in the second coordinate system, The storage medium according to Appendix 17, comprising: modifying the values ​​of one or more cells of the third set to generate the second set of features. (Note 21) Generating the aforementioned second set of features means To calculate the characteristics of the difference between the first angle information and the second angle information, The storage medium according to Appendix 20, comprising modifying the value of one or more cells of the third set using the difference characteristics between the first angle information and the second angle information. (Note 22) The aforementioned training image is a radar image generated by the radar, as described in Appendix 17, on the storage medium. (Note 23) The storage medium described in Appendix 22, wherein the first feature set represents, for each sub-region on the training image, the backscatter features at each of two or more points projected onto the sub-region along a line forming the first incidence angle from the image plane of the training image. (Note 24) Updating the aforementioned feature extraction model means The aforementioned first feature set is input into the task execution model to obtain the first result of the task, The second set of features is input into the task execution model to obtain the second result of the task, Calculating one or more losses based on the first result of the task, the second result of the task, and the ground truth data, The storage medium according to Appendix 17, comprising updating the trainable parameters of the feature extraction model and the task execution model based on the one or more losses. [Explanation of symbols]

[0088] 10 Training data 20 training images 22 Object 30 First angle information 32 1st angle of incidence 34 1st azimuth 40 Correct Data 50 Model Set 52 Feature Extraction Models 54 Task Execution Models 70 sensors 75 Radar 80 Feature Set 1 90 Second angle information 92 2nd angle of incidence 94 2nd azimuth 100 Second Feature Set 130 First Coordinate System 132 1st azimuth axis 134 First Range Axis 136 1st incident axis 140 World Coordinate System 146 Vertical upward axis 150 Second Coordinate System 152 2nd azimuth axis 154 Second Range Axis 156 Second incident axis 160 lines 200 images 210 sub-regions 220 Features of a rectangular prism 230 sequences 240 Features of a rectangular prism 250 Feature Modification Models 260 Features of a rectangular prism 270 Features of a rectangular prism 280 Features of a rectangular prism 290 Features of a rectangular prism 300 conversion models 310 Embedding layer 320 coding layer 330 Featured Rectangular Prism 340 conversion models 350 Embedding layer 360 coding layer 370 Features of a rectangular prism 1000 computers 1020 Bus 1040 processor 1060 memory 1080 storage device 1100 Input / Output Interface 1120 Network Interface 2000 training equipment 2020 Training Data Acquisition Department 2040 Angle information acquisition unit 2060 Feature Acquisition Unit 2080 conversion unit 2100 Update Department

Claims

1. The system has a training data acquisition means that acquires training data including training images, first angle information, and correct data. The aforementioned training image is an image of an object, generated by a sensor. The first angle information indicates a first incident angle, which is the incident angle of the sensor, and a first azimuth angle, which is the azimuth angle of the object captured in the training image. A feature acquisition means inputs the aforementioned training images into a feature extraction model and obtains a first feature set, which is a set of features extracted from the aforementioned training images. It includes an angle information acquisition means for acquiring second angle information indicating a second angle of incidence and a second azimuth angle, The second incidence angle is different from the first incidence angle, or the second azimuth angle is different from the first azimuth angle, or both. A transformation means that generates a second feature set by performing a coordinate transformation on the first feature set based on the first angle information and the second angle information, The system includes an update means for updating the feature extraction model based on the first feature set, the second feature set, and the ground truth data, The update means is, The aforementioned first feature set is input into the task execution model to obtain the first result of the task. The second set of features is input to the task execution model to obtain the second result of the task. Based on the first result of the task, the second result of the task, and the correct answer data, one or more losses are calculated. Based on the one or more losses, the trainable parameters of the feature extraction model and the task execution model are updated. The aforementioned correct answer data represents the correct execution result of the task, and is a training device.

2. The aforementioned first feature set is represented by a set of cells, Each cell in the first feature set has a feature value and coordinates in a first coordinate system defined using the first angle of incidence, The aforementioned second set of features is represented by a set of cells, Each cell in the second feature set has a feature value and coordinates in a second coordinate system defined using the second incidence angle, the first azimuth angle, and the second azimuth angle. The conversion means is By performing a coordinate transformation from the first coordinate system to the second coordinate system on the coordinates of each cell in the first feature set, the cell in the second feature set corresponding to each cell in the first feature set is identified. The training device according to claim 1, wherein the value of the cell in the first feature set is set to the cell in the second feature set corresponding to that cell.

3. The coordinate transformation from the first coordinate system to the second coordinate system is, This includes a transformation from the first coordinate system to the world coordinate system, The aforementioned world coordinate system is defined by a horizontal plane and a vertically upward axis representing the direction opposite to the direction of gravity. This includes rotating the world coordinate system around the vertically upward axis by an angle of rotation which is the difference between the first azimuth angle and the second azimuth angle, The training apparatus according to claim 2, comprising a transformation from the world coordinate system rotated by the aforementioned rotation angle to the second coordinate system.

4. The aforementioned first set of features is represented by a first set of cells, Each cell in the first feature set has a feature value and coordinates in a first coordinate system defined using the first angle of incidence, The aforementioned second set of features is represented by a second set of cells, Each cell in the second feature set has a feature value and coordinates in a second coordinate system defined using the second incidence angle, the first azimuth angle, and the second azimuth angle. The conversion means is By performing the coordinate transformation on the first feature set, the first feature set is transformed into a third set of cells in the second coordinate system. The training apparatus according to claim 1, which modifies the values ​​of one or more cells in the third set to generate the second feature set.

5. The conversion means is The characteristics of the difference between the first angle information and the second angle information are calculated. The training apparatus according to claim 4, which modifies the values ​​of one or more cells in the third set using the difference characteristics.

6. The training apparatus according to claim 1, wherein the training image is a radar image generated by radar.

7. The training apparatus according to claim 6, wherein the first feature set represents, for each sub-region on the training image, the backscatter features at each of two or more points projected onto the sub-region along a line whose angle with the image plane of the training image is a first angle of incidence.

8. The system includes a training data acquisition step that acquires training data including training images, first angle information, and ground truth data. The aforementioned training image is an image of an object, generated by a sensor. The first angle information indicates a first incident angle, which is the incident angle of the sensor, and a first azimuth angle, which is the azimuth angle of the object captured in the training image. A feature acquisition step involves inputting the aforementioned training images into a feature extraction model to obtain a first feature set, which is a set of features extracted from the training images. The method includes an angle information acquisition step, which acquires second angle information indicating a second angle of incidence and a second azimuth angle, The second incidence angle is different from the first incidence angle, or the second azimuth angle is different from the second azimuth angle, or both. A transformation step to generate a second feature set by performing a coordinate transformation on the first feature set based on the first angle information and the second angle information, The method includes an update step which updates the feature extraction model based on the first feature set, the second feature set, and the ground truth data, In the update step, The aforementioned first feature set is input into the task execution model to obtain the first result of the task. The second set of features is input to the task execution model to obtain the second result of the task. Based on the first result of the task, the second result of the task, and the correct answer data, one or more losses are calculated. Based on the one or more losses, the trainable parameters of the feature extraction model and the task execution model are updated. The aforementioned correct answer data represents the correct execution result of the task, and is a training method performed by a computer.

9. The computer is instructed to perform a training data acquisition step, which involves acquiring training data that includes training images, first angle information, and ground truth data. The aforementioned training image is an image of an object, generated by a sensor. The first angle information indicates a first incident angle, which is the incident angle of the sensor, and a first azimuth angle, which is the azimuth angle of the object captured in the training image. A feature acquisition step involves inputting the aforementioned training images into a feature extraction model to obtain a first feature set, which is a set of features extracted from the training images. The computer is instructed to perform an angle information acquisition step, which involves acquiring second angle information indicating a second angle of incidence and a second azimuth angle. The second incidence angle is different from the first incidence angle, or the second azimuth angle is different from the first azimuth angle, or both. A transformation step to generate a second feature set by performing a coordinate transformation on the first feature set based on the first angle information and the second angle information, The computer is made to perform an update step, which updates the feature extraction model based on the first feature set, the second feature set, and the ground truth data. In the update step, The aforementioned first feature set is input into the task execution model to obtain the first result of the task. The second set of features is input to the task execution model to obtain the second result of the task. Based on the first result of the task, the second result of the task, and the correct answer data, one or more losses are calculated. Based on the one or more losses, the trainable parameters of the feature extraction model and the task execution model are updated. The aforementioned correct answer data is a program that represents the correct execution result of the task.

Images