Information processing device, information processing method, and program

Abstract

The present disclosure relates to an information processing device, an information processing method, and a program, with which it is possible to determine an optimal configuration of an input modal without relying on prior knowledge. The information processing device is provided with: a new modal generation unit that generates a new modal having N channels (N > 1), which is obtained by combining a prescribed computation with modal data of a modal having P channels (P > 1) and is expressed by a modal transformation formula; and a training unit that trains a learning model by using the new modal having N channels and a training data set therefrom. The new modal generation unit generates a next new modal having N channels, said next new modal being more useful than the new modal having N channels. The training unit trains the learning model by using the next new modal having N channels and a training data set therefrom. The technology of the present disclosure can be applied to, for example, an information processing device that trains a learning model.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and more particularly to an information processing device, an information processing method, and a program that enable an optimal input modal configuration to be determined without relying on prior knowledge.

[0002] There are machine learning models that can be trained using different types of multi-modal data as input data, such as text, images, audio, and video. However, it is difficult to determine the optimal input modal configuration, i.e., the optimal combination of multi-modal data to be used as input data.

[0003] For example, Non-Patent Document 1 proposes a method of manually specifying the input modal configuration of a learning model from among candidate input modalities based on existing prior knowledge. Such a method may involve the possibility that a more optimal input modal configuration exists outside the scope of the existing prior knowledge, and the specified input modal configuration may be a non-optimal configuration.

[0004] On the other hand, for example, Non-Patent Document 2 and Patent Document 1 propose a technology in which a variety of modal data is prepared, features are extracted for each input modality using a dedicated encoder, and the data is input into a learning model designed specifically for a specific task for learning. With this technology, a variety of modal data must always be prepared, which results in an enormous model size for the learning model.

[0005] Japanese Patent Application Laid-Open No. 2022-137145

[0006] Mingqi Shao, Chongkun Xia, Zhendong Yang, Junnan Huang, Xueqian Wang: Transparent Shape from a Single View Polarization Image. ICCV 2023: 9243-9252Yupeng Liang, Ryosuke Wakaki, Shohei Nobuhara, Ko Nishino: Multimodal Material Segmentation. CVPR 2022: 19768-19776

[0007] As described above, a method that can determine the optimal input modal configuration without relying on prior knowledge about the learning model or learning task is desirable, but such a method has not yet been realized.

[0008] The present disclosure has been made in view of such circumstances, and makes it possible to determine an optimal input modal configuration without relying on prior knowledge.

[0009] An information processing device according to a first aspect of the present disclosure includes: a new modal generation unit that generates a new modal of N channels (N>1) represented by a modal transformation formula that combines modal data of a modal of P channels (P>1) with a predetermined calculation; and a learning unit that uses the new modal of N channels and its learning data set to train a learning model, wherein the new modal generation unit generates a new modal of the next N channels that is more useful than the new modal of the N channels, and the learning unit trains the learning model using the next new modal of N channels and its learning data set.

[0010] An information processing method according to a first aspect of the present disclosure includes: generating a new modal of N channels (N>1) represented by a modal transformation formula that combines modal data of a modal of P channels (P>1) with a predetermined operation; and training a learning model using the new modal of N channels and its training data set; generating a next new modal of N channels that is more useful than the new modal of N channels; and training a learning model using the next new modal of N channels and its training data set.

[0011] The program of the first aspect of the present disclosure causes a computer to execute the following processes: generating a new modal of N channels (N>1) represented by a modal transformation formula combining modal data of a modal of P channels (P>1) with a predetermined operation; and training a learning model using the new modal of N channels and its training data set; generating a next new modal of N channels that is more useful than the new modal of N channels; and training a learning model using the next new modal of N channels and its training data set.

[0012] A first aspect of the present disclosure includes generating a new N-channel (N>1) modal represented by a modal transformation formula combining modal data of a P-channel (P>1) modal with a predetermined operation, and training a learning model using the N-channel new modal and its training data set, where a next N-channel new modal that is more useful than the N-channel new modal is generated, and training a learning model is performed using the next N-channel new modal and its training data set.

[0013] The information processing device according to the second aspect of the present disclosure includes an estimation unit that generates new N-channel (N>1) modals represented by a modal transformation formula combining modal data of P-channel (P>1) modals with a predetermined calculation, and performs estimation using the learning model determined by the learning device using modal data of the P-channel modals corresponding to the input modal configuration determined by the learning device, by repeating a process of learning a learning model using the new N-channel modals and the learning data set to search for an input modal configuration that is highly useful.

[0014] In a second aspect of the present disclosure, a new N-channel (N>1) modal is generated, represented by a modal transformation formula combining modal data of a P-channel (P>1) modal with a predetermined operation, and a process of training a learning model is repeated using the new N-channel modal and its training data set. Estimation is performed using the learning model determined by the learning device, using modal data of the P-channel modal corresponding to the input modal configuration determined by the learning device, which searches for an input modal configuration that is more useful.

[0015] The program can be provided by transmitting it via a transmission medium or by recording it on a recording medium.

[0016] The information processing device may be an independent device or an internal block constituting a single device.

[0017] FIG. 1 is a block diagram showing a configuration example of an information processing device according to an embodiment of the present disclosure. FIG. 2 is a block diagram showing a configuration example of a learning data acquisition unit of FIG. 1. FIG. 3 is a block diagram showing configuration examples of a modal configuration optimization unit and a modal learning unit of FIG. 1. FIG. 4 is a diagram explaining a new modal generation process by a new modal generation unit. FIG. 5 is a diagram explaining a new modal generation process by a new modal generation unit. FIG. 6 is a diagram explaining a new modal generation process by a new modal generation unit. FIG. 7 is a flowchart explaining an optimal input modal search process by the information processing device of FIG. 1. FIG. 8 is a block diagram showing a first configuration example of an information processing system. FIG. 9 is a block diagram showing a second configuration example of an information processing system. FIG. 10 is a block diagram showing a configuration example of an embodiment of a computer to which the technology of the present disclosure is applied.

[0018] Hereinafter, modes for carrying out the technology of the present disclosure (hereinafter referred to as embodiments) will be described with reference to the accompanying drawings. The description will be given in the following order: 1. Block diagram of information processing device 2. Configuration example of learning data acquisition unit 3. Configuration example of modal configuration optimization unit and modal learning unit 4. New modal generation process by modal configuration optimization unit 5. Optimal input modal search process 6. Summary of information processing device 7. Utilization of optimal input modal configuration and learning model 8. Computer configuration example

[0019] 1. Block Diagram of Information Processing Apparatus> FIG. 1 is a block diagram showing an example configuration of an information processing apparatus according to an embodiment of the present disclosure.

[0020] The information processing device 1 in Fig. 1 autonomously determines an optimal input modal configuration for a predetermined learning model from among multiple modal data, including different types, and learns model parameters of the learning model using the determined input modal configuration. The information processing device 1 is configured, for example, with a general-purpose personal computer, a server device, a dedicated terminal device, or the like. The input modal represents predetermined information (modality) provided as input to the learning model, and there are various types of input modal, such as text, image, audio, video, and measurement data.

[0021] The information processing device 1 includes a learning data acquisition unit 10 , a modal configuration optimization unit 20 , a modal learning unit 30 , and an input unit 40 .

[0022] The training data acquisition unit 10 acquires a set of training data (hereinafter also referred to as a training data set), which is modal data that is the source of input modals and is used for training and evaluating a training model, and stores it in an internal storage unit (training data DB 54 in FIG. 2 ). The training data acquisition unit 10 supplies a portion of the training data set selected from the training data sets stored in the storage unit to the modal configuration optimization unit 20 as appropriate.

[0023] The modal configuration optimization unit 20 newly generates candidate modals that are candidates for the optimal input modal, and supplies them to the modal learning unit 30 together with the training data set acquired from the training data acquisition unit 10. The modal configuration optimization unit 20 also acquires, from the training data acquisition unit 10, evaluation results of a learning model trained using the candidate modals and the training data set. The modal configuration optimization unit 20 repeatedly updates the candidate modals in accordance with the evaluation results, thereby optimizing the candidate modals and determining the optimal input modal configuration.

[0024] The modal learning unit 30 learns a learning model using the candidate modals and the learning data set supplied from the modal configuration optimization unit 20. The configuration of the learning model is specified by learning model information from the input unit 40. The modal learning unit 30 supplies the modal configuration optimization unit 20 with an evaluation result of the learning model, which is the result of learning.

[0025] The input unit 40 includes, for example, a touch panel, switches, buttons, a keyboard, a mouse, a microphone, an input terminal, etc., and acquires information input or specified by user operation and supplies it to each unit within the information processing device 1 as appropriate.

[0026] The information processing device 1 is configured as described above. In the information processing device 1, the modal configuration optimization unit 20 determines candidate modals and a learning data set, and the modal learning unit 30 uses the determined candidate modals and learning data set to train and evaluate a learning model. This process is repeated until an optimal input modal configuration is determined. In other words, the modal configuration optimization unit 20 determines candidate modals and a learning data set, and the modal learning unit 30 uses the determined candidate modals and learning data set to train and evaluate a learning model, thereby determining an optimal input modal configuration.

[0027] 2. Example of Configuration of Learning Data Acquisition Unit FIG. 2 is a block diagram showing an example of the configuration of the learning data acquisition unit 10. As shown in FIG.

[0028] The learning data acquisition unit 10 includes a measurement unit 51 , a CG rendering unit 52 , a learning data DB 54 , and a ground truth generation unit 53 .

[0029] The training data acquisition unit 10 is supplied with dataset acquisition setting information input by a user from the input unit 40. The dataset acquisition setting information is supplied to either one or both of the measurement unit 51 and the CG rendering unit 52. The dataset acquisition setting information is information indicating a training dataset to be acquired. The dataset acquisition setting information includes, for example, the type of modal to be acquired, the acquisition method, and the number of data. The types of modal to be acquired include, for example, luminance information, color information, polarization information, distance information, and non-visible information. The luminance information is information including luminance values ​​(intensity), such as a grayscale image or a binary image. The color information is information including, for example, RGB, HSV (Hue, Saturation, Value), YCbCr (luminance signal Y, color difference signals Cb, Cr), multispectrum, etc. The polarization information is information specifying, for example, the polarization angles of the polarized image (0°, 45°, 90°, and 135°), the degree of linear polarization (DoLP), the angle of linear polarization (AoLP), the azimuth angle (Azimuth), and the zenith angle (Zenith). The distance information is information including, for example, distance information using the ToF method (ToF information), distance information using phase difference (ZAF information), distance information using a stereo camera (parallax information), and distance information expressed as a point cloud (3D information). The non-visible information is information including, for example, near-infrared (NIR) in the wavelength range of 750 nm or more and less than 1000 nm, short-wave infrared (SWIR) in the wavelength range of 1000 nm or more and less than 2500 nm, etc. The acquisition method of the modal to be acquired includes information specifying whether it is to be acquired by actual measurement or non-actual measurement. The number of data of the modal to be acquired includes, for example, the number of images, the number of data, etc.

[0030] The measurement unit 51 is configured with various sensors, for example, an RGB sensor, a monochrome sensor, a polarization sensor, a depth sensor (ToF sensor), a stereo camera, a spectrum sensor, a gyro sensor, an acceleration sensor, a temperature sensor, an IMU (Inertial Measurement Unit), etc. When acquisition of a training dataset by actual measurement is specified as the modal acquisition method of the dataset acquisition setting information, the measurement unit 51 generates the specified training dataset by actual measurement and supplies it to the training data DB 54.

[0031] The CG rendering unit 52 generates CG images by performing rendering using (data of) 3D models of various objects stored in the internal memory. When acquisition of a training dataset by non-actual measurement is specified as the modal acquisition method in the dataset acquisition setting information, the CG rendering unit 52 generates the specified training dataset by performing rendering and supplies it to the training data DB 54.

[0032] The training data DB 54 stores training data sets generated by the measurement unit 51 and the CG rendering unit 52. Ground truth information generated by the ground truth generation unit 53 is added to at least some of the training data sets stored in the training data DB 54. The training data sets stored in the training data DB 54 include a training training data set used for training a training model and an evaluation training data set used for evaluating the training model.

[0033] The ground truth generation unit 53 assigns ground truth information to at least a portion of the training data set stored in the training data DB 54 in accordance with the ground truth setting information supplied from the input unit 40. The ground truth information is information indicating the correct answer for the task performed by the training model. For example, if the task performed by the training model is image region segmentation, the ground truth information can be configured as a binary image indicating whether each pixel in the image belongs to a region for the training data. For example, if the task performed by the training model is object detection within an image, the ground truth information can be configured as a bonding box indicating the object region or a label image indicating a label indicating the type of object for each pixel. For example, if the task performed by the training model is object shape estimation, the ground truth information can be configured as a normal map indicating the object's normal information.

[0034] The training data acquisition unit 10 configured as described above acquires a training data set, which is modal data that is the basis for candidate modals, and stores it in the training data DB 54. The training data set stored in the training data DB 54 is supplied to the modal configuration optimization unit 20 as necessary. In the following description, it is assumed that the training data set supplied from the training data DB 54 to the modal configuration optimization unit 20 also includes ground truth information.

[0035] 3. Configuration Example of Modal Configuration Optimizer and Modal Learning Unit FIG. 3 is a block diagram showing a configuration example of the modal configuration optimizer 20 and modal learning unit 30. As shown in FIG.

[0036] The modal configuration optimization unit 20 includes a baseline modal setting unit 71 and a new modal generation unit 72. The modal learning unit 30 includes a baseline modal learning unit 81, a learned model output unit 82, and a convergence determination unit 83.

[0037] The baseline modal setting unit 71 determines the modal of the learning model (baseline) and sets it in the baseline modal learning unit 81. The modal of the learning model is composed of N channels (N>0) and is expressed by a modal transformation formula T that performs transformation using one or more modals. The modal transformation formula T may be a formula that represents data of one modal stored in the learning data DB 54.

[0038] In the initial setting, the baseline modal setting unit 71 determines the modal of the learning model based on baseline modal initial setting information supplied from the input unit 40. The user can manually specify the modal of the learning model for the initial N channels using the input unit 40. Alternatively, the user may specify using the input unit 40 that the modal of the learning model for the initial N channels be randomly selected from among a large number of modals stored in the learning data DB 54. Information specifying the modal determined by the selection method specified by the user using the input unit 40 is supplied from the input unit 40 to the baseline modal setting unit 71 as baseline modal initial setting information.

[0039] On the other hand, in the second and subsequent settings, the baseline modal setting unit 71 determines the modal of the learning model using the modal transformation formula T supplied from the new modal generation unit 72 .

[0040] The baseline modal setting unit 71 acquires a learning data set, which is modal data used to calculate the modal transformation formula T of the modal of the determined learning model, from the learning data DB 54, and supplies it to the baseline modal learning unit 81 of the modal learning unit 30 together with the modal transformation formula T, and sets it in the learning model of the baseline modal learning unit 81. For example, based on the baseline modal initial setting information, the baseline modal setting unit 71 sets the modal transformation formula T representing an image of three channels of R, G, and B, and the learning data set of images of three channels of R, G, and B acquired from the learning data DB 54, in the learning model of the baseline modal learning unit 81. Also, based on the baseline modal initial setting information, the baseline modal setting unit 71 sets the modal transformation formula T representing an image of three channels of R, G, and B, acquired from the learning data DB 54, in the learning model of the baseline modal learning unit 81.

[0041] The new modal generation unit 72 searches for a better new modal compared with the modal set in the learning model of the baseline modal learning unit 81 by the baseline modal setting unit 71. Specifically, the new modal generation unit 72 acquires, from the training data DB 54, a training data set, which is modal data corresponding to a P-channel candidate modal based on the candidate modal initial setting information supplied from the input unit 40. The candidate modal initial setting information includes at least information indicating the number of channels P of the candidate modal to be used when generating an N-channel modal. The number of channels of the P-channel and the N-channel is independent of each other and can be set to any value. The user can manually specify the P-channel candidate modal using the input unit 40. Alternatively, the user can specify using the input unit 40 that the P-channel candidate modal be randomly selected from among the multiple modals stored in the training data DB 54. Information specifying the P-channel candidate modal determined using the selection method specified by the user is supplied from the input unit 40 to the new modal generation unit 72 as candidate modal initial setting information. For example, when P=4 and the candidate modal initial setting information indicates a four-channel candidate modality consisting of a three-channel image of R, G, and B and a grayscale image, the new modal generation unit 72 acquires a training data set that is modal data corresponding to the P-channel candidate modality from the training data DB 54. For example, when the candidate modal initial setting information indicates a four-channel candidate modality consisting of a grayscale image, an NIR image, a range image of a ToF sensor, and an edge detection image, the new modal generation unit 72 acquires a training data set that is modal data corresponding to the P-channel candidate modality from the training data DB 54.

[0042] The new modal generation unit 72 generates multiple N-channel new modals using the P-channel candidate modals. The number of N-channel new modals to be generated here is assumed to be k. The new modal generation unit 72 then determines, from among the k generated N-channel new modals, an N-channel new modal that is more useful than the modal of the current learning model. The new modal generation unit 72 outputs the modal transformation formula T of the determined N-channel new modal to the baseline modal setting unit 71.

[0043] When the modal learning unit 30 finishes learning the learning model using the new modals of N channels output by the new modal generation unit 72 to the baseline modal setting unit 71, an evaluation result is supplied from the trained model output unit 82 of the modal learning unit 30. When the evaluation result is supplied, the new modal generation unit 72 takes the evaluation result into consideration and generates the next new modal of N channels (the modal transformation formula T), and supplies it to the baseline modal setting unit 71.

[0044] The baseline modal learning unit 81 of the modal learning unit 30 is supplied with the modal transformation formula T and a learning dataset from the baseline modal setting unit 71. Furthermore, the baseline modal learning unit 81 is supplied with learning model information from the input unit 40. The learning model information is information specified by the user via the input unit 40 to define the learning model, and includes information on the type of learning model and the specific structure of the learning model. The user can determine the type of learning model and the specific structure of the learning model in accordance with the task via the input unit 40. Examples of the type of learning model that can be specified include CNN and Transformer. Examples of the specific structure of the learning model that can be specified include ResNet50 and U-Net. Hereinafter, the learning model determined by the learning model information will be referred to as learning model f.

[0045] The baseline modal learning unit 81 sets the modal transformation formula T and the learning dataset for learning supplied from the baseline modal setting unit 71 to the structure of the learning model f specified by the learning model information, and trains the learning model f. The baseline modal learning unit 81 also quantitatively evaluates the trained learning model f using the training dataset for evaluation supplied from the baseline modal setting unit 71, and calculates the evaluation result. For example, in a learning model that estimates a numerical value, the baseline modal learning unit 81 calculates the evaluation result of the learning model f using MAE (mean absolute error), MSE (mean square error), etc. as evaluation indices. For example, in a learning model that performs classification, the baseline modal learning unit 81 calculates the evaluation result of the learning model f using accuracy rate, precision rate, etc. as evaluation indices. The baseline modal learning unit 81 supplies the model parameters and evaluation result of the learning model f obtained by training to the trained model output unit 82 and the convergence determination unit 83.

[0046] The trained model output unit 82 stores the model parameters and evaluation results of the trained training model f supplied from the baseline modal training unit 81 in an internal memory (storage unit). The trained model output unit 82 may also acquire training model information defining the training model f from the baseline modal training unit 81 and store it in its internal memory together with the model parameters. If a completion instruction is not supplied from the convergence determination unit 83, the trained model output unit 82 outputs the model parameters and evaluation results of the trained model for the modal (of the modal transformation formula T) supplied from the baseline modal setting unit 71 to the new modal generation unit 72. When the new modal generation unit 72 receives the model parameters and evaluation results for the current modal, it again executes the process of searching for a new modal that is better than the current modal. On the other hand, if a completion instruction is supplied from the convergence determination unit 83 to the trained model output unit 82, the trained model output unit 82 does not output the model parameters and evaluation results of the current trained model to the new modal generation unit 72. In this case, the new modal generation unit 72 does not execute the process of searching for a better new modal, and therefore the process of autonomously searching for an optimal input modal configuration is terminated.

[0047] The convergence determination unit 83 determines whether the learning model f has converged based on a predetermined convergence condition. For example, if the convergence determination unit 83 determines that the learning model f has reached a specified target performance based on the evaluation result supplied from the baseline modal learning unit 81, the convergence determination unit 83 determines that the convergence condition has been satisfied and that the learning model f has converged. Also, for example, if the convergence determination unit 83 determines that the number of iterations of searching the input modal has reached a specified number (e.g., 100 times) given as the convergence condition, the convergence determination unit 83 determines that the convergence condition has been satisfied and that the learning model f has converged. If the convergence determination unit 83 determines that the learning model f has converged, it outputs a completion instruction to the trained model output unit 82.

[0048] The modal configuration optimization unit 20 and the modal learning unit 30 are configured as described above.

[0049] 4. New Modal Generation Processing by Modal Configuration Optimizer> Next, the new modal generation processing performed by the new modal generation unit 72 of the modal configuration optimizer 20 will be described with reference to FIGS.

[0050] As a prerequisite for the new modal generation process described in Figures 4 to 6, it is assumed that the baseline modal setting unit 71 determines the initial N-channel modal and sets it in the baseline modal learning unit 81, that the baseline modal learning unit 81 learns a learning model f corresponding to the task based on the learning model information, and that the model parameters and evaluation results of the initial learning model f are output from the learned model output unit 82 to the new modal generation unit 72.

[0051] First, as shown in FIG. 4, the new modal generation unit 72 determines a candidate modal for the P channel based on the candidate modal initial setting information supplied from the input unit 40. The new modal generation unit 72 acquires modal data of the determined candidate modal from the training data DB 54 as a training data set. In the example of FIG. 4, the candidate modal z １ ~z ４ As the four-channel image of the polarization sensor at polarization angles of 0°, 45°, 90°, and 135°, the candidate modal z ５ As one channel of the range image by the ToF sensor, the candidate modal z ６ As one channel of the NIR image by the NIR sensor, and the candidate modal z ７ Modal data for a total of seven channels (P=7), including one channel of the edge-detected image, is acquired as a training data set from the training data DB 54. The user may specifically specify candidate modals for these seven channels using the input unit 40, and information on the specified candidate modals may be supplied from the input unit 40 to the new modal generation unit 72 as initial candidate modal information. Alternatively, the user may specify only the number of channels P using the input unit 40, and information on the specified number of channels P may be supplied to the new modal generation unit 72 as initial candidate modal information, and the new modal generation unit 72 may randomly determine a candidate modal for the specified number of channels P.

[0052] Next, the new modal generator 72 generates candidate modal z for the seventh channel (P channel) as shown in FIG. P Using (P=1 to 7), k new modal x's with N channels are generated. In the example of Figure 5, N=2, and k new modal x's each consisting of two channels are generated. １ ~x ｋ Here is an example of a new modal x ｊ (j=1 to k) is the modal transformation formula T ｊ (z) is defined as the modal transformation formula T ｊ (z) can be generated using a symbolic operation technique used in data-driven reinforcement learning. For example, the new modal generator 72 generates +, −, ×, ÷, Σ, e -x ,√,sin,cos,etc. from various basic mathematical operation sets, and modal transformation formula T ｊ (z) is automatically generated, and k modal transformation equations T １ (z)~T ｋ (z) is generated. Examples of open source software for symbolic operation technology include PhySO (Physical Symbolic Optimization). PhySO is disclosed at "https: / / github.com / WassimTenachi / PhySO" and "https: / / arxiv.org / pdf / 2303.03192.pdf".

[0053] For example, the first new modal x １ The modal transformation equation for two channels is T １ (z) is calculated by the new modal generating unit 72 as follows:

[0054] Modal transformation formula T １ (z) is the modal transformation equation T of the first channel １１ (z) = x １１ and the modal transformation equation T of the second channel １２ (z) = x １２ The modal transformation equation for the first channel, T １１ (z) is the candidate modal z of the seven channels PAmong them, the candidate modal z １ ~z ４ Four-channel candidate modal z P The modal transformation equation for the second channel, T, is １２ (z) is the candidate modal z of the seven channels P Among them, the candidate modal z ５ ~z ７ Three-channel candidate modal z P It consists of a formula to calculate the average of the images.

[0055] Next, the second new modal x ２ The modal transformation equation for two channels is T ２ (z) is calculated by the new modal generating unit 72 as follows:

[0056] Modal transformation formula T ２ (z) is the modal transformation equation T of the first channel ２１ (z) = x ２１ and the modal transformation equation T of the second channel ２２ (z) = x ２２ The modal transformation equation for the first channel, T ２１ (z) is the candidate modal z of the seven channels P Among them, the candidate modal z １ ~z ４ The modal transformation equation for the second channel, T, is calculated using images taken at polarization angles of 0°, 45°, 90°, and 135° by the polarization sensor. ２２ (z) is the candidate modal z of the seven channels P Among them, the candidate modal z １ ~z ４ It is composed of an equation for calculating the linear polarization angle AoLP using images taken by the polarization sensor at polarization angles of 0°, 45°, 90°, and 135°.

[0057] Modal transformation formula T ３ (z)~T ｋ Similarly, for (z), the two-channel modal transformation formula T ３ (z)~T ｋ (z) is calculated.

[0058] Next, the new modal generator 72 generates k new modals x １ ~x ｋ Regarding the usefulness of the learning model f, R １ (X) ~ R ｋ Calculate (X). Benefit R ｊ As will be described later, (X) is a new modal model x that does not change the output of the learning model f even if the amount of noise σ added to the image of each channel is increased. In other words, it is a highly robust model. ｊ The new modal generation unit 72 generates k new modals x １ ~x ｋ Benefits of R １ (X) ~ R ｋ (X) and find the most useful R ｊ New modal x with (X) ｊ The new modal generation unit 72 determines the N-channel new modal that is more useful. The new modal generation unit 72 supplies the baseline modal setting unit 71 with the modal transformation formula T of the determined N-channel new modal.

[0059] If the task in the example of Figure 5 is a "lane detection task" to detect lanes on a road, the modal transformation formula T １ (z) Beneficial R １ (X) as R １ (X) = 0.2 was obtained. The average of the images produces a blurred image, and the modal transformation formula T １ (z) Beneficial R １ (X) is not high. On the other hand, the modal transformation formula T ２ (z) Beneficiality R ２ (X) is usefulness R ２ The linear polarization degree DoLP and linear polarization angle AoLP have the effect of highlighting lanes with high contrast, and the usefulness R ２ (X) is considered to be high. In this way, k new modal x １ ~x ｋ Benefits of R １ (X), R ２ (X), R ３ (X), ..., R ｋ5, the modal transformation formula T(X) is calculated and becomes 0.2, 1.1, 0.3, ..., 0.2, respectively. In this case, the new modal generation unit 72 selects the modal transformation formula T(X) with the highest usefulness R(X). ２ (z) is determined to be a new modal of N channels that is more useful.

[0060] In the example of FIG. 6, the task is different from the example of FIG. 5 and is an example of a "car and person detection task" in which cars and people around a road are detected. Using a learning model f corresponding to the "car and person detection task," k new modal models x １ ~x ｋ Benefits of R １ (X) ~ R ｋ The result of calculating (X) is the usefulness R １ (X), R ２ (X), R ３ (X), ..., R ｋ 6, the modal transformation formula T(X) is calculated to be 0.5, 0.1, 2.6, ..., 0.3, respectively. In this case, the new modal generation unit 72 selects the modal transformation formula T(X) with the highest usefulness R(X). ３ (z) is determined to be a new modal of N channels that is more useful.

[0061] Next, we explain how to calculate the usefulness R(X) of a new modal x of N channels.

[0062] Now, let us assume that the new modal x is an image x of N channels obtained by a predetermined sensor. The new modal generation unit 72 generates an image x' by adding one channel of the same Gaussian noise Δx to each channel of the image x. x' = x + Δx (1) The image x' in equation (1) has a Gaussian distribution N(x, Σ(σ) = diag(σ1 2 ,σ2 2 ,…,σ ｎ 2 )) and the image will be σ ｉ represents the variance of Gaussian noise added to the i-th pixel of image x. diag() represents a diagonal matrix. n is the number of pixels in image x.

[0063] Next, the new modal generation unit 72 calculates the output difference D between the learning model f(x) and the learning model f(x') when the images x and x' before and after adding the Gaussian noise Δx are input to the learning model f. D = ∥f(x) - f(x')∥ ２ (2) In equation (2), ||A|| represents the norm of A.

[0064] Next, the new modal generator 72 generates a noise amount vector σ={σ １ , σ ２ , σ ３ , ..., σ ｎ} is calculated. In equation (3), argmax[A] represents the noise amount vector σ that maximizes [A]. In equation (3), ω represents the ratio between the output difference D and σ ｉ According to equation (3), log(σ ｉ ) (Σlog(σ p )) and the square of the output difference D (D ２ The noise amount vector σ is calculated so that the difference between the noise amount for each pixel and the noise amount {σ １ , σ ２ , σ ３ , ..., σ ｎ On the other hand, the square of the output difference D (D ２ ) is preferable because it means that the noise resistance is higher. Therefore, the new modal generator 72 uses log(σ ｉ ) (Σlog(σ p )) and the square of the output difference D (D ２ ) and log(σ ｉ ) (Σlog(σ p )) and the square of the output difference D (D ２ The noise amount vector σ is calculated by dividing the pixel value of each pixel by the maximum noise {σ １ , σ ２ , σ ３, ..., σ ｎ} corresponds to the noise map image σ.

[0065] Next, the new modal generator 72 calculates the usefulness R(x) of the N-channel image x using the noise amount vector σ as in equation (4). The usefulness R(x) is the noise σ of each pixel that composes the noise map image σ. ｉ The greater the noise, the higher the usefulness R(x).

[0066] As described above, the usefulness R(x) of the N-channel image x is calculated. The new modal generation unit 72 calculates the usefulness R(X) of the N-channel new modal x by calculating the sum of the usefulness R(x) calculated in the same manner for each of the N-channel images x of the new modal x in the training data set, as shown in equation (5).

[0067] 7, the optimal input modal search process executed by the information processing device 1 will be described. This process is started when the user inputs baseline modal initial setting information, learning model information, candidate modal initial setting information, etc., via the input unit 40 and issues a command to start the process. It is also assumed that, prior to this process, the learning data set required for learning the learning model f has already been stored in the learning data DB 54 of the learning data acquisition unit 10.

[0068] First, in step S1, the baseline modal setting unit 71 of the modal configuration optimization unit 20 determines whether the initial learning model f is being set.

[0069] If it is determined in step S1 that the initial learning model f has been set, the process proceeds to step S2, where the baseline modal setting unit 71 sets the N-channel modal to the learning model f of the baseline modal learning unit 81 of the modal learning unit 30 based on the baseline modal initial setting information supplied from the input unit 40. In addition, the baseline modal setting unit 71 acquires a learning data set, which is modal data used to calculate the modal transformation formula T of the set modal, from the learning data DB 54, and supplies the learning data set to the baseline modal learning unit 81.

[0070] On the other hand, if it is determined in step S1 that the initial learning model f is not set, the process proceeds to step S3, where the baseline modal setting unit 71 sets the N-channel modal to the learning model f of the baseline modal learning unit 81 of the modal learning unit 30 based on the modal transformation formula T generated by the new modal generation unit 72. In addition, the baseline modal setting unit 71 obtains a learning data set, which is modal data used to calculate the modal transformation formula T of the set modal, from the learning data DB 54, and supplies the learning data set to the baseline modal learning unit 81.

[0071] In step S4 , the baseline modal learning unit 81 uses the modal transformation formula T and the learning data set supplied from the baseline modal setting unit 71 to learn the learning model f.

[0072] In step S5, the baseline modal learning unit 81 quantitatively evaluates the trained learning model f using the training dataset for evaluation and calculates the evaluation result. After calculating the evaluation result, the baseline modal learning unit 81 supplies the model parameters of the trained learning model f obtained by training and the evaluation result to the trained model output unit 82 and the convergence determination unit 83.

[0073] In step S6, the trained model output unit 82 stores the model parameters and evaluation results of the trained model supplied from the baseline modal training unit 81 in its internal memory.

[0074] In step S7, the convergence determination unit 83 determines whether the learning model f has converged based on a predetermined convergence condition. For example, the convergence determination unit 83 determines that the search process has converged when the learning model f has reached a specified target performance based on the evaluation result or when the number of iterations of searching the input modal has reached a predetermined number.

[0075] If it is determined in step S7 that the learned model f has not yet converged, the processing proceeds to step S8, and the learned model output unit 82 outputs the model parameters and evaluation results of the learned model supplied from the baseline modal learning unit 81 to the new modal generation unit 72 of the modal configuration optimization unit 20.

[0076] Next, in step S9, the new modal generation unit 72 acquires modal data of the candidate modals for the P channel from the learning data DB 54, generates k new modals for the next N channels, and calculates the usefulness R(X) of each. At least one of the candidate modals for the P channel may be changed, or all may be left unchanged and only the modal transformation formula T may be changed.

[0077] In step S10, the new modal generation unit 72 determines whether the modal with the maximum usefulness R(X) among the k new modals of the N channels generated exceeds the usefulness R(X) of the modal of the current learning model f.

[0078] If it is determined in step S10 that the new modal with the greatest usefulness R(X) does not exceed the usefulness R(X) of the modal of the current learning model f, the process returns to step S9, whereby k new modals are generated again.

[0079] On the other hand, if it is determined in step S10 that the new modal with the maximum usefulness R(X) exceeds the usefulness R(X) of the modal of the current learning model f, the process proceeds to step S11, where the new modal generation unit 72 outputs the modal transformation formula T of the new modal with the maximum usefulness R(X) to the baseline modal setting unit 71. Thereafter, the process returns to step S1, and the processes from step S1 onwards are executed. In other words, the learning model f is trained using the modal transformation formula T of the N-channel new modal with improved usefulness R(X) and the training data set.

[0080] On the other hand, if it is determined in step S7 that the learned model f has converged, the process proceeds to step S12, where the convergence determination unit 83 outputs a completion instruction to the learned model output unit 82. Having acquired the completion instruction, the learned model output unit 82 does not output the model parameters and evaluation results of the current learned model to the new modal generation unit 72, so that a new modal for N channels is not generated, and the optimal input modal search process ends.

[0081] The optimum input modal search process by the information processing device 1 is executed as described above.

[0082] In the above-mentioned optimal input modal search process, the information processing device 1 generates k new modals (of modal transformation formulas T) for N channels in step S9, selects the best new modal from the k new modals that maximizes the usefulness R(X), and trains and evaluates the learning model f.

[0083] However, in step S9, a process may be adopted in which one new modal (modal transformation formula T) that improves the modal usefulness R(X) of the current learning model f is generated, and the learning model f is trained and evaluated.

[0084] In the process of generating k new modals, the loop processing of steps S1 to S11 is repeated a smaller number of times due to the loop processing of steps S9 and S10. On the other hand, in the process of generating one new modal that improves usefulness R(X), the loop processing of steps S9 and S10 can be omitted, but the loop processing of steps S1 to S11 is repeated a larger number of times. Regardless of which process is adopted, the loop processing results in the search for and determination of the optimal input modal configuration.

[0085] 6. Summary of Information Processing Device As described above, the information processing device 1 includes a new modal generation unit 72 that generates a new modal for N channels (N > 1) represented by a modal transformation formula combining modal data of a modal for P channels (P > 1) with a predetermined calculation, and a baseline modal learning unit 81 that trains a learning model f using the new modal for N channels and its training data set. The new modal generation unit 72 generates a new modal for the next N channels that is more useful than the new modal for N channels, and the baseline modal learning unit 81 trains the learning model f using the new modal for the next N channels and its training data set. The modal transformation formula is determined by a combination of predetermined calculations selected without human intervention from among multiple calculations. The generation of a new modal for the next N channels that is more useful by the new modal generation unit 72 and the training of the learning model f by the baseline modal learning unit 81 using the generated new modal for the next N channels and its training data set are repeated until a predetermined convergence condition is satisfied.

[0086] This allows the configuration of input modal to be generated autonomously without relying on prior knowledge or the structure of a specific learning model f, and the optimal input modal configuration can be determined. From the candidate modalities of the P channel, the optimal input modal configuration can be autonomously searched for using data-driven mathematical formula search technology. There is no need to expand the scale of the learning model itself in order to search for the optimal input modal configuration. By setting the learning model information, it can be applied to any learning model.

[0087] 7. Optimal Input Modal Configuration and Utilization of Learning Model Next, an optimal input modal configuration determined by the information processing device 1 as a learning device that learns the learning model f and a configuration that utilizes the learning model f will be described.

[0088] <First Configuration Example of Information Processing System> FIG. 8 is a block diagram showing a first configuration example of an information processing system that utilizes an optimal input modal configuration and a learning model.

[0089] 8 includes an information processing device 1, a modal acquisition unit 110, and a model estimation unit 120. The modal acquisition unit 110 and the model estimation unit 120 may be configured as a single information processing device 150, or may each be configured as an independent device.

[0090] The modal acquisition unit 110 acquires input modal configuration information output from the modal configuration optimization unit 20. The input modal configuration information is information indicating the optimal input modal configuration determined by the modal configuration optimization unit 20. The modal acquisition unit 110 is configured with one or more sensors, devices, etc., such as those included in the above-described measurement unit 51. For example, the modal acquisition unit 110 has at least one sensor or device, such as an RGB sensor, a polarization sensor, or a depth sensor (ToF sensor). The modal acquisition unit 110 selects and executes a sensor capable of acquiring modal data of the modal indicated by the input modal configuration information, and outputs the generated sensing data as modal data to the model estimation unit 120. The modal acquisition unit 110 may be a device configured only with sensors capable of acquiring modal data of the modal indicated by the input modal configuration information.

[0091] The model estimation unit 120 acquires learning model information and model parameters from the modal learning unit 30. The learning model information is information indicating the specific structure of the learning model f when the baseline modal learning unit 81 of the information processing device 1 searches for the optimal input modal configuration. The learning model information includes information on the type of learning model, such as CNN or Transformer, the specific structure of the learning model, such as ResNet50 or U-Net, and the modal transformation formula T. The model parameters are learned model parameters of the learning model f when the baseline modal learning unit 81 searches for the optimal input modal configuration.

[0092] The model estimation unit 120 generates a trained model by setting the model parameters from the modal learning unit 30 to the training model f indicated by the training model information. Then, the model estimation unit 120 inputs the sensing data output as modal data from the modal acquisition unit 110 into the trained model and performs estimation using the trained model. The model estimation unit 120 outputs the estimation result to an external device or displays it on a display or the like.

[0093] According to the first configuration example of the information processing system, modal data of the optimal input modal configuration searched for by the information processing device 1 can be obtained, and estimation can be performed using the optimal learning model f corresponding to the desired task.

[0094] <Second Configuration Example of Information Processing System> FIG. 9 is a block diagram showing a second configuration example of an information processing system that utilizes an optimal input modal configuration and a learning model.

[0095] The information processing system of the first configuration example shown in FIG. 9 includes an information processing device 1 and a data acquisition device design unit 200 .

[0096] The data acquisition device design unit 200 acquires the input modal configuration information output from the modal configuration optimization unit 20. The data acquisition device design unit 200 designs sensors and devices that realize the input modal configuration indicated by the acquired input modal configuration information.

[0097] According to the second configuration example of the information processing system, it is possible to design and manufacture a device that can acquire modal data of the optimal input modal configuration searched for by the information processing device 1. The device designed and manufactured according to the second configuration example may be employed as the modal acquisition unit 110 of the first configuration example shown in FIG.

[0098] 8. Computer Configuration Example The series of processes performed by the information processing device 1, the information processing device 150, etc. described above can be executed by hardware or software. When the series of processes are executed by software, the programs that make up the software are installed in a computer. Here, the computer includes a microcomputer built into dedicated hardware, and a general-purpose personal computer, for example, that can execute various functions by installing various programs.

[0099] FIG. 10 is a block diagram showing an example of the hardware configuration of a computer that executes a series of processes performed by the information processing device 1, the information processing device 150, etc., using a program.

[0100] In the computer, a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, and a RAM (Random Access Memory) 303 are interconnected by a bus 304.

[0101] An input / output interface 305 is also connected to the bus 304. An input unit 306, an output unit 307, a storage unit 308, a communication unit 309, and a drive 310 are connected to the input / output interface 305.

[0102] The input unit 306 includes a keyboard, mouse, microphone, touch panel, input terminal, etc. The output unit 307 includes a display, speaker, output terminal, etc. The storage unit 308 includes a hard disk, SSD (Solid State Drive), RAM disk, non-volatile memory, etc. The communication unit 309 includes a network interface, etc. The drive 310 drives a removable recording medium 311 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0103] In a computer configured as described above, the CPU 301 performs the above-described series of processes by, for example, loading a program stored in the storage unit 308 into the RAM 303 via the input / output interface 305 and the bus 304 and executing the program. The RAM 303 also stores data and the like necessary for the CPU 301 to execute various processes.

[0104] The program executed by the computer (CPU 301) can be provided by being recorded on a removable recording medium 311 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0105] In a computer, the program can be installed in the storage unit 308 via the input / output interface 305 by inserting the removable recording medium 311 into the drive 310. The program can also be received by the communication unit 309 via a wired or wireless transmission medium and installed in the storage unit 308. Alternatively, the program can be installed in the ROM 302 or the storage unit 308 in advance.

[0106] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0107] In this specification, the steps described in the flowcharts may be performed in chronological order in the order described, but they do not necessarily have to be processed in chronological order, and may be performed in parallel or at any necessary timing, such as when a call is made.

[0108] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0109] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the technology of the present disclosure.

[0110] For example, it is possible to adopt a configuration in which all or part of the above-described device configurations are combined as appropriate.

[0111] For example, the technology of the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0112] Each step described in the above flowchart can be executed by one device or can be shared and executed by multiple devices. Furthermore, if one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0113] The effects described in this specification are merely examples and are not limiting, and there may be effects other than those described in this specification.

[0114] The technology disclosed herein may employ the following configuration: (1) An information processing device comprising: a new modal generation unit that generates a new modal of N channels (N > 1) represented by a modal transformation formula combining modal data of a modal of P channels (P > 1) with a predetermined operation; and a learning unit that trains a learning model using the new modal of N channels and its training data set, wherein the new modal generation unit generates a next new modal of N channels that is more useful than the new modal of N channels, and the learning unit trains the learning model using the next new modal of N channels and its training data set. (2) The information processing device described in (1), wherein the modal transformation formula is determined by a combination of the predetermined operations selected from a plurality of operations without human intervention. (3) The information processing device according to (1) or (2), wherein the new modal generation unit generates a new modal for the next N channels that is highly beneficial, and the learning unit uses the generated new modal for the next N channels and its learning dataset to train the learning model, repeatedly until a predetermined convergence condition is met. (4) The information processing device according to any of (1) to (3), wherein the new modal generation unit generates a plurality of new modals for the N channels and outputs the most beneficial new modal for the N channels from among the plurality of new modals for the N channels as the next new modal for the N channels. (5) The information processing device according to any of (1) to (4), wherein the new modal generation unit acquires candidate modal initial setting information including the number of channels P, and determines the modal for the P channels based on the candidate modal initial setting information. (6) The information processing device according to any one of (1) to (5), wherein the new modal generation unit changes at least one of the modals of the P channel and generates a new modal of the next N channel using the changed modal of the P channel. (7) The information processing device according to any one of (1) to (6), wherein the learning unit acquires learning model information and sets the modal transformation formula and a learning data set in the structure of the learning model specified by the learning model information to learn the learning model.(8) The information processing device according to any one of (1) to (7), wherein the usefulness is defined using the amount of noise in the image. (9) The information processing device according to any one of (1) to (8), wherein the P-channel modal data includes image data obtained by at least one of an RGB sensor, a monochrome sensor, a polarization sensor, a depth sensor, or a spectral sensor. (10) An information processing method comprising: generating a new N-channel (N > 1) modal represented by a modal transformation formula combining a predetermined operation with modal data of a P-channel (P > 1) modal; and training a learning model using the N-channel new modal and a training data set thereof, wherein the information processing method generates a next N-channel new modal that is more useful than the N-channel new modal; and training a learning model using the next N-channel new modal and the training data set thereof. (11) A program for causing a computer to execute the process of: generating a new N-channel (N>1) modal represented by a modal transformation formula combining modal data of a P-channel (P>1) modal with a predetermined operation; and training a learning model using the N-channel new modal and its learning data set, generating a next N-channel new modal that is more useful than the N-channel new modal, and training a learning model using the next N-channel new modal and its learning data set. (12) An information processing device comprising: an estimation unit that generates a new N-channel (N>1) modal represented by a modal transformation formula combining modal data of a P-channel (P>1) modal with a predetermined operation, and repeats the process of training a learning model using the N-channel new modal and its learning data set, and performs estimation using the learning model determined by the learning device using modal data of the P-channel modal corresponding to the input modal configuration determined by the learning device to search for an input modal configuration that is more useful.

[0115] REFERENCE SIGNS LIST 1 Information processing device, 10 Learning data acquisition unit, 20 Modal configuration optimization unit, 30 Modal learning unit, 40 Input unit, 51 Measurement unit, 52 CG rendering unit, 53 Ground truth generation unit, 71 Baseline modal setting unit, 72 New modal generation unit, 81 Baseline modal learning unit, 82 Model output unit, 83 Convergence determination unit, 110 Modal acquisition unit, 120 Model estimation unit, 150 Information processing device, 200 Data acquisition device design unit, 302 ROM, 304 Bus, 305 Input / output interface, 306 Input unit, 307 Output unit, 308 Storage unit, 309 Communication unit, 310 Drive, 311 Removable recording medium

Claims

1. An information processing device comprising: a new modal generation unit that generates a new modal of N channels (N>1) represented by a modal transformation formula that combines modal data of a modal of P channels (P>1) with a predetermined calculation; and a learning unit that uses the new modal of N channels and its learning data set to train a learning model, wherein the new modal generation unit generates a new modal of the next N channels that is more useful than the new modal of the N channels, and the learning unit trains the learning model using the next new modal of N channels and its learning data set.

2. The information processing device according to claim 1, wherein the modal transformation formula is determined by a combination of the predetermined operations selected from a plurality of operations without manual intervention.

3. The information processing device according to claim 1, wherein the new modal generation unit generates a new modal of the next N channels that is highly useful, and the learning unit uses the generated new modal of the next N channels and its learning data set to train the learning model, are repeated until a predetermined convergence condition is met.

4. The information processing device according to claim 1, wherein the new modal generation unit generates a plurality of new modals for the N channels, and outputs the most useful new modal for the N channels from among the plurality of new modals for the N channels as the next new modal for the N channels.

5. The information processing device according to claim 1, wherein the new modal generation unit acquires candidate modal initial setting information including the number of channels P, and determines the modal of the P channels based on the candidate modal initial setting information.

6. The information processing device according to claim 1, wherein the new modal generation unit changes at least one of the modals of the P channel and generates a new modal of the next N channel using the P channel modal after the change.

7. The information processing device according to claim 1, wherein the learning unit acquires learning model information, sets the modal transformation formula and the learning data set in the structure of the learning model specified by the learning model information, and performs learning of the learning model.

8. The information processing device according to claim 1, wherein the usefulness is defined using the amount of noise in the image.

9. The information processing device according to claim 1, wherein the modal data of the P-channel modal includes image data obtained by at least one of an RGB sensor, a monochrome sensor, a polarization sensor, a depth sensor, or a spectral sensor.

10. An information processing method comprising: generating a new N-channel (N>1) modal represented by a modal transformation formula combining modal data of a P-channel (P>1) modal with a predetermined operation; and training a learning model using the N-channel new modal and its training data set; generating a next N-channel new modal that is more useful than the N-channel new modal; and training a learning model using the next N-channel new modal and its training data set.

11. A program for causing a computer to execute the following processes: generating a new N-channel (N>1) modal represented by a modal transformation formula combining modal data of a P-channel (P>1) modal with a predetermined operation; training a learning model using the new N-channel modal and its training data set; generating a next new N-channel modal that is more useful than the new N-channel modal; and training a learning model using the next new N-channel modal and its training data set.

12. An information processing device having an estimation unit that generates a new N-channel (N>1) modal represented by a modal transformation formula combining modal data of a P-channel (P>1) modal with a predetermined calculation, and searches for an input modal configuration that is highly useful by repeating the process of learning a learning model using the new N-channel modal and its learning data set, and performs estimation using the learning model determined by the learning device using the modal data of the P-channel modal corresponding to the input modal configuration determined by the learning device.

Images