Information processing apparatus, information processing method, and program
The information processing device uses a trained model with diverse neural networks to enhance camera optical system parameter inference accuracy and correct image distortions by employing device-specific corrections.
Patent Information
- Application Number
- JP2024141894
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2026-02-05
AI Technical Summary
Existing techniques for inferring camera optical system parameters face accuracy issues, necessitating improvements in parameter inference accuracy.
An information processing device utilizing a trained model with multiple neural networks of different structures, including a transformer-type encoder and a convolutional neural network, to extract features from image data and infer optical system parameters, with correction mechanisms based on device identification and averaging.
Enhances the accuracy of inferring camera optical system parameters and improves image correction by leveraging features from different perspectives and accounting for device-specific variations.
Smart Images
Figure 2026019957000001_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.
Background Art
[0002] Techniques for inferring parameters of an optical system of a camera using Deep Learning are known (see, for example, Non-Patent Document 1).
Prior Art Documents
Non-Patent Documents
[0003]
Non-Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] In the prior art, depending on the input image, it may not be possible to obtain the inference accuracy of the parameters, and there is room for improvement in the inference of the parameters of the optical system of the camera.
[0005] The present invention has been made in view of these points, and aims to improve the accuracy of inferring parameters of a camera's optical system. [Means for solving the problem]
[0006] An information processing device according to a first aspect of the present invention includes an acquisition unit that acquires calculation target image data, which is image data captured by an imaging device and is image data for which parameters of the optical system of the imaging device are to be calculated; a memory unit that stores a trained model including: (1) a first neural network that receives the calculation target image data as input and outputs a first feature that is a feature of an image represented by the calculation target image data; (2) a second neural network that receives the calculation target image data as input and outputs a second feature that is a feature of the image represented by the calculation target image data, the second neural network being different from the first neural network; and (3) an output network that is a neural network that receives the first feature and the second feature as input and outputs parameters of the optical system of the imaging device that captured the calculation target image data; and an inference unit that inputs the calculation target image data to the trained model and outputs parameters corresponding to the calculation target image data.
[0007] The first neural network may be configured to have a transformer type encoder, and the second neural network may be configured to have a convolutional neural network (CNN) type encoder.
[0008] The acquisition unit may acquire multiple pieces of calculation target image data captured by the same imaging device, and also acquire correction target image data that is image data to be corrected, the inference unit inputs each of the multiple pieces of calculation target image data into the trained model and outputs parameters corresponding to each of the multiple pieces of calculation target image data, and the information processing device may further be configured to have a correction unit that corrects the correction target image data based on an average value of the parameters corresponding to each of the multiple pieces of calculation target image data.
[0009] The acquisition unit (1) acquires the calculation target image data and imaging device identification information for identifying an imaging device, the imaging device identification information being the imaging device that captured the calculation target image data, in association with each other, and (2) acquires the correction target image data, which is image data to be corrected, and the imaging device identification information of the imaging device that captured the correction target image data, in association with each other. The inference unit stores in a storage unit a parameter table that associates the imaging device identification information of the imaging device that captured the calculation target image data with parameters output based on the calculation target image data associated with the imaging device identification information. The information processing device may further be configured to have a correction unit that corrects the correction target image data based on parameters associated with the imaging device identification information that corresponds in the parameter table to the imaging device identification information of the imaging device that captured the correction target image data.
[0010] The parameters of the optical system of the imaging device may include a distortion coefficient and a focal length of the optical system, and the correction unit may correct distortion of the correction target image data caused by the optical system of the imaging device. An information processing method of a second aspect of the present invention includes the steps of: acquiring calculation target image data, which is image data captured by an imaging device and is image data for which parameters of the optical system of the imaging device are to be calculated, executed by a computer; and inputting the calculation target image data into a trained model having: (1) a first neural network that receives the calculation target image data as input and outputs a first feature that is a feature of an image represented by the calculation target image data; (2) a second neural network that receives the calculation target image data as input and outputs a second feature that is a feature of the image represented by the calculation target image data, the second neural network being different from the first neural network; and (3) an output network that is a neural network that receives the first feature and the second feature as input and outputs parameters of the optical system of the imaging device that captured the calculation target image data, and outputting parameters corresponding to the calculation target image data.
[0011] A program according to a third aspect of the present invention causes a computer to execute the following steps: acquiring calculation target image data, which is image data captured by an imaging device and is image data for which parameters of the optical system of the imaging device are to be calculated; and inputting the calculation target image data into a trained model having: (1) a first neural network that receives the calculation target image data as input and outputs a first feature that is a feature of an image represented by the calculation target image data; (2) a second neural network that receives the calculation target image data as input and outputs a second feature that is a feature of the image represented by the calculation target image data, the second neural network being different from the first neural network; and (3) an output network that is a neural network that receives the first feature and the second feature as input and outputs parameters of the optical system of the imaging device that captured the calculation target image data, and outputting parameters corresponding to the calculation target image data. [Effects of the Invention]
[0012] According to the present invention, it is possible to improve the accuracy of inferring the parameters of the camera's optical system. [Brief explanation of the drawings]
[0013] [Figure 1] 1 is a diagram for explaining an overview of an information processing system S according to an embodiment. [Figure 2] FIG. 1 is a diagram for explaining the structure of a trained model M. [Figure 3] FIG. 1 is a diagram for explaining the structure of a trained model M. [Figure 4] 1 is a block diagram showing a configuration of an information processing device 1. FIG. [Figure 5] 3 is a diagram showing an example of the data structure of a parameter table stored in a storage unit 12. FIG. [Figure 6] 3 is a flowchart showing the flow of processing in the information processing device 1. DETAILED DESCRIPTION OF THE INVENTION
[0014] [Outline of Information Processing System S] 1 is a diagram illustrating an overview of an information processing system S according to an embodiment. The information processing system S is a system for inferring parameters of an optical system of an imaging device that captures an image. The information processing system S includes an information processing device 1 and an information terminal 2.
[0015] The information processing device 1 is a device for inferring parameters of the optical system of an imaging device that captured image data. As an example, the information processing device 1 infers parameters related to image distortion. The information processing device 1 corrects the image based on the inferred parameters. When image data is input, the information processing device 1 stores a trained model that outputs parameters of the optical system of the imaging device that captured the image. Examples of parameters of the optical system of the imaging device that the trained model infers are the distortion coefficient and focal length. Parameters of the imaging device that the trained model infers may be, for example, the optical center or shear coefficient.
[0016] The information terminal 2 is a terminal used by a user of the information processing system S. The information terminal 2 inputs image data for learning or inference to the information processing device 1. The information terminal 2 also instructs the information processing device 1 to perform learning or inference. The information terminal 2 acquires image data corrected by the information processing device 1 and displays the acquired image data.
[0017] The information processing device 1 acquires image data for training. The image data for training is image data for training a trained model. The structure of the trained model M will be described with reference to FIG. 2.
[0018] The trained model M has a first neural network NA, a second neural network NB, and an output network NC. The first neural network NA receives calculation target image data as input and outputs a first feature amount that is a feature amount of an image represented by the calculation target image data. The second neural network NB receives calculation target image data as input and outputs a second feature amount that is a feature amount of an image represented by the calculation target image data.
[0019] The first neural network NA and the second neural network NB have different structures. Specifically, the first neural network NA has a Transformer-type encoder, and the second neural network NB has a CNN (Convolutional Neural Network)-type encoder. As an example, the first neural network NA may be the known ConvFormer, and the second neural network NB may be the known ConvNeXt-V2.
[0020] By extracting features using multiple neural networks with different structures, it becomes possible to capture various features in image data, which is expected to improve the accuracy of parameter inference.In addition, by adopting a transformer-type encoder with an attention mechanism in one of the neural networks to extract features, it becomes possible to infer parameters based on information from distant positions in the image, which is expected to result in improved parameter inference accuracy.
[0021] The output network NC is a neural network that receives the first feature amount and the second feature amount as input and outputs parameters of the optical system of the imaging device that captured the calculation target image data. As an example, the output network NC may be configured to receive, as input, a vector that concatenates the first feature amount and the second feature amount.
[0022] 2 shows an example in which the output network NC is configured with one neural network, but when there are multiple parameters to be output, the output network NC may have an independent network for each parameter to be output. That is, the trained model M may have output networks (NC1 and NC2) for each of the parameters to be output (parameter A and parameter B) as shown in FIG.
[0023] 2 and 3 show an example in which two types of parameters are output, but this is not limiting. That is, the trained model M may output one type of parameter, or three or more types of parameters. Below, the information processing system S will be described using an example in which the parameters output by the trained model M are a focal length and a distortion coefficient.
[0024] Returning to FIG. 1, an overview of the processing of the information processing system S will be described. The information processing device 1 acquires training image data from the information terminal 2 ((1) in FIG. 1). The training image data is image data used for training the trained model M. The training image data is associated with parameters of the imaging device that captured the training image data.
[0025] The information processing device 1 performs learning using the learning image data and the parameters of the imaging device that captured the learning image data as training data, and generates a trained model M ((2) in FIG. 1). The information processing device 1 stores the generated trained model M.
[0026] The information processing device 1 acquires calculation target image data from the information terminal 2 ((3) in FIG. 1). The target image data is image data captured by an imaging device for which parameters of the optical system of the imaging device are to be calculated. The information processing device 1 inputs the calculation target image data into the trained model M, and outputs parameters ((4) in FIG. 1). The information processing device 1 may determine parameters to be used for image correction based on multiple sets of target image data captured by the imaging device, as will be described later.
[0027] The information processing device 1 acquires correction target image data from the information terminal 2 ((5) in FIG. 1). The correction target image data is image data that is the target of image correction. The correction target image data is captured by the same imaging device that captured the calculation target image data.
[0028] The information processing device 1 corrects the correction target image data based on the parameters output by the trained model M based on the calculation target image data ((6) in FIG. 1). More specifically, the information processing device 1 corrects the distortion of the correction target image data based on the focal length and distortion coefficient output by the trained model M, and generates corrected image data.
[0029] The information processing device 1 outputs the corrected image data ((7) in FIG. 1). As an example, the information processing device 1 causes the display unit of the information terminal 2 to display the corrected image data.
[0030] The trained model M is configured to extract features from different perspectives by extracting features using neural networks with multiple different structures. Therefore, features from different perspectives can be used to estimate the parameters of the optical system, so the information processing system S can improve the accuracy of inferring the parameters of the camera's optical system, and as a result, the accuracy of image correction can be improved.
[0031] In the above description, an example has been described in which the information processing device 1 acquires image data from the information terminal 2, but this is not limiting. The information processing device 1 may acquire image data from an imaging device (not shown) or from an external device (not shown).
[0032] 4 is a block diagram showing the configuration of the information processing device 1. The information processing device 1 has a communication unit 11, a storage unit 12, and a control unit 13. The control unit 13 has an acquisition unit 131, an inference unit 132, a display control unit 133, and a correction unit 134.
[0033] The communication unit 11 is a communication interface for transmitting and receiving data to and from other devices via a network. The storage unit 12 is a storage medium including a ROM (Read Only Memory), a RAM (Random Access Memory), an SSD (Solid State Drive), a hard disk drive, etc. The storage unit 12 stores in advance a program to be executed by the control unit 13. The storage unit 12 stores a trained model M.
[0034] The control unit 13 is a processor such as a CPU (Central Processing Unit), etc. The control unit 13 executes a program stored in the storage unit 12, thereby functioning as an acquisition unit 131, an inference unit 132, a display control unit 133, and a correction unit 134.
[0035] The acquisition unit 131 acquires the calculation target image data. The acquisition unit 131 may acquire the calculation target image data from the information terminal 2, or may acquire the calculation target image data from an external device (not shown).
[0036] The inference unit 132 inputs the calculation target image data into the trained model M, and causes the model M to output parameters corresponding to the calculation target image data. The inference unit 132 inputs the calculation target image data into the trained model M, and causes the model M to output the focal length and distortion coefficient of the imaging device that captured the calculation target image data. As an example, the inference unit 132 stores the focal length and distortion coefficient output by the trained model M in the memory unit 12. The display control unit 133 may cause the information terminal 2 to display a screen that displays the parameters output by the trained model M.
[0037] By configuring the information processing device 1 to infer parameters of the optical system using a trained model M configured to extract features from different perspectives using multiple neural networks with different structures, the accuracy of inferring parameters of the camera's optical system can be improved.
[0038] [Correction processing] The acquisition unit 131 acquires correction target image data. The correction unit 134 corrects distortion of the correction target image data caused by the optical system of the imaging device. Specifically, the correction unit 134 corrects the correction target image data based on the focal length and distortion coefficient output by the trained model M, and generates corrected image data. Techniques for correcting images based on the focal length and distortion coefficient of the imaging device are known, and the correction unit 134 may correct the correction target image data based on known camera calibration techniques. The display control unit 133 may cause the information terminal 2 to display a screen that displays the corrected image data generated by the correction unit 134.
[0039] Depending on the input image data, errors may occur in the parameters output by the trained model M. Therefore, the information processing device 1 may be configured to correct the image based on the average value of the parameters output by the trained model M based on image data captured by the same imaging device. This is because if the imaging device is the same, it can be assumed that the parameters of the optical system to be inferred will also be the same.
[0040] The acquisition unit 131 acquires multiple calculation target image data captured by the same imaging device. The inference unit 132 inputs each of the multiple calculation target image data into the trained model M, and outputs parameters corresponding to each of the multiple calculation target image data. Specifically, the inference unit 132 calculates the average values of the focal lengths and distortion coefficients output by the trained model M based on each of the multiple calculation target image data.
[0041] The correction unit 134 corrects the correction target image data based on the average values of the parameters corresponding to each of the plurality of calculation target image data. Specifically, the correction unit 134 corrects the correction target image data acquired by the acquisition unit 131 based on the average values of the focal length and the distortion coefficient calculated by the inference unit 132.
[0042] It is assumed that abnormal values may be included in the parameters output by the trained model M. Therefore, the correction unit 134 may be configured to remove statistically abnormal values and take the average value of the parameters.
[0043] The inference unit 132 calculates the average value and standard deviation of the parameters output by the trained model M based on multiple pieces of calculation target image data. The inference unit 132 calculates the average value based on the parameters output by the trained model M, excluding parameters whose standard deviation is equal to or greater than a predetermined threshold value or equal to or less than a threshold value.
[0044] By configuring the information processing device 1 in this way based on the average value of the parameters output by the trained model M, it is possible to improve the accuracy of estimating the parameters of the optical system.
[0045] Even if imaging devices are manufactured using the same types of parts in the same process, errors may occur in the parameters when the parts are assembled. Therefore, the information processing device 1 may be configured to identify parameters for each imaging device and correct images captured by each imaging device based on the parameters for that imaging device.
[0046] The acquisition unit 131 acquires the calculation target image data in association with the imaging device ID of the imaging device that captured the calculation target image data. The imaging device ID is identification information for identifying the imaging device. As an example, the imaging device ID is included in the metadata associated with the calculation target image data.
[0047] The inference unit 132 inputs the acquired calculation target image data into the trained model M and outputs parameters. The inference unit 132 stores in the storage unit 12 a parameter table that associates the imaging device ID of the imaging device that captured the calculation target image data with the parameters output based on the calculation target image data associated with the imaging device ID.
[0048] Fig. 5 is a diagram showing an example of the data structure of a parameter table stored in the storage unit 12. The parameter table shown in Fig. 5 has at least "imaging device ID", "focal length", and "distortion coefficient".
[0049] The acquisition unit 131 acquires the correction target image data in association with the imaging device ID of the imaging device that captured the correction target image data. As an example, the imaging device ID is included in the metadata associated with the correction target image data.
[0050] The correction unit 134 corrects the image data to be corrected based on parameters associated with an imaging device ID that corresponds in the parameter table to the imaging device ID of the imaging device that captured the image data to be corrected. The correction unit 134 refers to the parameter table stored in the storage unit 12 and identifies parameters that correspond to the imaging device ID associated with the image to be corrected. The correction unit 134 performs processing to correct the image data to be corrected based on the identified parameters.
[0051] By configuring the information processing device 1 to correct images using parameters calculated for the imaging device that captured the correction target image data, it is possible to correct images captured by the same type of device based on individually different parameters. In particular, when correcting images captured by a large number of the same type of imaging devices, it is possible to improve the accuracy of correcting image distortion caused by variations between the imaging devices.
[0052] [Processing flow in information processing device 1] Fig. 6 is a flowchart showing the flow of processing in the information processing device 1. The flowchart shown in Fig. 6 starts from the point when learning of the trained model M is completed and it becomes possible to accept calculation target image data.
[0053] The acquisition unit 131 acquires calculation target image data (S01). The inference unit 132 inputs the calculation target image data to the trained model M, and causes the trained model M to output parameters of the imaging device that captured the calculation target image data (S02).
[0054] The acquisition unit 131 acquires the correction target image data (S03). The correction unit 134 identifies the parameters of the imaging device that captured the correction target image data (S04). The correction unit 134 corrects the correction target image data based on the identified parameters (S05). Then, the information processing device 1 ends the process.
[0055] [Effects of information processing device 1] As described above, by inferring the parameters of the optical system using a trained model configured to extract features using multiple neural networks with different structures, the information processing device 1 can improve the accuracy of inferring the parameters of the camera's optical system.
[0056] The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by functionally or physically distributing or integrating any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination also have the effects of the original embodiments. [Explanation of symbols]
[0057] 1. Information processing equipment 2. Information terminal 11 Communications Department 12 Storage section 13 Control Unit 131 Acquisition Department 132 Reasoning part 133 Display control unit 134 Correction Unit
Claims
1. an acquisition unit that acquires calculation target image data, which is image data captured by an imaging device and is image data that is a target for calculating parameters of an optical system of the imaging device; a memory unit for storing a trained model including: (1) a first neural network that receives the calculation target image data as input and outputs a first feature amount that is a feature amount of an image represented by the calculation target image data; (2) a second neural network that receives the calculation target image data as input and outputs a second feature amount that is a feature amount of an image represented by the calculation target image data, the second neural network being different from the first neural network; and (3) an output network that is a neural network that receives the first feature amount and the second feature amount as input and outputs parameters of an optical system of an imaging device that captured the calculation target image data; an inference unit that inputs the calculation target image data into the trained model and outputs parameters corresponding to the calculation target image data; An information processing device having the above.
2. the first neural network has a transformer-type encoder; The second neural network has a CNN (Convolutional Neural Network) type encoder. The information processing device according to claim 1 .
3. the acquisition unit acquires a plurality of pieces of calculation target image data captured by the same imaging device, and also acquires correction target image data that is image data to be corrected; The inference unit inputs each of the plurality of calculation target image data into the trained model and outputs parameters corresponding to each of the plurality of calculation target image data; the information processing device further includes a correction unit that corrects the correction target image data based on an average value of parameters corresponding to each of the plurality of calculation target image data; The information processing device according to claim 1 .
4. The acquisition unit (1) The calculation target image data and imaging device identification information for identifying an imaging device are acquired in association with each other, the imaging device identification information being the imaging device that captured the calculation target image data; and (2) acquiring correction target image data, which is image data to be corrected, in association with imaging device identification information of an imaging device that captured the correction target image data; the inference unit stores in a storage unit a parameter table that associates imaging device identification information of an imaging device that captured the calculation target image data with parameters output based on the calculation target image data associated with the imaging device identification information; the information processing device further includes a correction unit that corrects the correction target image data based on parameters associated with imaging device identification information that corresponds in the parameter table to imaging device identification information of an imaging device that captured the correction target image data. The information processing device according to claim 1 .
5. the parameters of the optical system of the imaging device include a distortion coefficient and a focal length of the optical system; the correction unit corrects distortion of the correction target image data caused by an optical system of the imaging device.
5. The information processing device according to claim 3 or 4.
6. The computer executes acquiring calculation target image data, which is image data captured by an imaging device and is image data for which parameters of an optical system of the imaging device are to be calculated; (1) a first neural network that receives the calculation target image data as an input and outputs a first feature amount that is a feature amount of an image represented by the calculation target image data; (2) a second neural network that receives the calculation target image data as an input and outputs a second feature amount that is a feature amount of an image represented by the calculation target image data, the second neural network being different from the first neural network; and (3) an output network that is a neural network that receives the first feature amount and the second feature amount as input and outputs parameters of an optical system of an imaging device that captured the calculation target image data, and a step of inputting the calculation target image data into the trained model and outputting parameters corresponding to the calculation target image data; An information processing method comprising:
7. On the computer, acquiring calculation target image data, which is image data captured by an imaging device and is image data for which parameters of an optical system of the imaging device are to be calculated; (1) a first neural network that receives the calculation target image data as an input and outputs a first feature amount that is a feature amount of an image represented by the calculation target image data; (2) a second neural network that receives the calculation target image data as an input and outputs a second feature amount that is a feature amount of an image represented by the calculation target image data, the second neural network being different from the first neural network; and (3) an output network that is a neural network that receives the first feature amount and the second feature amount as input and outputs parameters of an optical system of an imaging device that captured the calculation target image data, and a step of inputting the calculation target image data into the trained model and outputting parameters corresponding to the calculation target image data; A program that executes the following.