Information processing device, parameter derivation method, image processing device, and parameter setting method

Abstract

An information processing device according to the present technology comprises: an inverse processing unit that infers a pre-processed image, which is an image prior to image processing, on the basis of a processed image, which is an image after image processing by an image processing unit, and motion detection information transmitted by an image processing device, the image processing device being provided with a motion detection unit that detects motion of a host device and the image processing unit that performs image processing, on the basis of motion detection information from the detection unit, on the image to be processed; and a parameter derivation unit that derives a parameter to be set in the image processing unit on the basis of a result of image processing trials using a candidate parameter for the pre-processed image inferred by the inverse processing unit.

Description

Information processing device, parameter derivation method, image processing device, and parameter setting method

[0001] The present technology relates to an information processing device, a parameter derivation method, an image processing device, and a parameter setting method, and in particular to a technology for performing appropriate parameter adjustment for an image processing unit that performs image processing based on motion detection information of the device.

[0002] Known image processing techniques include a technique for processing a target image based on motion detection information obtained by a motion detection unit such as an IMU (Inertial Measurement Unit). A typical example of this is image processing as an Electric Image Stabilizer (EIS; hereinafter referred to as "stabilization processing"), which electronically removes vibration components caused by camera shake or the like in an imaging device.

[0003] Here, it is conceivable that image processing parameters are adjusted so that the processed image satisfies a predetermined standard. For example, in the case of the stabilization processing described above, there are users who want to remove as much vibration component as possible, users who want to tolerate a certain degree of vibration, and users who want to add vibration because it is insufficient. In this case, the parameters can be adjusted to obtain image processing characteristics that meet the preferences of the users.

[0004] As a related prior art, the following Patent Document 1 can be cited: Patent Document 1 discloses a technique for automatically adjusting control parameters of an electric motor, in which appropriate control parameters are derived based on operating characteristic data obtained when the electric motor is operated using a plurality of control parameters.

[0005] JP 2016-19304 A

[0006] In parameter adjustment, it is conceivable to perform trial image processing for parameter derivation using an image that the image processing device actually uses for image processing, taking into consideration the actual usage environment of the image processing device that executes image processing. In this case, the information processing device that performs parameter derivation acquires the image that will actually be used for image processing, i.e., the pre-processed image before image processing, from the image processing device, and performs image processing on the acquired pre-processed image to obtain an evaluation image for parameter derivation.

[0007] However, there may be some circumstances where the image processing device cannot acquire the unprocessed image from the information processing device. For example, when communication between the image processing device and the information processing device is performed using a predetermined wireless communication band and the image processing device is configured to transmit a processed image, which is an image after image processing, to the outside using the wireless communication band, there may be circumstances where the bandwidth is insufficient and the unprocessed image cannot be transmitted separately from the processed image.

[0008] This technology has been developed in consideration of the above circumstances, and aims to enable appropriate parameter adjustment for image processing in an image processing device even when the information processing device that derives parameters is unable to obtain a pre-processed image from the image processing device.

[0009] The information processing device according to the present technology includes: an image processing device including a motion detection unit that detects motion of the device itself; and an image processing unit that performs image processing on an image to be processed based on motion detection information by the motion detection unit; an inverse processing unit that estimates a pre-processed image that is the image before the image processing based on the motion detection information and a processed image that is the image after image processing by the image processing unit, and the motion detection information, and the image processing device includes: an image processing device including: a motion detection unit that detects motion of the device itself;

[0010] In addition, an image processing device according to the present technology includes a motion detection unit that detects the motion of the device itself, an image processing unit that performs image processing on an image to be processed based on motion detection information from the motion detection unit, a transmission processing unit that performs processing to transmit to an external device a processed image, which is the image after image processing by the image processing unit, and the motion detection information, and a setting processing unit that performs processing to set parameters derived based on the results of trial image processing using candidate parameters on a pre-processed image, which is the image before the image processing estimated based on the processed image and the motion detection information, to the image processing unit. By performing the processing to transmit the processed image and the motion detection information to the external device as described above, the information processing device as the external device can adopt a technique in which the pre-processed image estimated from the processed image is used to derive parameters, thereby eliminating the need for the image processing device to acquire the pre-processed image from the image processing device when deriving parameters. Furthermore, the setting processing unit allows the image processing device to perform image processing using the derived parameters.

[0011] 16 is a block diagram showing an example configuration of a parameter adjustment system as an embodiment according to the present technology. FIG. 17 is a block diagram showing an example configuration of an image processing device as an embodiment. FIG. 18 is a block diagram showing an example configuration of an information processing device as an embodiment. FIG. 19 is a diagram for explaining an example algorithm of stabilization processing in an embodiment. FIG. 19 is a diagram illustrating an example of a lattice point mesh projected from a virtual celestial sphere onto an image plane of a captured image. FIG. 19 is an explanatory diagram of a segment matrix. FIG. 19 is a diagram showing a segment matrix and a lattice point mesh superimposed on an image plane. FIG. 19 is an explanatory diagram of an example technique for determining reference coordinates of segment representative points. FIG. 19 is an explanatory diagram of interpolation processing performed when rendering a stabilized image. FIG. 19 is a block diagram showing an example configuration of a stabilizer processing unit included in an image processing device as an embodiment. FIG. 19 is an explanatory diagram of functions possessed by an image processing device for realizing a parameter adjustment method as an embodiment. FIG. 19 is an explanatory diagram of functions possessed by an information processing device for realizing a parameter adjustment method as an embodiment. FIG. 19 is an explanatory diagram of an example functional configuration of an inverse stabilizer processing unit. FIG. 19 is a diagram showing an example GUI for accepting an instruction operation for image characteristic conditions. FIG. 19 is an explanatory diagram of the camera center (image capture center) and the image frame center. FIG. 19 is a flowchart showing an example processing procedure of an information processing device for realizing the parameter adjustment method as an embodiment. FIG. 19 is a flowchart showing details of the parameter derivation processing shown in FIG.

[0012] Hereinafter, with reference to the accompanying drawings, embodiments according to the present technology will be described in the following order: <1. Parameter adjustment system as an embodiment> [1-1. System configuration example] [1-2. Configuration example of image processing device] [1-3. Configuration example of information processing device] [1-4. Stabilization processing] <2. Parameter adjustment method as an embodiment> <3. Processing procedure> <4. Modification example> <5. Summary of embodiment> <6. Present technology>

[0013] 1. Parameter Adjustment System as an Embodiment 1-1. System Configuration Example FIG. 1 is a block diagram showing a configuration example of a parameter adjustment system as an embodiment according to the present technology. The parameter adjustment system is configured to include at least an imaging device 1 and an information processing device 2. The imaging device 1 is an embodiment of an image processing device according to the present technology, and includes a motion detection unit that detects the motion of the imaging device itself, and an image processing unit that performs image processing on an image to be processed based on motion detection information from the motion detection unit. Although not shown here, the imaging device 1 includes an IMU (Inertial Measurement Unit) as the motion detection unit, and performs image processing on the captured image based on the motion detection information from the IMU as an electric image stabilizer (EIS) that removes vibration components through electronic processing. Hereinafter, the EIS processing will be referred to as "stabilization processing." For ease of explanation, "stabilization processing" will sometimes be abbreviated to "stabilization processing."

[0014] The information processing device 2 is an embodiment of an information processing device according to the present technology, and is configured as a computer device including, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc., and performs processing to derive parameters to be set in an image processing unit in the image processing device. Specifically, in this example, processing to derive parameters for stabilize processing in the imaging device 1 is performed.

[0015] Here, in this embodiment, a system is assumed in which images captured by an imaging device 1 attached to a moving object M, specifically, captured images after stabilization processing, are transmitted from the imaging device 1 to an external device. Specifically, this example assumes a system in which onboard images of a motorcycle or car race are transmitted from each imaging device 1 as content images for broadcast or distribution. In this case, the moving object M corresponds to a motorcycle or car participating in a race, and there are usually multiple moving objects M. At least one imaging device 1 is attached to each moving object M. Each imaging device 1 is capable of wireless data communication with the external device.

[0016] When applying this technology to racing scenes such as those described above, there is a demand for customizing the image characteristics of images after stabilizer processing in accordance with the user's preferences, etc. For this reason, it is desirable to adjust stabilizer processing parameters so that image characteristics that meet the user's intentions can be obtained.

[0017] Regarding parameter adjustment, it is conceivable that the user manually changes the parameters as appropriate and derives parameters that will achieve the intended image characteristics through trial and error. However, in this case, the user must be familiar with the parameters of the stabilization process, which is a burden for the user. For this reason, this embodiment adopts a method of automating the parameter derivation. Specifically, in this example, the information processing device 2 performs the parameter derivation process.

[0018] When deriving parameters in the information processing device 2, the image actually used for the stabilizer processing is obtained from the imaging device 1, i.e., the captured image before the stabilizer processing (hereinafter referred to as the "pre-processed image"), and stabilizer processing is performed on the obtained pre-processed image to obtain an evaluation image for deriving parameters.

[0019] In this case, it is conceivable that the pre-processed images are stored in the imaging device 1 while the moving body M is traveling, and are then acquired by the information processing device 2 from the imaging device 1 after the moving body M has finished traveling. However, when applied to racing, there are time constraints, such as the need to complete parameter adjustment before the start of the second practice run using pre-processed images from the first practice run, and if the information processing device 2 were to acquire pre-processed images from each imaging device 1 after the run has finished as described above, the parameter adjustment would not be completed in time.

[0020] For these reasons, in application to racing, it is desirable to transmit (wirelessly transmit) unprocessed images during the first practice run to the information processing device 2 in approximately real time while the vehicle is running. In this case, since it is desired to obtain onboard images of each moving body M as content images even during the first practice run, each imaging device 1 should transmit images after stabilizer processing as well as unprocessed images.

[0021] However, in a communication system used to transmit onboard images of a race, onboard images from multiple image capture devices 1 are transmitted using a common wireless communication band, and the wireless communication band used for image transmission is therefore constrained, making it extremely difficult to transmit unprocessed images along with stabilized images from each image capture device 1.

[0022] In this embodiment, in view of the above circumstances, the purpose is to achieve appropriate parameter adjustment for image processing in the imaging device 1 even when the information processing device 2 that performs parameter derivation is unable to obtain an unprocessed image from the imaging device 1.

[0023] 2 is a block diagram showing an example of the configuration of the imaging device 1. As shown in the figure, the imaging device 1 includes an imaging optical system 11, an image sensor 12, an image signal processing unit 13, an IMU 14, a stabilizer processing unit 15, a camera control unit 16, and a communication unit 17. The image sensor 12, the image signal processing unit 13, the IMU 14, the stabilizer processing unit 15, the camera control unit 16, and the communication unit 17 are connected via a bus 18, and are capable of mutual data communication.

[0024] The imaging optical system 11 includes lenses such as a cover lens, a zoom lens, and a focus lens, as well as an iris mechanism. Light (incident light) from a subject is guided by the imaging optical system 11 and collected on the light receiving surface of the image sensor 12.

[0025] The image sensor 12 is configured as a solid-state imaging element, such as a CCD (Charge Coupled Device) type, a CMOS (Complementary Metal Oxide Semiconductor) type, etc. The image sensor 12 has pixels, each having a photoelectric conversion element such as a photodiode, arranged two-dimensionally, and each pixel performs photoelectric conversion on incident light from a subject, thereby obtaining a captured image as digital data.

[0026] The image signal processing unit 13 performs preprocessing, synchronization, YC generation, resolution conversion, and other processes on the captured image obtained by the image sensor 12. Preprocessing includes clamping the black levels of R (red), G (green), and B (blue) to a predetermined level for the captured image, and correction between the R, G, and B color channels. Synchronization involves color separation, which ensures that image data for each pixel contains all R, G, and B color components. For example, when a Bayer array color filter is used, demosaic processing is performed as the color separation process. YC generation generates (separates) a luminance (Y) signal and a color (C) signal from the R, G, and B image data. Resolution conversion involves executing resolution conversion on image data that has undergone various signal processes.

[0027] The IMU 14 functions as a motion detection unit that detects the motion of the device itself. The IMU 14 is configured with an acceleration sensor and an angular velocity sensor. Specifically, in this example, the acceleration sensor is configured as a triaxial acceleration sensor that can detect acceleration acting on each of the X-axis, Y-axis, and Z-axis that define a three-dimensional space, and the angular velocity sensor is configured as a triaxial angular velocity sensor that can detect angular velocity based on each of the X-axis, Y-axis, and Z-axis. Hereinafter, the motion detection information of the device itself obtained by the IMU 14 will be referred to as "IMU data." In this example, the IMU data includes acceleration data for each of the X-axis, Y-axis, and Z-axis, and angular velocity data for each of the X-axis, Y-axis, and Z-axis, detected by the triaxial acceleration sensor and triaxial angular velocity sensor.

[0028] The stabilizer processing unit 15 performs stabilizer processing on the image captured by the image sensor 12, specifically, in this example, the RGB image obtained via the image signal processing unit 13, based on the IMU data obtained by the IMU 14. In this example, the stabilizer processing includes a function to remove vibration components as well as a function to maintain horizontality.

[0029] The camera control unit 16 is configured with, for example, a microcomputer having a CPU, ROM, and RAM, and the CPU executes various processes according to programs stored in the ROM or programs loaded into the RAM, thereby performing overall control of the imaging device 1. For example, the camera control unit 16 performs various operational controls, such as control related to starting and stopping the imaging operation by the image sensor 12, the image signal processing by the image signal processing unit 13, and the stabilizer processing by the stabilizer processing unit 15. The camera control unit 16 also performs various data communications with external devices via the communication unit 17. In this example, the communication unit 17 is configured to be capable of performing data communications at least with the information processing device 2. Specifically, it is configured to be capable of performing wireless data communications with the information processing device 2.

[0030] 3 is a block diagram showing an example of the configuration of an information processing device 2. As shown in the figure, the information processing device 2 includes a CPU 21. The CPU 21 functions as an arithmetic processing unit that performs various processes, and executes the various processes in accordance with a program stored in a ROM 22 or a program loaded from a storage unit 29 to a RAM 23. The RAM 23 also stores data necessary for the CPU 21 to execute the various processes, as appropriate. The CPU 21, the ROM 22, and the RAM 23 are connected to one another via a bus 24.

[0031] An input / output interface (I / F) 25 is also connected to the bus 24. An input unit 26 including an operator or an operation device is connected to the input / output interface 25. For example, the input unit 26 may be various operators or operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, or a remote controller. An operation is detected by the input unit 26, and a signal corresponding to the detected operation is interpreted by the CPU 21.

[0032] A display unit 27, such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, and an audio output unit 28, such as a speaker, are connected integrally or separately to the input / output interface 25. The display unit 27 is used to display various types of information, and is configured, for example, by a display device provided in the housing of the information processing device 2, or a separate display device connected to the information processing device 2.

[0033] The display unit 27 displays various types of information on the display screen based on instructions from the CPU 21. For example, the display unit 27 displays various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the CPU 21. The display unit 27 also displays images specified by user operations, for example, based on instructions from the CPU 21.

[0034] The input / output interface 25 may be connected to a storage unit 29 configured with a HDD or solid-state memory, or a communication unit 30 configured with a modem or the like.

[0035] The communication unit 30 performs communication processing via a transmission path such as the Internet, and communication with various devices via wired / wireless communication, bus communication, etc. In particular, the communication unit 30 in this embodiment is configured to be able to perform data communication with the imaging device 1 (communication unit 17).

[0036] A drive 31 is also connected to the input / output interface 25 as required, and a removable recording medium 32 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted thereon.

[0037] The drive 31 can read data files such as programs used in various processes from the removable recording medium 32. The read data files are stored in the storage unit 29, and images and sounds contained in the data files are output on the display unit 27 and the audio output unit 28. Furthermore, the computer programs and the like read from the removable recording medium 32 are installed in the storage unit 29 as needed.

[0038] In the information processing device 2 having the above-described hardware configuration, for example, software for the processing of this embodiment can be installed via network communication by the communication unit 30 or via the removable recording medium 32. Alternatively, the software may be stored in advance in the ROM 22, the storage unit 29, etc.

[0039] The information processing device 2 is not limited to being configured as a single computer device as shown in Fig. 3, but may be configured as a system of multiple computer devices. The multiple computer devices may be systemized using a LAN (Local Area Network) or the like, or may be located in a remote location using a VPN (Virtual Private Network) using the Internet or the like. The multiple computer devices may include computer devices as a server group (cloud) available through a cloud computing service.

[0040] [1-4. Regarding Stabilization Processing] An example algorithm for stabilization processing in the embodiment will be described with reference to Fig. 4 to Fig. 9. In the present technology, the specific algorithm for stabilization processing is not particularly limited, and various known algorithms can be adopted, but for confirmation, an example will be described here.

[0041] Basically, the stabilizer process as an EIS can be considered as a process of cutting out an output image from an input image, as illustrated in Figure 4A, and involves performing a process to identify which position in the input image the value should be referenced for each pixel in the output image.

[0042] 4B is an explanatory diagram of a lattice point mesh used in stabilization processing. The lattice point mesh has an outer frame that is the same shape and size as the image frame of the input image, and as shown in the figure, multiple dividing lines are defined within the outer frame in both the vertical and horizontal directions, and these dividing lines form multiple squares (5 x 4 = 20 squares in the example shown). In the lattice point mesh, each square is called a "mesh," and the vertices of each mesh are called "lattice points."

[0043] In the stabilization process, motion detection information by the IMU 14 is associated with such a grid point mesh. In the stabilization process of this example, quaternion data is used as the motion detection information by the IMU 14. The quaternion data represents rotation in three-dimensional space and can be calculated based on the acceleration data and angular velocity data for the X-axis, Y-axis, and Z-axis detected by the aforementioned three-axis acceleration sensor and three-axis angular velocity sensor. Essentially, in the stabilization process, reference coordinates for each pixel of the output image are calculated so that the rotation represented by such quaternion data (movement of the imaging device 1) is canceled.

[0044] It is conceivable to associate a single piece of quaternion data obtained at any timing during the exposure period, such as the start of exposure of the image to be processed for stabilization or the center of the exposure period, with the grid point mesh. However, in this example, in order to suppress rolling shutter distortion, quaternion data at multiple timings during the exposure period of the target image is associated with the grid point mesh. Specifically, in this example, quaternion data based on IMU data detected at the corresponding timing is associated with each horizontal line (five in this example) of grid points in the grid point mesh (see quaternion data q0 to q4 in the figure). This allows for improved resolution in the time direction of the stabilization process and suppression of rolling shutter distortion compared to using one piece of motion detection information per image to be processed.

[0045] In the stabilizer processing of this example, in order to correct the lens distortion in the imaging optical system 11, the lattice point mesh is deformed based on the lens distortion correction data as shown in FIG. 4C.

[0046] Furthermore, in the stabilization process of this example, the deformed lattice point mesh is projected onto a virtual celestial sphere as shown in Fig. 4D. As shown in the figure, the virtual celestial sphere is defined as a sphere whose center is the viewpoint of the image capture device 1 and whose radius is the focal length F.

[0047] In the stabilization process of this example, a grid point mesh is projected onto this virtual celestial sphere, and each grid point in the grid point mesh is then rotated on the virtual celestial sphere based on the quaternion data (three-dimensional rotation). This rotation contributes to the removal of vibration components.

[0048] After the above rotation, the lattice point mesh on the virtual celestial sphere is projected onto the image plane of the captured image (i.e., three-dimensional information is converted into two-dimensional information).

[0049] FIG. 5 illustrates an example of a lattice point mesh projected from a virtual celestial sphere onto the image plane of a captured image. In the figure, each lattice point of the projected lattice point mesh is represented by a triangle. By projecting the lattice point mesh onto the image plane of the captured image in this manner, the coordinates on the image plane are identified for each lattice point. The coordinate information of the input image is defined in the image plane to which the lattice point mesh is projected. Furthermore, as described above in FIG. 4B , the lattice point mesh is originally defined to have the same size and shape as the input image. Therefore, by performing the above projection, the amount of movement and direction of movement from the original position can be determined for each lattice point. In the figure, the difference vectors Du and Dv indicating the amount of movement and direction of movement from the original position are illustrated for the top left lattice point in the projected lattice point mesh.

[0050] When performing the above-described projection, it is possible to adjust the curvature of the output image by shifting the position of the viewpoint during projection (so-called free curvature perspective projection).

[0051] In the stabilizer process of this example, a process is performed to determine the reference coordinates for each pixel of the output image based on the coordinate information on the image plane of each grid point identified by performing the projection described above.

[0052] 6 is an explanatory diagram of a segment matrix. As with the lattice point mesh, for the output image, instead of defining squares in units of one pixel, squares consisting of multiple pixels, i.e., segments, are defined, and the reference coordinates are first calculated in units of segments.

[0053] In this example, the image size of the input image is, for example, 4k, and the image size of the output image, i.e., the overall size of the segment matrix, is, for example, FHD (Full High Definition) size. The number of segments in the segment matrix can be any number, but as an example for the purpose of explanation, the number of segments in the segment matrix is ​​assumed to be 8 x 5 = 40. Here, the center coordinates of the segment matrix coincide with the center coordinates of the input image defined in the image plane onto which the lattice point mesh is projected.

[0054] 7 shows the segment matrix and the lattice point mesh superimposed on the image plane. In the figure, the representative point of each segment (hereinafter referred to as the "segment representative point") is indicated by a ● mark. As shown in the figure, in this example, the segment representative point is the upper left vertex of the segment.

[0055] To find the reference coordinates for each segment, it is first determined in which mesh of the lattice point mesh the segment representative point exists. This determination process is called segment search processing.

[0056] The segment search process determines to which mesh in the lattice point mesh each segment representative point in the segment matrix belongs.

[0057] It should be noted here that the grid point mesh only associates coordinate information at the level of each grid point. Although the projection of the grid point mesh described in Fig. 5 can identify the positional relationship between each grid point of each mesh and the segment representative point, it does not necessarily identify the coordinates (reference coordinates) of the input image corresponding to the segment representative point.

[0058] Therefore, the reference coordinates of the segment representative point are determined based on the difference vectors Du and Dv (information indicating the amount and direction of movement from the original position) of the grid points.

[0059] 8 is an explanatory diagram of an example of a technique for determining reference coordinates for a segment representative point whose mesh has been identified by a segment search process. In the diagram, segment representative points within a mesh are represented by ● marks. First, an interpolation ratio for determining reference coordinates for the target segment representative point is calculated based on its positional relationship with the center of gravity G of the mesh identified by the segment search. For example, the interpolation ratio is determined as a set of a vertical offset amount v and a horizontal offset amount u from the center of gravity G, as shown in the diagram.

[0060] Then, using this interpolation ratio, the difference vectors Du and Dv of a lattice point (e.g., the upper left lattice point) in the mesh to which the target segment representative point belongs are interpolated. The difference vector obtained by this interpolation is referred to as the "interpolated difference vector." When the coordinates of the target segment representative point are used as the reference, the interpolated difference vector functions as information indicating the amount and direction of movement from those coordinates to the reference coordinates.

[0061] Therefore, once an interpolated differential vector is determined for a segment representative point, the reference coordinates of the segment representative point can be determined using the interpolated differential vector. Specifically, the reference coordinates of the segment representative point can be determined as coordinates obtained by offsetting the coordinates of the segment representative point in accordance with the movement direction and amount indicated by the interpolated differential vector.

[0062] The process of calculating the interpolation difference vector based on the interpolation ratio for the segment representative point as described above and determining the reference coordinate of the segment representative point based on the interpolation difference vector is called a remesh data generation process.

[0063] By performing the above remesh data generation process for each segment representative point, the reference coordinates of the four vertices of each segment in the segment matrix are determined.

[0064] In rendering a stabilized image, first, the reference coordinates of at least three of the four vertices of the segment determined as described above are used to determine the reference coordinates of each pixel (i.e., each output pixel) within the segment by linear interpolation.

[0065] Once the reference coordinates for each output pixel are determined, the value of the output pixel can be determined using the value of the pixel in the input image indicated by the reference coordinates (hereinafter referred to as the "reference pixel Pr.") In this example, the value of the output pixel is determined by two-dimensional interpolation processing using the pixel values ​​of multiple pixels in the input image near the reference coordinates.

[0066] FIG. 9 is an explanatory diagram of this interpolation process. The diagram schematically illustrates how a reference pixel Pr in an input image is identified from the reference coordinates of an output pixel (target output pixel in the diagram) to be rendered in a segment matrix. In this example, the value of the output pixel is determined by interpolation using pixel values ​​within a reference area Ar that includes the reference pixel Pr and its surrounding pixels. The reference area Ar is an area of ​​m pixels x m pixels (m is a natural number greater than or equal to 3) centered on the reference pixel Pr. Note that, while the diagram illustrates the reference area Ar as an area of ​​3 pixels x 3 pixels = 9 pixels centered on the reference pixel Pr, this is merely an example for illustrative purposes and does not limit the size of the reference area Ar.

[0067] In this case, the interpolation process may be performed using, for example, a Lanczos (Lanczos-2) filter. Specifically, a hybrid filter that blends a Lanczos filter with a Gaussian filter from the perspective of preventing aliasing may be used. This hybrid filter is effective for interpolation in a RAW format in which the image format is arranged in RGGB, and is particularly effective in preventing aliasing in the high frequency band.

[0068] In the stabilization process of this example, the above-described interpolation process is performed for each output pixel, thereby obtaining a stabilized image as an image after stabilization process.

[0069] 10 is a block diagram showing an example of the configuration of the stabilizer processing unit 15 that realizes the stabilizer processing described above. As shown in the figure, the stabilizer processing unit 15 has a memory unit 40, a lattice point mesh generator 41, a sampling unit 42, a projection unit 43, a rotation unit 44, a projection unit 45, a segment search unit 46, a remesh data generator 47, and a rendering unit 48.

[0070] A captured image is input to the memory unit 40, and the captured image is temporarily stored as an input image.

[0071] The lattice point mesh generator 41 generates the above-mentioned lattice point mesh. In this example, in order to perform lens distortion correction, the lattice point mesh generator 41 generates a lattice point mesh (see FIG. 4C ) that has been deformed based on the lens distortion correction data.

[0072] The sampling unit 42 receives the IMU data from the IMU 14. The sampling unit 42 samples the input IMU data so that each mesh row is associated with motion detection information at the corresponding timing, as described above with reference to FIG. 4B. The sampling unit 42 in this example generates quaternion data based on the sampled IMU data.

[0073] The projection unit 43 projects the lattice point mesh generated by the lattice point mesh generation unit 41 onto the virtual celestial sphere (see FIG. 4D).

[0074] The rotation unit 44 rotates each lattice point of the lattice point mesh projected onto the virtual celestial sphere by the projection unit 43 based on the motion detection information (quaternion data in this example) for each mesh row input from the sampling unit 42.

[0075] The projection unit 45 performs processing to project the lattice point mesh that has been rotated by the rotation unit 44 onto the image plane of the captured image (see FIG. 5).

[0076] The segment search unit 46 performs the segment search process described above based on the coordinate information of each lattice point of the lattice point mesh identified by the projection by the projection unit 45. This segment search process identifies which mesh in the lattice point mesh each segment representative point in the segment matrix belongs to.

[0077] Based on the results of the segment search process by the segment search unit 46, the remesh data generation unit 47 performs the above-mentioned remesh data generation process, i.e., calculates an interpolation difference vector based on the interpolation ratio (see u, v in Figure 8) for the segment representative point, and performs a process to determine the reference coordinates of the segment representative point based on the interpolation difference vector.

[0078] The rendering unit 48 performs a process of rendering the stabilized image based on the remesh data (reference coordinates of each segment representative point) obtained by the remesh data generation unit 47 and the input image stored in the memory unit 40. Specifically, for each output pixel, reference coordinates are calculated based on the remesh data, and the value of the output pixel is calculated based on the value of the reference pixel Pr in the input image specified by the reference coordinates. Specifically, in this example, the value of the output pixel is calculated by interpolation using the values ​​of each pixel in the reference area Ar as described in Figure 9.

[0079] 2. Parameter Adjustment Method as an Embodiment As described above, in this embodiment, on the premise that it is not possible to cause the information processing device 2 to acquire an unprocessed image from the imaging device 1, an object is to enable the information processing device 2 to appropriately derive parameters and achieve appropriate parameter adjustment.

[0080] To achieve this objective, in this embodiment, the imaging device 1 and the information processing device 2 each perform the following processing: That is, the imaging device 1 transmits the processed image by the stabilizer processing unit 15 and the motion detection information by the IMU 14 to the information processing device 2, and the information processing device 2 estimates an unprocessed image based on the processed image and the motion detection information transmitted by the imaging device 1, and derives parameters to be set in the stabilizer processing unit 15 based on the results of trial image processing using candidate parameters on the estimated unprocessed image.

[0081] By adopting the method of using a pre-processed image estimated from a processed image for parameter derivation as described above, it becomes unnecessary to have the information processing device 2 acquire the pre-processed image from the imaging device 1 when deriving the parameters. Therefore, even if the information processing device 2 that derives the parameters cannot acquire the pre-processed image from the imaging device 1, the information processing device 2 can appropriately derive the parameters, and it is possible to achieve appropriate parameter adjustment for image processing in the imaging device 1.

[0082] A specific example of a parameter adjustment method according to an embodiment will be described with reference to Figures 11 and 12. In the following description, parts that are the same as parts that have already been described will be assigned the same reference numerals and descriptions thereof will be omitted.

[0083] 11 is an explanatory diagram of functions possessed by the imaging device 1 for realizing the parameter adjustment method according to the embodiment. As shown in FIG. 11, the camera control unit 16 in the imaging device 1 functions as a camera-side transmission processing unit F11 and a setting processing unit F12. The camera-side transmission processing unit F11 performs processing to transmit a processed image, which is an image after image processing by the image processing unit, and motion detection information to an external device. Specifically, the camera-side transmission processing unit F11 in this example performs processing to transmit a stabilized image obtained by the stabilizer processing unit 15 and IMU data obtained by the IMU 14 to the information processing device 2 via the communication unit 17.

[0084] As described above, this example assumes that parameter adjustment will be completed by the time of the second practice run, and therefore the transmission of the stabilized image and IMU data described above will be performed during the first practice run. Furthermore, this example assumes that evaluation of the stabilized image by the stabilizer processing attempted in parameter derivation will be performed using multiple frames of images, and the camera-side transmission processor F11 transmits multiple frames of data for the stabilized image and IMU data. For example, the transmission of the stabilized image and IMU data may be performed continuously for a time period equivalent to approximately one lap around the course.

[0085] The setting processing unit F12 will be explained later.

[0086] 12 is an explanatory diagram of functions of the information processing device 2 for realizing the parameter adjustment method according to the embodiment. In the information processing device 2, the CPU 21 has functions as a data separation unit F21, an inverse stabilizer processing unit F22, a parameter derivation unit F23, a target value reception unit F24, a parameter transmission processing unit F29, and a preview display processing unit F30.

[0087] The communication data including the stabilized image and IMU data transmitted by the camera-side transmission processing unit F11 described above is received by the communication unit 30. The data separation unit F21 performs processing to separate the stabilized image and IMU data from the communication data received in this manner.

[0088] The inverse stabilizer processing unit F22 performs processing to estimate a pre-stabilization image as a pre-processing image based on the stabilized image obtained by the data separator F21 and the IMU data.

[0089] The inverse stabilization process for estimating the pre-stabilization image from the stabilized image can basically be realized by converting the coordinates of each pixel of the stabilized image on an image plane based on the above-mentioned interpolated difference vector obtained from the IMU data. The image plane referred to here is an image plane in the same coordinate system as the input image to the stabilizer 15.

[0090] 13 is an explanatory diagram of an example of the functional configuration of the inverse stabilizer processing unit F22. As shown in the figure, the inverse stabilizer processing unit F22 is similar to the stabilizer processing unit 15 in that it has a lattice point mesh generation unit 41, a sampling unit 42, a rotation unit 44, a projection unit 45, a segment search unit 46, and a remesh data generation unit 47. The inverse stabilizer processing unit F22 has an inverse stabilizer image generation unit 50 instead of the rendering unit 48.

[0091] In this case, the remesh data generating unit 47 generates, as the remesh data, data indicating the interpolated difference vectors of each segment representative point, rather than data indicating the reference coordinates of each segment representative point.

[0092] The inverse stabilized image generation unit 50 receives the remesh data indicating the interpolated difference vectors of the representative points of each segment as described above, and calculates the difference vectors of each pixel (i.e., each output pixel) within the segment by linear interpolation using the reference coordinates of at least three vertices for each segment. This results in a difference vector being calculated for each pixel (each output pixel) of the stabilized image. After calculating the difference vectors for each pixel of the stabilized image in this way, the inverse stabilized image generation unit 50 performs coordinate transformation on the image plane for each pixel of the stabilized image based on the calculated difference vectors. The image obtained by this coordinate transformation may contain missing pixels (i.e., pixels with no pixel values) due to factors such as lens distortion correction performed during the stabilization process. The inverse stabilized image generation unit 50 performs pixel value interpolation processing for the missing pixels resulting from the coordinate transformation described above, using the pixel values ​​of surrounding pixels.

[0093] By the inverse stabilization processing as described above, coordinate transformation is performed on the image plane of the stabilized image so that the effect of the stabilization processing by the stabilizer 15 is cancelled, and as a result, the pre-stabilization image is estimated.

[0094] In the inverse stabilization process, the pre-stabilization image is generated as an image of the same size as the image frame size of the input image (4k size in this example). At this time, pixels in the surrounding area other than the image area to which pixel values ​​have been assigned by the coordinate transformation and interpolation processes in the inverse stabilization process are assigned a predetermined pixel value, such as pixel value = 0.

[0095] The pre-stabilization image obtained by the inverse stabilizer processing unit F22 will be referred to as the "estimated input image" hereinafter.

[0096] The estimated input image obtained by the inverse stabilizer processing unit F22 is stored in the storage unit 29 to be used in the stabilizer processing trial for deriving parameters. The IMU data separated by the data separation unit F21 is also stored in the storage unit 29 to be used in the stabilizer processing trial for deriving parameters.

[0097] The parameter derivation unit F23 derives parameters to be set in the stabilizer processing unit 15 based on the results of trial stabilizer processing using candidate parameters on the estimated input image (pre-processed image) estimated by the inverse stabilizer processing unit F22.

[0098] In this example, the parameter derivation unit F23 derives parameters that satisfy the image characteristic conditions for the stabilized image specified by a user operation. To this end, the CPU 21 in this example functions as a target value acceptance unit F24 that displays a GUI for specifying the image characteristic conditions on the display unit 27 and accepts the command for specifying the image characteristic conditions.

[0099] 14 shows an example of a reception screen Gs for receiving an instruction operation for image characteristic conditions. In this example, five items, namely, levelness, vibration, smoothness, object, and camera center, are defined as image characteristic conditions for a stabilized image that can be specified by the user when deriving parameters.

[0100] The levelness refers to the degree of angular difference between the horizontal direction of the space captured in the image and the direction of the horizontal line in the image. In stabilization processing, the levelness can change due to factors such as gravitational acceleration acting on the imaging device 1. In particular, when applied to racing scenes, the gravitational acceleration acting on the imaging device 1 tends to be large, which has a significant impact on the levelness. Regarding the levelness, even if the processing has resulted in appropriate leveling, the stabilized image may be perceived as not being level due to an optical illusion. From this perspective, too, it is effective to be able to specify the levelness as an image characteristic condition for deriving parameters.

[0101] The vibration is an example of an image characteristic condition related to vibration of an image, and specifically refers to the degree of vibration removal and the degree of vibration addition. Vibration can have the effect of increasing the sense of realism. The vibration addition instruction can specify the level of vibration addition, from no removal (non-removal).

[0102] Smoothness is an example of an image characteristic condition related to image vibration, and is an image characteristic condition related to the ability to follow movement (movement of the imaging device 1). The lower the ability to follow movement (i.e., the higher the stabilizer effect), the smoother the stabilized image will be. For clarity, maximum smoothness means that the output image does not move, and therefore corresponds to a state in which the stabilizer effect is maximum. In this example, smoothness can be specified to follow both the acceleration and deceleration sides of the movement.

[0103] The target object is an image characteristic condition related to the area of ​​the target subject in the image, and specifically in this example, the percentage of the area of ​​the target subject in the image can be specified. For example, the percentage of the area of ​​the road, sky, etc. in the image can be specified.

[0104] The camera center is an image characteristic condition related to the deviation between the camera center (image capture center) and the center of the image frame. The image capture center (camera center) is an estimated center of the optical axis of the imaging device 1 when there is no vibration due to camera shake. In a stabilized image, there may be deviation between the image capture center and the center of the image frame due to the influence of gravitational acceleration acting on the imaging device 1, etc. (see FIG. 15). In this example, the image characteristic condition for the camera center allows the user to specify the allowable deviation of the center of the image frame from the center of the camera.

[0105] As shown in FIG. 14 , the reception screen Gs of this example displays various controls for performing instruction operations for the above-mentioned items of levelness, vibration, smoothness, object, and camera center. Specifically, for levelness, four controls Sh are displayed for adjusting the tilt of a bar that resembles a horizon, as shown in the figure. Specifically, two controls Sh are displayed for adjusting the height of the left end of the horizontal bar, and two controls Sh are displayed for adjusting the height of the right end of the same bar. The user can specify the degree of tilt of the horizontal bar using these four controls Sh. The operation for specifying the tilt of this bar is the operation for specifying levelness.

[0106] Furthermore, for vibration, controls Sv are displayed that can specify the vibration removal level and vibration application level, and the user can specify the vibration removal level and vibration application level using the controls Sv.

[0107] Regarding smoothness, as the operators Sn for specifying the tracking ability for the movement, an operator Sn for specifying the tracking ability on the acceleration side and an operator Sn for specifying the tracking ability on the deceleration side are displayed, and the user can use these operators Sn to specify the tracking ability on the acceleration side and the deceleration side, respectively.

[0108] For the target object, a designation image display area Ag is provided, which displays an image for designating a target subject from various subjects appearing in the image, and also displays controls Su and Sd for indicating the proportion of the area of ​​the designated target subject within the image. In the designation image display area Ag, the image for designating the target subject is displayed by a target value receiving unit F24. The target value receiving unit F24 performs a subject classification process, such as semantic segmentation, on the estimated input image used in the stabilizer process attempted by the parameter derivation unit F23 to derive parameters, and superimposes visual information indicating the area of ​​each subject present in the image on the image, displaying it in the designation image display area Ag. The user can designate a target subject by performing an area designation operation, such as a click or touch, on the image displayed in the designation image display area Ag. The operators Su and Sd are operators that respectively instruct to increase or decrease the value of the area ratio, and the user can use at least one of these operators Su and Sd to instruct the area ratio that the target subject occupies within the image.

[0109] For the camera center, an operator Sc for specifying the allowable amount of deviation from the camera center is displayed, and the user can use this operator Sc to specify the allowable amount of deviation from the camera center.

[0110] The reception screen Gs is provided with an execute button B1 and a clear button B2. The execute button B1 is a button for issuing an instruction to execute a parameter derivation process by the parameter derivation unit F23, and the clear button B2 is a button for issuing an instruction to clear the designated values ​​for each item of the image characteristic conditions.

[0111] In response to operation of the execute button B1, the CPU 21 executes a parameter derivation process using the values ​​of each item, namely levelness, vibration, smoothness, object, and camera center, specified by user operation on the reception screen Gs as target values.

[0112] 12, a trial parameter set group Ps is stored in the storage unit 29 of the information processing device 2. The trial parameter set group Ps is a set of candidate parameters that are used in the stabilizer processing that the parameter derivation unit F23 attempts in the parameter derivation processing, and is stored.

[0113] Here, the parameters that can be set in the stabilizer processing are extremely diverse, and it is difficult to list all of the specific parameters. To cite one example, parameters that can be set in the stabilizer processing include parameters related to timing adjustment of IMU data and various filter processes. Other parameters that can be set in the stabilizer processing include parameters related to the processing of the projection unit 43, rotation unit 44, and projection unit 45 described above. Furthermore, when the projection processing in the projection unit 45 performs the free curvature perspective projection described above, parameters related to the free curvature perspective projection are also included.

[0114] Varying the combination of parameters such as those exemplified above will result in changes in the image characteristics of the stabilized image, such as the levelness, vibration, smoothness, object, and camera center. Therefore, by trying out various parameter sets, it is possible to derive the optimal parameter set that satisfies the specified image characteristic conditions.

[0115] 12, the parameter derivation unit F23 has a stabilizer processing unit F25, a trial control unit F26, an image evaluation unit F27, and an optimal parameter determination unit F28. The stabilizer processing unit F25 performs stabilizer processing on an input image based on IMU data, similar to the stabilizer processing unit 15. The stabilizer processing unit F25 uses an estimated input image stored in a storage unit 29 as an input image and performs stabilizer processing based on IMU data (IMU data synchronized with the exposure timing of the estimated input image) also stored in the storage unit 29.

[0116] The trial control unit F26 controls the stabilizer processing of the stabilizer processing unit F25. Specifically, the trial control unit F26 performs control to set a parameter set selected from the trial parameter set group Ps in the storage unit 29 in the stabilizer processing unit F25.

[0117] The image evaluation unit F27 evaluates the image characteristics of the stabilized image obtained by the stabilizer processing unit F25 based on the image characteristic conditions specified by the user. Specifically, the image evaluation unit F27 evaluates the image characteristics of the stabilized image based on the values ​​(target values) of the items of levelness, vibration, smoothness, object, and camera center accepted by the target value accepting unit F24.

[0118] Here, the evaluation of the levelness is performed by calculating the difference between the levelness specified by the user and the levelness of the stabilized image as an evaluation value. As described above, the levelness refers to the angular difference between the horizon in the image and the horizontal line direction of the image. The levelness of the stabilized image can be determined, for example, by extracting straight line segments from the stabilized image using a Hough transform or the like and calculating the angle (angle with respect to the horizontal line direction) of the dominant straight line segment among the extracted straight line segments. In this case, if no horizontal object is present, the level is inferred from vertical objects such as buildings and utility poles. The image evaluation unit F27 calculates the difference between the calculated levelness of the stabilized image and the levelness specified by the user as a levelness evaluation value.

[0119] Furthermore, vibration evaluation is performed by calculating, as an evaluation value, the difference between the vibration level (removal level or excitation level) specified by the user and the vibration level in the stabilized image. Here, the vibration level is represented by the magnitude and period of the vibration (movement per unit time) of the subject in the image. Therefore, the vibration level can be evaluated by calculating the difference between the vibration level represented by the magnitude and period of the vibration of the subject in the image and the vibration level specified by the user. The magnitude and period of the vibration of the subject in the stabilized image can be obtained by performing optical flow processing on stabilized images over a certain period of time. The image evaluation unit F27 calculates the vibration level as an index of vibration based on the magnitude and period of vibration thus determined for the stabilized image, and calculates the difference between this vibration level and the vibration level specified by the user as a vibration evaluation value.

[0120] The smoothness is evaluated by calculating, as an evaluation value, the difference between the level of smoothness on the acceleration and deceleration sides specified by the user and the level of smoothness on the acceleration and deceleration sides in the stabilized image. The level of smoothness on the acceleration and deceleration sides in the stabilized image can be determined by performing optical flow processing on stabilized images over a certain period of time. The image evaluation unit F27 determines the level of smoothness on the acceleration and deceleration sides in the stabilized image and calculates the difference between the level of smoothness on the acceleration and deceleration sides specified by the user. The smoothness evaluation value is then calculated as a composite value, such as the average or total value of the difference values ​​thus determined for each of the acceleration and deceleration sides.

[0121] The evaluation of the object is performed by calculating, as an evaluation value, the difference between the area proportion of the target subject specified by the user and the area proportion of the same subject in the stabilized image (the proportion of the area that occupies the image). Specifically, the image evaluation unit F27 performs a subject classification process such as semantic segmentation on the stabilized image to identify the area of ​​the target subject specified by the user, and calculates the proportion of the area that the target subject area occupies in the image. The image evaluation unit F27 then calculates, as an object evaluation value, the difference between the area proportion thus calculated and the area proportion of the target subject specified by the user.

[0122] The camera center is evaluated by calculating the difference between the allowable deviation from the camera center specified by the user and the deviation between the camera center and the center of the image frame in the stabilized image as an evaluation value. The camera center in a stabilized image can be estimated by performing image analysis on stabilized images over a certain period of time. The image evaluation unit F27 calculates the deviation between the estimated camera center and the center of the image frame for the stabilized image to be evaluated, and calculates the difference between this deviation and the allowable deviation specified by the user as the camera center evaluation value.

[0123] The optimal parameter determination unit F28 determines an optimal parameter set based on the evaluation values ​​calculated by the image evaluation unit F27. Specifically, the optimal parameter determination unit F28 in this example determines a parameter set that can satisfy the image characteristic conditions specified by the user based on the above-mentioned levelness evaluation value, vibration evaluation value, smoothness evaluation value, object evaluation value, and camera center evaluation value.

[0124] In this example, the stabilizer processing by the stabilizer processing unit F25 is performed for all or a predetermined number of parameter sets prepared as the trial parameter set group Ps. Therefore, the trial control unit F26 repeatedly controls the stabilizer processing unit F25 to execute the stabilizer processing with a certain parameter set set, and then execute the stabilizer processing with the next parameter set set. In addition, the image evaluation unit F27 calculates the above-mentioned evaluation value for each of the stabilized images processed with each parameter set set.

[0125] The optimal parameter determination unit F28 in this example determines an optimal parameter set based on the evaluation values ​​calculated by the image evaluation unit F27 for the multiple parameter sets. In this example, because evaluation values ​​for multiple items are obtained, the optimal parameter determination unit F28 calculates an overall evaluation value for comprehensively evaluating the multiple items. Specifically, the optimal parameter determination unit F28 calculates a composite value (e.g., average or total value) of the levelness evaluation value, vibration evaluation value, smoothness evaluation value, and object evaluation value excluding the camera center evaluation value as the overall evaluation value. The camera center evaluation value is used as a separate evaluation criterion from the overall evaluation value because the evaluation criterion is the allowable deviation from the camera center. That is, the optimal parameter determination unit F28 basically determines the parameter set with the smallest overall evaluation value as the optimal parameter set for the multiple parameter sets. However, it excludes parameter sets with camera center evaluation values ​​less than 0 from the optimal parameter set determination based on such overall evaluation values. As a result, a further evaluation is made based on the overall evaluation value for parameter sets that satisfy the condition that the allowable deviation from the camera center is less than the deviation specified by the user, and as a result, the parameter set that satisfies all of the image characteristic conditions specified on the reception screen Gs of Figure 14 is determined to be the optimal parameter set.

[0126] The CPU 21 in this example includes a preview display processing unit F30. The preview display processing unit F30 performs processing to display, on the display unit 27, a stabilized image obtained by performing stabilization processing using the parameters derived by the parameter derivation unit F23 (the parameter set determined by the optimal parameter determination unit F28).

[0127] 14, the reception screen Gs has a preview area Ap for displaying a preview by the preview display processing unit F30. In response to the optimal parameter set being determined by the optimal parameter determination unit F28, the preview display processing unit F30 causes the stabilization processing unit F25 to execute stabilization processing with the determined parameter set set, and displays the stabilized image obtained by the stabilization processing in the preview area Ap. This allows the user to check the contents of the stabilized image that will be obtained when the parameter set derived by the parameter derivation unit F23 in response to operation of the execute button B1 described above is set.

[0128] As shown in the figure, an Apply button B3 and an Undo button B4 are arranged near the preview area Ap on the reception screen Gs. The Apply button B3 is a button for instructing that the derived parameter set be applied as a parameter set to be set in the stabilizer processing unit 15 of the imaging device 1, and the Undo button B4 is a button for instructing that the image characteristic condition instructions be restored to the instructions given the previous time the Execute button B1 was operated.

[0129] 12 , the parameter transmission processing unit F29 performs processing to transmit the parameters derived by the parameter derivation unit F23 to the imaging device 1. Specifically, in response to operation of the apply button B3 on the reception screen Gs, the parameter transmission processing unit F29 performs processing to transmit the parameter set determined by the optimal parameter determination unit F28 to the imaging device 1 that transmitted the stabilized image and IMU data via the communication unit 30. For clarity, the imaging device 1 that transmitted the stabilized image and IMU data is the imaging device 1 that transmitted the stabilized image that was the source of the estimated input image used to derive the parameters to be transmitted.

[0130] 11 performs processing to set parameters derived on the information processing device 2 side in the stabilizer processing unit 15. Specifically, the setting processing unit F12 performs processing to set the parameter set transmitted by the parameter transmission processing unit F29 in the stabilizer processing unit 15.

[0131] As a result, in the imaging device 1, the parameters of the stabilizer processing unit 15 are adjusted to the optimum parameters.

[0132] 3. Processing Procedure An example of a processing procedure of the information processing device 2 for realizing the parameter adjustment method according to the embodiment described above will be described with reference to the flowchart of Fig. 16. The processing shown in Fig. 16 is executed by the CPU 21 in the information processing device 2 based on a program stored in a storage device such as the ROM 22 or the storage unit 29.

[0133] 16, the CPU 21 executes a GUI display process in step S101, that is, a process of displaying the reception screen Gs shown in FIG.

[0134] In step S102 following step S101, the CPU 21 executes input response processing, and further determines whether an execution operation has been performed in the following step S103. The determination processing in step S103 is processing for determining whether the execute button B1 on the reception screen Gs has been operated. If it is determined in step S103 that an execution operation has not been performed, the CPU 21 executes the input response processing of step S102. In the input response processing of step S102, processing according to the input operation on the screen is performed, such as updating the display content of the reception screen Gs in accordance with various operators (operators Sh, Sv, etc.) on the reception screen Gs.

[0135] If it is determined in step S103 that an execution operation has been performed, the CPU 21 proceeds to step S104 and performs a parameter derivation process.

[0136] 17 is a flowchart showing details of the parameter derivation process in step S104. In the parameter derivation process in step S104, the CPU 21 first performs a process in step S201 to set a set identifier n to 1. The set identifier n is an identifier for identifying a parameter set to be used in the process from among the parameter sets in the trial parameter set group Ps.

[0137] In step S202 following step S201, the CPU 21 executes stabilization processing using the nth parameter set. That is, the nth parameter set is acquired from the trial parameter set group Ps, and stabilization processing is performed on the estimated input image stored in the storage unit 29 with the parameter set set in its set state. In this example, image characteristic evaluation for the parameter set is performed using stabilized images over a certain period of time, so in step S202 the CPU 21 executes stabilization processing on each of the estimated input images over the certain period of time (i.e., multiple frames of estimated input images). At this time, it goes without saying that the IMU data used is data synchronized with the estimated input image to be processed.

[0138] In step S203 following step S202, the CPU 21 performs evaluation processing of the stabilized image. That is, as explained above as the processing of the image evaluation unit F27, evaluation processing related to image characteristics of the stabilized image is performed, specifically, in this example, processing to calculate a comprehensive evaluation value of the horizontality evaluation value, vibration evaluation value, smoothness evaluation value, and object evaluation value, as well as a camera center evaluation value.

[0139] In step S204 following step S203, the CPU 21 performs a process of storing the evaluation value, that is, a process of storing the evaluation value calculated in step S203 in a predetermined storage device such as the storage unit 29 or the RAM 23.

[0140] In step S205 following step S204, the CPU 21 determines whether the set identifier n is equal to or greater than a predetermined value N (N is a natural number at least equal to 2). Here, the predetermined value N may be set to the total number of parameter sets prepared as the trial parameter set group Ps, or may be set to a predetermined number less than the total number.

[0141] If it is determined in step S205 that the set identifier n is not equal to or greater than the predetermined value N, the CPU 21 proceeds to step S206, increments the set identifier n by 1, and returns to step S202, whereupon evaluation is performed on the next parameter set.

[0142] If it is determined in step S205 that the set identifier n is equal to or greater than the predetermined value N, the CPU 21 proceeds to step S207, where it performs a process of determining an optimal parameter set based on the evaluation values. In this example, the CPU 21 determines the optimal parameter set based on the overall evaluation value and the camera center evaluation value for each parameter set stored in step S204. Specifically, the CPU 21 excludes parameter sets with camera center evaluation values ​​less than 0 from the tried parameter sets, and performs a process of determining the optimal parameter set for the remaining parameter sets based on the overall evaluation values. In other words, in this example, the CPU 21 determines the parameter set with the smallest overall evaluation value as the optimal parameter set. Note that if there are multiple parameter sets with the smallest overall evaluation value, one optimal parameter set is determined according to a predetermined priority. For example, the priority in this case may be based on the camera center evaluation value. Alternatively, arbitrary selection may be performed instead of selection based on priority.

[0143] 16, in response to the execution of the parameter derivation process in step S104 described above, the CPU 21 executes a preview display process of an image stabilized using the derived parameters in step S105. That is, the CPU 21 executes the stabilization process with the parameter set derived in step S104 set, and executes a process of displaying the stabilized image obtained by the stabilization process on the display unit 27, specifically in the preview area Ap of the reception screen Gs in this example.

[0144] In response to the execution of the preview display process in step S105, the CPU 21 waits for either an apply operation or an undo operation in steps S106 and S107, i.e., waits for either an operation of the apply button B3 or an operation of the undo button B4.

[0145] If it is determined in step S107 that an Undo operation has been performed, the CPU 21 returns to step S102. Although not shown in the drawings, if it is determined in step S107 that an Undo operation has been performed, the CPU 21 performs a process of returning the indication values ​​of each item on the reception screen Gs, such as the horizontality, to the indication values ​​at the time of the previous operation of the execute button B1, and then proceeds to step S102.

[0146] On the other hand, if it is determined in step S106 that the application operation has been performed, the CPU 21 proceeds to step S108 to execute a parameter transmission process, i.e., to transmit the parameter set derived in step S104 via the communication unit 30 to the image capture device 1 that is the sender of the stabilized image and IMU data.

[0147] As described above, the imaging device 1 performs processing to set the transmitted parameter set in the stabilizer processing unit 15. As a result, the parameters in the stabilizer processing unit 15 are adjusted to the optimal parameters.

[0148] Although the above example illustrates a method of evaluating image characteristics for a predetermined number of parameters and deriving parameters based on the results, the parameter derivation method is not limited to this method. For example, it is possible to adopt a method in which image characteristics are evaluated each time a parameter is tried, and if the evaluation result satisfies a predetermined standard, the parameter is determined to be the optimal parameter. This method can reduce the number of stabilizer processes and evaluation processes required for parameter derivation.

[0149] Furthermore, in the above, with regard to the derivation of parameters for the stabilizer processing unit 15, an explanation was given of deriving parameters to be set in the stabilizer processing unit 15 of one imaging device 1 based on stabilized images and IMU data from the imaging device 1, but stabilized images and IMU data can be transmitted from multiple imaging devices 1 to the information processing device 2, and the parameter derivation unit F23 can also derive parameters to be set in the stabilizer processing unit 15 for each imaging device 1 based on the pre-processed image (estimated input image) estimated for each imaging device 1 by the inverse stabilizer processing unit F22.

[0150] This makes it possible to derive appropriate parameters for each imaging device 1, even if differences in the image characteristics of processed images (stabilized images) occur between imaging devices 1 due to differences in the usage environment. Therefore, it is possible to achieve appropriate parameter adjustment for each imaging device 1 in accordance with its usage environment. In particular, when applied to car or motorcycle racing scenes, the optimal parameters will differ depending on the course, vehicle settings, the driver's driving style, etc. For this reason, it is preferable to derive optimal parameters for each imaging device 1 as described above.

[0151] 4. Modifications Although the embodiments have been described above, the present technology is not limited to the specific examples described above and can employ various modified configurations. For example, while a car or a motorcycle has been described above as an example of a moving body M to which the imaging device 1 is attached, the type of moving body M is not particularly limited. For example, the moving body M may be a person or an animal. As an example, the imaging device 1 may be attached to a player or a referee in a sport. When the imaging device 1 is attached to a player, the optimal parameters may differ depending on the movement characteristics of the moving body M according to the type of sport, so it is preferable to perform parameter adjustment according to the present technology. Furthermore, since differences in movement characteristics may occur between the player and the referee, it is effective to perform parameter adjustment for each player and referee. Furthermore, since the optimal parameters may differ depending on the attachment location, such as the head or chest, it is also effective to perform parameter adjustment for each attachment location. Furthermore, it is not necessary to attach the imaging device 1 to the moving body M. For example, the imaging device 1 may be attached to some fixed object as a fixed-point surveillance camera. Even in this case, if the imaging device 1 is placed outdoors, it may be vibrated by wind, etc., and even if it is placed indoors, it may be vibrated by an earthquake, etc., so it is possible that stabilization processing will be required.

[0152] Furthermore, although stabilization processing has been described above as an example of image processing based on motion detection information, image processing according to the present technology is not limited to stabilization processing. For example, in the fields of VR (Virtual Reality) and AR (Augmented Reality), when adjusting the viewpoint of an image presented to a head-mounted display (HMD) wearer based on motion detection information from the HMD using a technique such as the free-curvature perspective projection described above, the viewpoint adjustment process may correspond to image processing according to the present technology. In the case of VR, the target image for image processing may be a computer-generated (CG) image rather than a captured image. As can be understood from this point, the image processing device according to the present technology is not limited to an imaging device having an imaging unit, but may broadly include devices that perform image processing based on motion detection information on images other than captured images, such as CG images.

[0153] In the above example, multiple prepared parameter sets are sequentially tried in the trial of the stabilizer process for parameter derivation, but each parameter may be individually adjustable. In this case, it is also possible to fix some parameters and adjust only specific parameters. It is also possible to select the parameter sets to be tried in parameter derivation based on image characteristic conditions specified by the user.

[0154] Although the above description exemplifies a method for causing an image processing device to acquire derived parameters in which an information processing device that has performed parameter derivation transmits the derived parameters to the image processing device, other methods are also conceivable for causing an image processing device to acquire derived parameters. For example, a method is conceivable in which an information processing device that has performed parameter derivation transmits the derived parameters to another device, and the other device then transmits the derived parameters to the image processing device. Alternatively, a method is conceivable in which the image processing device acquires the derived parameters via a removable recording medium such as a USB (Universal Serial Bus) memory.

[0155] 5. Summary of the Embodiment As described above, the information processing device (2) according to the embodiment includes an image processing device (imaging device 1) equipped with a motion detection unit that detects the motion of the device itself and an image processing unit that performs image processing on an image to be processed based on motion detection information from the motion detection unit. The image processing device also includes an inverse processing unit (inverse stabilizer processing unit F22) that estimates a pre-processed image, which is an image before image processing, based on the motion detection information and a processed image, which is an image after image processing by the image processing unit, transmitted from the image processing device. The inverse processing unit (F23) also includes a parameter derivation unit (F23) that derives parameters to be set in the image processing unit based on the results of trial image processing using candidate parameters on the pre-processed image estimated by the inverse processing unit. By adopting a method of using a pre-processed image estimated from a processed image for parameter derivation as described above, it is not necessary for the image processing device to acquire the pre-processed image from the image processing device when deriving the parameters. Therefore, even if the information processing device that derives the parameters cannot acquire the pre-processed image from the image processing device, the information processing device can appropriately derive the parameters, thereby enabling appropriate parameter adjustment for image processing in the image processing device.

[0156] Furthermore, in the information processing device according to the embodiment, the image processing device is an imaging device having an imaging unit (image sensor 12), and the image processing unit performs, as image processing, stabilization processing on an image captured by the imaging unit. This allows the information processing device to appropriately derive parameters even when the information processing device that derives parameters cannot acquire an image before stabilization processing from the imaging device, thereby enabling appropriate parameter adjustment for stabilization processing in the imaging device.

[0157] Furthermore, in an information processing device according to an embodiment, multiple imaging devices transmit processed images and motion detection information to the information processing device, and the parameter derivation unit derives parameters to be set in the image processing unit for each imaging device based on the pre-processed images estimated by the inverse processing unit for each imaging device. This makes it possible to derive appropriate parameters for each imaging device, even if differences in the image characteristics of processed images (stabilized images) between imaging devices occur due to differences in usage environment. Therefore, it is possible to achieve appropriate parameter adjustment for each imaging device according to its usage environment.

[0158] Furthermore, in the information processing device according to the embodiment, the parameter derivation unit derives parameters that satisfy the image characteristic conditions for the processed image specified by a user operation. As a result, when adjusting the parameters of the image processing unit, the user only needs to specify the image characteristic conditions for the processed image, eliminating the need for the user to manually consider the parameter combinations to be set during trial or to manually specify the parameters. This reduces the user's workload when adjusting parameters. Furthermore, because the only required operation is to specify the image characteristic conditions, the time required for operation is reduced, thereby shortening the time required for parameter derivation. Furthermore, even a user who is not familiar with image processing parameters can derive appropriate parameters.

[0159] Furthermore, the information processing device according to the embodiment includes an operation accepting unit (target value accepting unit F24) that accepts the operation of specifying the image characteristic conditions by displaying a GUI for specifying the image characteristic conditions on the display unit. By accepting the operation of specifying the image characteristic conditions via the GUI, the user can perform the operation while being presented with visual information to assist the user in specifying the image characteristic conditions, allowing the user to perform the operation intuitively. This facilitates the operation of specifying the image characteristic conditions and reduces the operational burden on the user related to specifying the image characteristic conditions.

[0160] Furthermore, in the information processing device according to the embodiment, the image processing device is an imaging device having an imaging unit, and the image processing is stabilization processing for an image captured by the imaging unit. This allows parameters of the image processing unit to be adjusted so that the image characteristics that the user intends can be obtained from the image characteristics that can be achieved by the stabilization processing.

[0161] Furthermore, in the information processing device according to the embodiment, the image characteristic conditions are conditions related to horizontality, which allows the parameters of the image processing unit to be adjusted so that the horizontality of the processed image matches the horizontality intended by the user.

[0162] In the information processing device according to the embodiment, the image characteristic conditions are conditions related to vibration, which allows the parameters of the image processing unit to be adjusted so that the vibration mode of the processed image matches the mode intended by the user.

[0163] Furthermore, in the information processing device according to the embodiment, the image characteristic condition is a condition related to the area of ​​the target subject in the image, which allows the parameters of the image processing unit to be adjusted so that the area of ​​the target subject in the processed image matches the area intended by the user.

[0164] Furthermore, in an information processing device according to an embodiment, the image characteristic conditions are conditions related to the deviation between the imaging center and the center of the image frame. In stabilized images, deviation between the imaging center and the center of the image frame may occur due to factors such as acceleration acting on the imaging device. With the above configuration, parameters of the image processing unit can be adjusted so that the deviation between the imaging center and the center of the image frame is as intended by the user, for example, by preventing the deviation between the imaging center and the center of the image frame from exceeding a predetermined allowable deviation.

[0165] Furthermore, the information processing device according to the embodiment includes a preview display processing unit (F30) that performs processing to display on the display unit a processed image obtained by performing image processing using the parameters derived by the parameter derivation unit. This allows the user to check whether the image characteristic conditions intended by the user are satisfied by looking at the processed image obtained when the derived parameters are set, thereby improving the ease of checking whether the desired parameters have been derived.

[0166] Furthermore, in the information processing device according to the embodiment, the preview display processing unit displays the processed image on the same display unit that displays the GUI for specifying the image characteristic conditions (see FIG. 14 ). By displaying a preview of the processed image on the display unit that displays the GUI for specifying the image characteristic conditions, if the user checks the preview image and decides to change the specified image characteristic conditions and redo the parameter derivation, the user can easily change the image characteristic conditions, thereby improving convenience.

[0167] A parameter derivation method as an embodiment is a parameter derivation method in which an information processing device includes a motion detection unit that detects the motion of the device itself and an image processing unit that performs image processing on an image to be processed based on the motion detection information by the motion detection unit, and the image processing device estimates a pre-processed image that is an image before image processing based on the motion detection information and a processed image that is an image after image processing by the image processing unit, and derives parameters to be set in the image processing unit based on the results of trial image processing using candidate parameters for the estimated pre-processed image.With this parameter derivation method, it is possible to obtain the same functions and effects as the information processing device as the above-mentioned embodiment.

[0168] The image processing device according to the embodiment includes a motion detection unit (IMU 14) that detects the motion of the device itself, an image processing unit (stabilization processing unit 15) that performs image processing on an image to be processed based on motion detection information from the motion detection unit, a transmission processing unit (camera-side transmission processing unit F11) that transmits a processed image (an image after image processing by the image processing unit) and motion detection information to an external device, and a setting processing unit (camera-side transmission processing unit F12) that sets parameters derived based on the results of image processing trials using candidate parameters on a pre-processed image (an image before image processing estimated based on the processed image and the motion detection information) in the image processing unit. By transmitting the processed image and motion detection information to the external device as described above, the information processing device as the external device can use a pre-processed image estimated from the processed image to derive parameters, eliminating the need for the image processing device to acquire the pre-processed image from the image processing device when deriving parameters. Furthermore, the setting processing unit allows the image processing device to perform image processing using derived parameters. Therefore, even if the information processing device that derives the parameters cannot acquire the unprocessed image from the image processing device, it is possible to ensure that the parameters are adjusted appropriately for image processing in the image processing device.

[0169] A parameter setting method as an embodiment is a parameter setting method for an image processing device that includes a motion detection unit that detects the motion of the device itself and an image processing unit that performs image processing on an image to be processed based on the motion detection information by the motion detection unit, in which a processed image that is an image after image processing by the image processing unit and the motion detection information are transmitted to an external device, and the external device sets parameters derived based on the results of trial image processing using candidate parameters on a pre-processed image that is an image before image processing estimated based on the processed image and the motion detection information in the image processing unit. This parameter setting method can also achieve the same functions and effects as the image processing device as the above-mentioned embodiment.

[0170] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0171] <6. The Present Technology> Note that the present technology may also have the following configuration. (1) An information processing device comprising: an image processing device including: a motion detection unit that detects motion of its own device; and an image processing unit that performs image processing on an image to be processed based on motion detection information by the motion detection unit; and an inverse processing unit that estimates a pre-processed image that is the image before the image processing, based on the motion detection information and a processed image that is the image after image processing by the image processing unit, transmitted by the image processing device; and a parameter derivation unit that derives parameters to be set in the image processing unit based on a result of trialing the image processing using candidate parameters on the pre-processed image estimated by the inverse processing unit. (2) The image processing device is an imaging device having an imaging unit, and the image processing unit performs stabilization processing on an image captured by the imaging unit as the image processing. (3) The information processing device according to (2), wherein a plurality of the imaging devices transmit the processed image and the motion detection information to the information processing device, and the parameter derivation unit derives parameters to be set in the image processing unit for each of the imaging devices based on the pre-processing image estimated by the inverse processing unit for each of the imaging devices. (4) The information processing device according to any of (1) to (3), wherein the parameter derivation unit derives parameters that satisfy image characteristic conditions for the processed image specified by a user operation. (5) The information processing device according to (4), further comprising an operation accepting unit that displays a GUI on a display unit for performing an operation to specify the image characteristic conditions and accepts an operation to specify the image characteristic conditions. (6) The information processing device according to (4) or (5), wherein the image processing device is an imaging device having an imaging unit, and the image processing is stabilization processing targeting images captured by the imaging unit. (7) The information processing device according to (6), wherein the image characteristic conditions are conditions related to horizontality. (8) The information processing device according to (6), wherein the image characteristic condition is a condition related to vibration. (9) The information processing device according to (6), wherein the image characteristic condition is a condition related to an area of ​​a target subject in an image.(10) The information processing device according to (6), wherein the image characteristic condition is a condition related to a deviation between the center of imaging and the center of an image frame. (11) The information processing device according to any of (4) to (10), further comprising a preview display processing unit that performs processing to display on a display unit a processed image obtained by performing the image processing using the parameters derived by the parameter derivation unit. (12) The information processing device according to (11), wherein the preview display processing unit displays the processed image on the same display unit as a display unit that displays a GUI for performing an instruction operation for the image characteristic condition. (13) A parameter derivation method in which an information processing device estimates a pre-processed image, which is the image before the image processing, based on the motion detection information and a processed image that has been processed by the image processing unit and that has been transmitted by an image processing device including: a motion detection unit that detects motion of the information processing device itself; and an image processing unit that performs image processing on an image to be processed based on motion detection information by the motion detection unit; and derives parameters to be set in the image processing unit based on a result of trialing the image processing using candidate parameters on the estimated pre-processed image. (14) An image processing device comprising: a motion detection unit that detects the motion of the device itself; an image processing unit that performs image processing on an image to be processed based on motion detection information by the motion detection unit; a transmission processing unit that performs processing to transmit to an external device a processed image, which is the image after image processing by the image processing unit, and the motion detection information; and a setting processing unit that performs processing to set parameters derived based on results of the image processing trial using candidate parameters on a pre-processed image, which is the image before the image processing estimated by the external device based on the processed image and the motion detection information, to the image processing unit.(15) A parameter setting method for an image processing device having a motion detection unit that detects the motion of the device itself and an image processing unit that performs image processing on an image to be processed based on motion detection information by the motion detection unit, the parameter setting method comprising: transmitting a processed image, which is the image after image processing by the image processing unit, and the motion detection information to an external device; and setting, in the image processing unit, parameters derived based on the results of the external device attempting the image processing using candidate parameters on a pre-processed image, which is the image before the image processing estimated based on the processed image and the motion detection information.

[0172] DESCRIPTION OF SYMBOLS 1 Imaging device 2 Information processing device 12 Image sensor 13 Image signal processing unit 14 IMU 15 Stabilization processing unit 16 Camera control unit 17 Communication unit 21 CPU 22 ROM 23 RAM 26 Input unit 27 Display unit 29 Storage unit 30 Communication unit q0 to q3 Quaternion data Du, Dv Difference vector Pr Reference pixel Ar Reference area 40 Memory unit 41 Lattice point mesh generation unit 42 Sampling unit 43 Projection unit 44 Rotation unit 45 Projection unit 46 Segment search unit 47 Remesh data generation unit 48 Rendering unit F11 Camera side transmission processing unit F12 Setting processing unit F21 Data separation unit F22 Inverse stabilizer processing unit F23 Parameter derivation unit F24 Target value reception unit F25 Stabilization processing unit F26 Trial control unit F27 Image evaluation section F28 Optimum parameter determination section F29 Parameter transmission processing section F30 Preview display processing section Ps Trial parameter set group 50 Reverse stabilized image generation section Gs Reception screen B1 Execute button B2 Clear button B3 Apply button B4 Undo button Sh Operator Sv Operator Sn Operator Ag Designation image display area Sd Operator Su Operator Sc Operator Ap Preview area

Claims

1. An information processing device comprising: an image processing device having a motion detection unit that detects the motion of the device itself; and an image processing unit that performs image processing on an image to be processed based on motion detection information by the motion detection unit; an inverse processing unit that estimates a pre-processed image, which is the image before the image processing, based on a processed image, which is the image after image processing by the image processing unit, and the motion detection information transmitted by the image processing device; and a parameter derivation unit that derives parameters to be set in the image processing unit based on results of trialing the image processing using candidate parameters on the pre-processed image estimated by the inverse processing unit.

2. The information processing device according to claim 1, wherein the image processing device is an imaging device having an imaging section, and the image processing section performs, as the image processing, a stabilization process on an image captured by the imaging section.

3. The information processing device according to claim 2, wherein a plurality of the imaging devices transmit the processed images and the motion detection information to the information processing device, and the parameter derivation unit derives parameters to be set in the image processing unit for each of the imaging devices based on the pre-processed images estimated by the inverse processing unit for each of the imaging devices.

4. The information processing device according to claim 1, wherein the parameter derivation section derives parameters that satisfy image characteristic conditions for the processed image designated by a user operation.

5. The information processing device according to claim 4, further comprising an operation receiving section for receiving an instruction operation for the image characteristic conditions by displaying a GUI for performing an instruction operation for the image characteristic conditions on a display section.

6. The information processing device according to claim 4, wherein the image processing device is an imaging device having an imaging unit, and the image processing is a stabilization process for an image captured by the imaging unit.

7. The information processing device according to claim 6, wherein the image characteristic condition is a condition relating to horizontality.

8. The information processing device according to claim 6, wherein the image characteristic conditions are conditions relating to vibration.

9. The information processing device according to claim 6, wherein the image characteristic condition is a condition related to an area of ​​the target subject that occupies the image.

10. The information processing device according to claim 6, wherein the image characteristic condition is a condition related to a deviation between the center of imaging and the center of the image frame.

11. The information processing device according to claim 4, further comprising a preview display processing section that performs processing for displaying on a display section a processed image obtained by performing the image processing using the parameters derived by the parameter derivation section.

12. The information processing device according to claim 11, wherein the preview display processing section displays the processed image on the same display section on which a GUI for performing an operation to specify the image characteristic conditions is displayed.

13. A parameter derivation method in which an information processing device has a motion detection unit that detects the movement of the device itself, and an image processing unit that performs image processing on an image to be processed based on motion detection information by the motion detection unit, estimates a pre-processed image, which is the image before the image processing, based on a processed image, which is the image after image processing by the image processing unit, and the motion detection information, transmitted by the image processing device, and derives parameters to be set in the image processing unit based on a result of trialing the image processing using candidate parameters with the estimated pre-processed image as the target.

14. An image processing device comprising: a motion detection unit that detects the movement of the device itself; an image processing unit that performs image processing on an image to be processed based on motion detection information by the motion detection unit; a transmission processing unit that performs processing to transmit to an external device a processed image, which is the image after image processing by the image processing unit, and the motion detection information; and a setting processing unit that performs processing to set parameters derived based on results of trialing the image processing using candidate parameters on a pre-processed image, which is the image before the image processing estimated by the external device based on the processed image and the motion detection information, in the image processing unit.

15. A parameter setting method for an image processing device having a motion detection unit that detects the motion of the device itself, and an image processing unit that performs image processing on an image to be processed based on motion detection information by the motion detection unit, comprising the steps of: transmitting a processed image, which is the image after image processing by the image processing unit, and the motion detection information to an external device; and setting, in the image processing unit, parameters derived based on the results of trialing the image processing using candidate parameters on a pre-processed image, which is the image before the image processing estimated based on the processed image and the motion detection information.

Images