Signal processing device and signal processing method

Abstract

The signal processing device according to the present technique is provided with: an approximate curved surface rendering unit having a first layered autoencoder pretrained using, as learning input data, coordinate data of each element of an object consisting of a plurality of elements, the approximate curved surface rendering unit being configured to obtain, using the coordinate data of each element as input data, approximate curved surface data of an approximate curved surface rendering of the object in an intermediate layer of the first layered autoencoder; and a geometric modulation processing unit having a second layered autoencoder machine-learned using the approximate curved surface data as learning input data and using, as training data, a result of performing geometric modulation processing on the object through coordinate conversion of each element, the geometric modulation processing unit being configured to perform geometric modulation processing on the approximate curved surface data using the second layered autoencoder.

Description

[Technical field]

[0001] The present technology relates to a signal processing device and method, and in particular to a technology for performing geometric modulation processing on an object defined in space. [Background technology]

[0002] For example, in the field of image processing, there are cases where geometric modulation processing such as deformation, movement, rotation, etc. of an object defined in space is performed. An example of an object is polygon data, which is made up of a plurality of polygons as elements. In this case, the geometric modulation processing of the object is generally performed as a coordinate conversion processing of the coordinate data of each polygon.

[0003] As a related prior art, the following Patent Document 1 can be mentioned. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2014-66995 A Summary of the Invention [Problem to be solved by the invention]

[0005] Here, when performing geometric modulation processing of an object consisting of multiple elements as described above by coordinate transformation processing, coordinate transformation is performed for each element separately, which tends to increase the computational cost required for the geometric modulation processing.

[0006] The present technology has been made in consideration of the above circumstances, and aims to reduce the computational costs associated with geometric modulation processing of objects. [Means for solving the problem]

[0007] The signal processing device according to the present technology includes a first layered autoencoder that is pre-trained using coordinate data for each element of an object consisting of a plurality of elements as learning input data, an approximation surface generation unit that uses the coordinate data for each element as input data to obtain approximate surface data by approximating the object in an intermediate layer of the first layered autoencoder, and a second layered autoencoder that is machine-trained using the approximate surface data as learning input data and a result of performing geometric modulation processing on the object by coordinate transformation for each element as teacher data, and a geometric modulation processing unit that performs the geometric modulation processing on the approximate surface data by the second layered autoencoder. The geometric modulation process refers to a process that changes at least one of the shape, position, orientation, and size of an object. In this case, the change in orientation can be, for example, rotation. According to the above configuration, geometric modulation processing such as deformation, movement, and rotation of an object is performed on approximate surface data obtained by the first stacked autoencoder by dimensionally compressing the coordinate data of each element.

[0008] In the signal processing device related to the above-mentioned present technology, a stabilization process of an image captured by an imaging device is performed using a lattice point mesh having a plurality of lattice points arranged in the horizontal and vertical directions in an output coordinate system of the captured image, and in which movement information of the imaging device is associated with each of the lattice points, and the object can be configured as the lattice point mesh. As a result, the geometric modulation processing such as deformation and rotation of the lattice point mesh, which is performed in the stabilization processing using the lattice point mesh, is performed on the approximate curved surface data of the lattice point mesh.

[0009] The signal processing device according to the present technology described above can be configured to include a third layered autoencoder that is machine-trained using as training data the results of a segment search process that identifies in which of the squares in the lattice point mesh that has been subjected to the geometric modulation process each segment in a segment matrix formed by dividing the image frame of the output image by the stabilization process into segments, and the approximated surface data that has been subjected to the geometric modulation process as learning input data, and a lattice point mesh segment matrix conversion unit that obtains approximated surface data indicative of a correspondence between each of the segments and the squares in the lattice point mesh in an intermediate layer of the third layered autoencoder using as input data the approximated surface data that has been subjected to the geometric modulation process. This allows the approximated surface data of the lattice point mesh to be input, and the approximated surface data (approximated surface data obtained by converting the lattice point mesh into the domain of the segment matrix) showing the correspondence between each segment and the cells in the lattice point mesh to be obtained. In other words, it becomes possible to eliminate the need for relatively heavy processing such as generating approximated surface data corresponding to the segment matrix in obtaining approximated surface data corresponding to the segment search result.

[0010] In the signal processing device according to the present technology described above, the remesh data obtained by a remesh data generation unit that generates remesh data indicating the motion information at the granularity of the segment based on the result of the segment search process and the motion information associated with each of the lattice points is used as teacher data, and a fourth layered autoencoder is machine-learned using the approximated surface data subjected to the geometric modulation process and the approximated surface data obtained by the lattice point mesh segment matrix conversion unit as learning input data, and a remesh approximated surface data generation unit is provided in an intermediate layer of the fourth layered autoencoder that obtains the approximated surface data of the remesh data using the approximated surface data subjected to the geometric modulation process, the approximated surface data obtained by the lattice point mesh segment matrix conversion unit, and the motion information associated with each of the lattice points as input data. This makes it possible to consistently perform processes from the geometric modulation process of the lattice point mesh to the generation of remesh data by using approximate surface data.

[0011] In the signal processing device related to the above-mentioned present technology, the motion information for each pixel position obtained by each pixel motion information calculation unit that calculates the motion information of each pixel position within the image frame of the output image from the remesh data obtained by the remesh data generation unit is used as teacher data, and a fifth stacked autoencoder that has been machine-learned using the approximate surface data obtained by the remesh approximate surface data generation unit and coordinate data specifying the pixel position as learning input data can be configured to include a motion information decoding unit that receives as input the approximate surface data obtained by the remesh approximate surface data generation unit and the coordinate data specifying the pixel position, and outputs the motion information at specified coordinates within the image frame of the output image. This makes it possible to appropriately decode motion information for each output pixel position required in the stabilization process from the approximated surface data equivalent to the remesh data.

[0012] In the signal processing device related to the above-mentioned present technology, the second stacked autoencoder possessed by the geometric modulation processing unit learns an algorithm for geometric modulation of the approximated surface data for each different parameter setting for the geometric modulation processing, and the geometric modulation processing unit can be configured to switch the algorithm depending on the parameter setting. This makes it possible to allow parameter settings to be changed in the geometric modulation process of the approximated surface data.

[0013] Furthermore, the signal processing method according to the present technology is a signal processing method in which, for a first layered autoencoder that has been pre-trained using coordinate data for each element of an object consisting of a plurality of elements as learning input data, approximate surface data is obtained by approximating the object in an intermediate layer of the first layered autoencoder using the coordinate data for each element as input data, and the approximate surface data is used as learning input data, and a result of performing a geometric modulation process on the object by coordinate transformation for each element is used as training data, and the geometric modulation process is performed on the approximate surface data by a second layered autoencoder that has been machine-trained using the approximate surface data as learning input data and as training data. With such a signal processing method, the same effect as that of the signal processing device according to the present technology described above can be obtained. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing an example of the configuration of a signal processing device according to an embodiment of the present technology; [Diagram 2] 4 is a block diagram showing an example of the internal configuration of a stabilization processing unit in the embodiment. [Diagram 3] FIG. 13 is a diagram showing an image captured by an inclined camera. [Figure 4] FIG. 1 is a diagram showing the relationship between IMU quaternion and image input. [Diagram 5] FIG. 13 is a diagram illustrating an example of a lattice point mesh. [Figure 6] FIG. 11 is an explanatory diagram of coordinate transformation of a lattice point mesh. [Figure 7] FIG. 13 is a diagram for explaining the relationship between a segment matrix and a lattice point mesh. [Figure 8] FIG. 11 is an explanatory diagram of a segment search in the embodiment. [Figure 9] FIG. 11 is an explanatory diagram of triangular interpolation for determining reference coordinates for each segment position. [Figure 10] FIG. 13 is a diagram illustrating an example of triangular interpolation. [Figure 11] FIG. 11 is an explanatory diagram of remesh data. [Figure 12] FIG. 13 is an image diagram of determining reference coordinates in pixel position units from remesh data. [Figure 13] 11A and 11B are explanatory diagrams of an interpolation process using an interpolation filter; [Figure 14] 4 is a functional block diagram illustrating a functional configuration of a reference coordinate calculation unit. FIG. [Figure 15] FIG. 13 is a diagram for explaining the functional configuration of a lattice point mesh generating and shaping unit. [Figure 16] FIG. 13 is a diagram for explaining machine learning for implementing geometric modulation processing on approximate surface data. [Figure 17] This is an explanatory diagram of CNN. [Figure 18] FIG. 2 is an explanatory diagram of encoding and decoding of input data in a stacked autoencoder. [Figure 19] FIG. 2 is an explanatory diagram of quadrilateral elements of a lattice point mesh. [Figure 20] 11 is a diagram illustrating an example of an approximation curve of a lattice point mesh when the lattice point mesh is assumed to be one-dimensional. FIG. [Figure 21] 11 is a diagram illustrating an approximation curve of a segment matrix when the segment matrix is ​​assumed to be one-dimensional; FIG. [Figure 22] FIG. 11 is an explanatory diagram of an extended decoder included in the remesh extended decoding learning device. [Diagram 23] FIG. 13 is a diagram illustrating an example of the configuration of a reference coordinate generation unit using each learning device that has been trained. [Figure 24] FIG. 13 is an explanatory diagram of a modified example of a lattice point mesh. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, with reference to the accompanying drawings, embodiments of the present technology will be described in the following order. <1. Overall configuration of the imaging device> <2. About stabilization processing> [2-1. Internal structure of the stabilization processing unit] [2-2. Stabilization processing method used in the embodiment] [2-3. Geometric Modulation Processing as an Embodiment] <3. Modifications> <4. Summary of the embodiment> <5. This Technology>

[0016] <1. Overall configuration of the imaging device> 1 is a block diagram showing an example of the configuration of a signal processing device 1 according to an embodiment of the present technology. Here, a case is illustrated in which the signal processing device 1 is applied to an imaging device serving as a digital camera device. Specifically, the signal processing device 1 performs signal processing for electronic image stabilizer (EIS) on an image captured by the imaging device. In this specification, the electronic image stabilizer processing is referred to as "stabilization processing."

[0017] As shown in the figure, the signal processing device 1 includes an IMU (Inertial Measurement Unit) sensor 2, an image sensor 3, a preprocessing unit 4, a preprocessing unit 5, a quaternion calculation unit 6, and a stabilization processing unit 7.

[0018] The image sensor 3 is, for example, a complementary metal oxide semiconductor (CMOS) type or charge coupled device (CCD) type image sensor, and obtains a captured image by photoelectrically converting light received by a plurality of light receiving elements arranged two-dimensionally.

[0019] The captured image obtained by the image sensor 3 is subjected to predetermined pre-processing, such as white balance adjustment and demosaic processing, in the pre-processing unit 5, and then output to the stabilization processing unit .

[0020] The IMU sensor 2 has a motion sensor that detects the motion of the imaging device having the signal processing device 1, and outputs a signal indicating the detection result of the motion. The IMU sensor 2 in this example has a three-axis acceleration sensor and an angular velocity sensor as the motion sensor. In this example, the detection signals of the acceleration sensor and the angular velocity sensor (three axes x two = six systems) are output as a signal indicating the detection result of the motion. Hereinafter, the signals output by the IMU sensor 2 and indicating the results of the detection of the movement will be collectively referred to as "IMU signals."

[0021] The pre-processing unit 4 receives the IMU signal from the IMU sensor 2 , performs a predetermined process as pre-processing, and outputs the result to the quaternion calculation unit 6 . The quaternion calculation unit 6 calculates a quaternion that represents the attitude of the imaging device based on the IMU signal input via the pre-processing unit 4.

[0022] The quaternion calculated by the quaternion calculation unit 6 (hereinafter referred to as “IMU quaternion”) is input to the stabilization processing unit 7.

[0023] The stabilization processing unit 7 performs stabilization processing on the captured image input via the pre-processing unit 5 based on the IMU quaternion input in this manner.

[0024] <2. About stabilization processing> [2-1. Internal structure of the stabilization processing unit] FIG. 2 is a block diagram showing an example of the internal configuration of the stabilization processor 7. As shown in FIG. In the following explanation, the coordinate system of the input image to the stabilization processing unit 7, i.e., in this example, the coordinate system of the captured image that has been preprocessed by the preprocessing unit 5, will be referred to as the "input coordinate system," and the coordinate system of the output image from the stabilization processing unit 7, i.e., the stabilized output image, will be referred to as the "output coordinate system."

[0025] In the stabilization processing unit 7, since a process of obtaining a stabilized output image by cutting out a part of the input image is performed as electronic image stabilization (EIS), it is assumed that the number of pixels of the input image is greater than the number of pixels of the output image. Specifically, in this example, the input image is a 4k image (number of horizontal pixels=about 4000, number of vertical pixels=about 2000), and the output image is a 2k image (number of horizontal pixels=about 2000, number of vertical pixels=about 1000).

[0026] As shown in FIG. 2, the stabilization processing unit 7 includes a reference coordinate calculation unit 11, a buffer control unit 12, a buffer memory 13, a memory control unit 14, a cache memory 15, and an interpolation filter 16. The IMU quaternion is input to the reference coordinate calculation unit 11 from the quaternion calculation unit 6. The reference coordinate calculation unit 11 calculates a reference coordinate CR for each pixel of the output image based on the IMU quaternion. The reference coordinates CR are information that indicates which position value in the input coordinate system should be used as the value of each pixel position in the output coordinate system when cutting out an output image from an input image. In other words, the reference coordinates CR are information that indicates which position value in the input coordinate system should be referenced for each pixel position in the output coordinate system.

[0027] The buffer memory 13 is a memory that sequentially buffers one frame of input image, and the buffer control unit 12 controls writing and reading of image data to and from the buffer memory 13 .

[0028] The cache memory 15 is a memory used for cutting out an output image from an input image, and the memory control unit 14 controls writing and reading of image data to and from the cache memory 15 .

[0029] The memory control unit 14 obtains image data corresponding to the cut-out range from the image data buffered in the buffer memory 13 via the buffer control unit 12 , and writes the image data in the cache memory 15 .

[0030] In addition, the memory control unit 14 reads out image data for multiple pixels including the pixel in the input coordinate system indicated by the reference coordinate CR and its surrounding pixels (for example, image data for 4×4=16 pixels if the interpolation filter 16 is Lanczos2 interpolation) from the image data (image data of the input image) cached in the cache memory 15 as described above, for each pixel position in the output coordinate system, and outputs it to the interpolation filter 16.

[0031] The interpolation filter 16 sequentially inputs the image data for multiple pixels read for each pixel position in the output coordinate system by the memory control unit 14 as described above, and performs an interpolation process for each pixel position in the output coordinate system using a method described later to determine the value of each pixel position in the output coordinate system. This results in a stabilized output image.

[0032] [2-2. Stabilization processing method used in the embodiment] The stabilization processing technique employed in this embodiment will be described with reference to FIGS. The stabilization process removes the effects of camera tilt and movement from the captured image. Figure 3 shows the image captured by the tilted camera. The tilted state here means that the camera is tilted in the roll direction and the horizontal and vertical directions are not maintained. In this case, the image data obtained by capturing the image shows the subject tilted as shown in Figure 3B. By rotating such image data in the same direction as the tilt of the camera through stabilization processing, the image data shown in Fig. 3C can be obtained. This image data in Fig. 3C is the same as an image captured by a camera in a straight position (with no tilt in the roll direction) as shown in Fig. 3D. Rotation relative to the inclination of the orientation is performed in this way, and this is realized by rotating the pixel range to be cut out based on the orientation information when cutting out an output image smaller than the input image size from the input image.

[0033] The relationship between the IMU quaternion and image input is shown in Figure 4. When capturing images while moving the camera, the IMU quaternion will change even during a single frame. If IMU data is acquired, for example, every several lines, IMU quaternions (represented by r0, r1, r2, and r3 in the figure) are also acquired every few lines as shown. Here, four IMU quaternions are acquired during one frame period indicated by the vertical synchronization signal Vsync, but this is merely an example for explanatory purposes. In this case, IMU quaternion r0 corresponds to the image in the upper 1 / 4 of the frame, IMU quaternion r1 to the next 1 / 4 of the image, IMU quaternion r2 to the next 1 / 4 of the image, and IMU quaternion r3 to the last 1 / 4 of the image. Here, the "virtual line L1" in the figure indicates a virtual line corresponding to an IMU quaternion of the same value.

[0034] Conventionally, based on the premise that IMU data is acquired multiple times within one frame period as described above, multiple virtual lines L1 each corresponding to the same IMU quaternion value are assumed, and reference coordinates CR are fitted to each pixel position in the output coordinate system according to these virtual lines L1, and the input image is cut out based on the fitted reference coordinates CR to obtain a stabilized image.

[0035] However, it has been found that sufficient stabilization performance cannot be obtained by the stabilization process using such a virtual line L1.

[0036] Therefore, in this embodiment, a method using a lattice point mesh as shown in FIG. 5 is adopted. The lattice point mesh has a plurality of lattice points (represented by ▲ marks in the figure) arranged in both the horizontal and vertical directions. In a lattice point mesh, a plurality of lattice point rows, each of which is made up of a plurality of lattice points arranged in the horizontal direction, are arranged in the vertical direction. Alternatively, this can be said as a plurality of lattice point columns, each of which is made up of a plurality of lattice points arranged in the vertical direction, being arranged in the horizontal direction. In the grid point mesh, each grid point row corresponds to the virtual line L1 shown in Fig. 4, and each grid point row is associated with an IMU quaternion based on IMU data acquired at a timing corresponding to the row position. In other words, the IMU quaternion value associated with each grid point is the same for each grid point row.

[0037] Note that the figure shows an example in which the number of lattice points in each lattice point row of the lattice point mesh is 6, i.e., the number of divisions in the horizontal direction is 5, and the number of lattice points in each lattice point column is 5, i.e., the number of divisions in the vertical direction is 4, but the numbers of divisions in the horizontal and vertical directions of the lattice point mesh are not limited to these values.

[0038] The position of each grid point in the grid point mesh is managed as a position in the input coordinate system to correspond to the timing of IMU data acquisition. The reference coordinate calculation unit 11 converts the positions of the grid points in the input coordinate system into positions in the output coordinate system.

[0039] FIG. 6 is an explanatory diagram of coordinate transformation of a lattice point mesh. To convert the positions of the grid points to positions in the output coordinate system, we simply apply the same changes to the grid point mesh as to the input image. Specifically, as shown in Figure 6, first, the grid point mesh is subjected to lens distortion removal processing in response to the input image being subjected to lens distortion removal processing, and then rotated to the same orientation as the camera. This is the result of the conversion to the output coordinate system.

[0040] In the stabilization process of this example, a lattice point mesh converted into the output coordinate system as described above and a segment matrix as shown in FIG. 7A are used. The segment matrix represents the position of each segment (indicated by a ● mark in the figure) when the output image (image frame of the output image by stabilization processing) is divided into a predetermined number of segments. In this example, the size of one segment is, for example, 64 pixels x 64 pixels.

[0041] FIG. 7B shows the lattice point mesh and the segment matrix that have been coordinate-transformed into the output coordinate system, superimposed in the output coordinate system. The reason why the size of the lattice point mesh is larger than the size of the segment matrix is ​​that the size of the input image is larger than the size of the output image, as described above. By converting the lattice point mesh into the output coordinate system, it becomes possible to identify the positional relationship between the position of each segment in the segment matrix (● mark) and each lattice point in the lattice point mesh, as shown in the figure.

[0042] A reference coordinate calculation unit 11 calculates a reference coordinate CR for each segment based on the positional relationship between each segment and a grid point in the output coordinate system. For this purpose, first, the reference coordinate calculation unit 11 performs a segment search as shown in FIG. The segment search is a process for determining in which square of the lattice point mesh the segment position indicated by a ● mark is located for each segment constituting the segment matrix. Specifically, the reference coordinate calculation unit 11 identifies the included segment position for each square in the lattice point mesh by inside / outside determination. This inside / outside determination identifies which square of the lattice point mesh each segment position is located within. The reference coordinate CR at each segment position can be calculated based on the IMU quaternions at each of the four grid points that contain that segment position. In the following explanation, it is assumed that each grid point in the grid point mesh is associated with information on the reference coordinate CR calculated from the corresponding IMU quaternion. Hereinafter, the reference coordinate CR associated with each grid point in this way will be referred to as the "grid point reference coordinate."

[0043] The reference coordinate calculation unit 11 determines in which square of the lattice point mesh each segment position is located by inside / outside determination (segment search), and then calculates the reference coordinate CR for each segment position by triangular interpolation as shown in Figure 9. Specifically, this triangular interpolation uses the coordinates of the segment position, the coordinates of three of the four lattice points of the square that contains the segment position in the lattice point mesh, and further information on the lattice point reference coordinates associated with those lattice points. This triangular interpolation may be performed, for example, in a manner as shown in FIG.

[0044] By calculating the reference coordinates CR of each segment position by triangular interpolation, remesh data such as that shown in Fig. 11 can be obtained. This remesh data is data indicating the reference coordinates CR of each position at the segment granularity in the output coordinate system. In the figure, the reference coordinates CR of each position at the segment granularity, i.e., the reference coordinates CR calculated for each segment position, are represented by the ◆ mark.

[0045] The reference coordinate calculation unit 11 calculates the reference coordinates CR for each pixel position in the output image based on the remesh data described above.

[0046] FIG. 12 is an image diagram of determining the reference coordinates CR in pixel position units from the remesh data, and in the figure, the reference coordinates CR in pixel position units are represented by ■ marks. In this example, the reference coordinates CR are calculated by linear interpolation (bilinear interpolation) using remesh data (reference coordinates CR at segment granularity). Specifically, the reference coordinates CR are calculated by bilinear interpolation using the reference coordinates CR of each of the four corners of the segment that contains the target pixel position. The reason why the triangular interpolation mentioned above is not used in this case is that bilinear interpolation is lighter than triangular interpolation, and once the data is converted to remesh data, bilinear interpolation can provide sufficient accuracy. However, when triangular interpolation is implemented as a hardware circuit in an LSI (Large Scale Integrated circuit), it is considered advantageous in terms of circuit size to use this block to triangularly interpolate all pixels rather than providing a separate bilinear interpolation circuit.

[0047] By determining the reference coordinate CR for each pixel position of the output image, it is possible to specify which position in the input coordinate system should be referenced for each pixel position. However, since the reference coordinate CR is calculated by an interpolation process based on the remesh data as described above, it may be a value that includes a decimal point rather than an integer unit (i.e., a pixel unit in the input image). For this reason, the interpolation filter 16 shown in FIG. 2 is used to render the output image based on the reference coordinate CR.

[0048] FIG. 13 is an explanatory diagram of the interpolation process performed by the interpolation filter 16. As shown in FIG. Under the control of the memory control unit 14, pixel values ​​of a plurality of pixels required for rendering of each output pixel are sequentially input to the interpolation filter 16 from among the input image (pixel values) cached in the cache memory 15. Specifically, the pixel values ​​of a plurality of pixels required for rendering of each output pixel are data of an area consisting of a plurality of pixels including a pixel including a position in the input coordinate system indicated by the reference coordinate CR for the output pixel and pixels surrounding the pixel (see the area Ar surrounded by a thick frame in the drawing). For the sake of explanation, in the following, the pixel that includes the position of the input coordinate system indicated by the reference coordinate CR will be referred to as the "reference pixel Pr." Also, the pixel area required for rendering, including this reference pixel Pr and its surrounding pixels, will be referred to as the "reference area Ar." The reference area Ar is an area of ​​m pixels x m pixels (m is a natural number of 3 or more) centered on the reference pixel Pr. Note that in the drawings, the reference area Ar is an area of ​​3 pixels x 3 pixels = 9 pixels centered on the reference pixel Pr, but this is an example for explanation and does not limit the size of the reference area Ar.

[0049] The interpolation filter 16 obtains the value of the position indicated by the reference coordinate CR for the output pixel to be processed by an interpolation process using the values ​​of each pixel in the reference area Ar. For example, a Lanczos filter is used for this interpolation process. Specifically, a Lanczos2 filter or a hybrid filter blended with a Gaussian filter from the viewpoint of preventing aliasing can be used. This hybrid filter is effective for Lanczos2 interpolation in a RAW format in which the image format is arranged in RGGB, and is used to prevent aliasing in particular in the high frequency band. The interpolation filter 16 performs such an interpolation process for each output pixel in sequence, thereby obtaining a stabilized output image.

[0050] In rendering a stabilized output image, brightness adjustment for lens shading can be performed in addition to the calculation of pixel values ​​by the above-mentioned interpolation process of the interpolation filter 16. In that case, brightness control information and the like are provided for the grid points in addition to the reference coordinates CR, and gain adjustment is performed together with the interpolation process. Similarly, by having the grid points carry local motion information obtained from SLAM technology for self-location estimation and depth information from depth sensors such as ToF sensors that obtain image depth, the information can also be used for highly flexible coordinate correction and depth correction on a grid point basis.

[0051] FIG. 14 is a functional block diagram illustrating the functional configuration of the reference coordinate calculation unit 11 shown in FIG. As shown in the figure, the reference coordinate calculation unit 11 has functions as a lattice point mesh generating / shaping unit 21, a segment matrix generating unit 22, a segment searching unit 23, a remesh data generating unit 24, and each pixel coordinate interpolation unit 25.

[0052] The lattice point mesh generating / shaping unit 21 performs processes for generating a lattice point mesh, rotating the lattice point mesh for conversion to the output coordinate system described above (see FIG. 6), and forming the lattice point mesh.

[0053] Fig. 15 is a diagram for explaining the functional configuration of the lattice point mesh generating / forming unit 21. Note that Fig. 15 also shows an image diagram that shows a schematic representation of the process of forming a lattice point mesh, together with the functional configuration of the lattice point mesh generating / forming unit 21. As shown in the figure, the lattice point mesh generating and shaping unit 21 has the functions of a lattice point mesh generator 31, a lens distortion corrector 32, a projector 33, a rotator 34, a free curvature perspective projector 35, a scan controller 36, a clipper 37, and a lattice point reference coordinate calculator 38.

[0054] The lattice point mesh generating / shaping unit 21 generates and shapes a lattice point mesh. The lens distortion corrector 32 performs lens distortion correction processing on the lattice point mesh based on the lens parameters.

[0055] The projector 33 projects the lattice point mesh after the lens distortion correction process by the lens distortion corrector 32 onto the virtual celestial sphere. As the projection method, for example, central projection, equidistant projection, etc. can be adopted (the image diagram in the figure shows an example of central projection). The projector 33 in this example performs projection processing by either central projection or equidistant projection based on the projection parameters.

[0056] The rotator 34 rotates the lattice point mesh projected onto the virtual celestial sphere by the projector 33 based on the IMU quaternion. This rotation has the effect of rotating the mesh in the same direction as the camera as described above. For the rotation, information indicating the amount of rotation in the IMU quaternion (parameter of the amount of rotation) is used.

[0057] The free curvature perspective projector 35 projects (reprojects) the lattice point mesh rotated by the rotator 34 onto a plane by free curvature perspective projection based on the projection parameters. By employing free curvature perspective projection, it is possible to impart a desired lens effect to the reprojected lattice point mesh, and to create an output image. The projection parameters are parameters for specifying the form of such a lens effect. The scanning controller 36 performs affine transformation processing for appropriate scale setting and offset change for the lattice point mesh projected onto the plane. The scanning controller 36 performs the scale setting and offset change based on predetermined parameters such as predefined scale / offset parameters.

[0058] The clipper 37 performs clipping processing based on clip parameters on the lattice point mesh after processing by the scan controller 36 to prevent the data from exceeding the fixed-point precision. If a lattice point exceeds the fixed-point data range, the segment to which that lattice point belongs is treated as an invalid segment, and exceptional processing for black screen output is performed at a later stage. By the clipping process by the clipper 37, the coordinates of each lattice point in the output coordinate system are determined.

[0059] The grid point reference coordinate calculator 38 calculates the reference coordinates (the grid point reference coordinates described above) of each grid point in the grid point mesh based on the IMU quaternion.

[0060] Returning to the explanation of FIG. The coordinate information in the output coordinate system of each lattice point determined by the clipping process of the clipper 37 as described above is supplied from the lattice point mesh generating / shaping unit 21 to the segment searching unit 23 as each lattice point coordinate information. Moreover, the respective lattice point reference coordinates obtained by the respective lattice point reference coordinate calculator 38 are supplied from the lattice point mesh generating / shaping unit 21 to the remesh data generating unit 24 .

[0061] The segment search unit 23 performs the above-mentioned segment search (inside / outside determination: see Figs. 7 and 8) based on the segment matrix generated by the segment matrix generation unit 22 and each lattice point coordinate information supplied from the lattice point mesh generation / shaping unit 21. As a result, for each segment position in the segment matrix, four lattice points that include that segment position are specified.

[0062] The remesh data generation unit 24 performs the above-mentioned triangular interpolation (see Figs. 9 and 10) for each segment position based on the information of each lattice point reference coordinate supplied from the lattice point mesh generation / shaping unit 21 and the information of the segment search result by the segment search unit 23, to generate remesh data (see Fig. 11). As described above, the remesh data can be said to be in other words the reference coordinates CR at the segment granularity. The remesh data generating unit 24 outputs the generated remesh data to each pixel coordinate interpolation unit 25 .

[0063] Each pixel coordinate interpolation unit 25 determines a reference coordinate CR for each pixel position of the output image based on the remesh data. As described above, the reference coordinate CR for each pixel position is determined by performing bilinear interpolation based on the remesh data. Each pixel coordinate interpolation unit 25 outputs the reference coordinates CR of each pixel position to the memory control unit 14 shown in FIG.

[0064] The memory control unit 14 shown in FIG. 2 controls writing of data from the buffer memory 13 to the cache memory 15 based on the reference coordinates CR. Furthermore, the memory control unit 14 sequentially reads out data of a reference area Ar (see FIG. 13) corresponding to each pixel position of the output image from the cache memory 15 based on the reference coordinates CR, and outputs the data to the interpolation filter 16. As a result, the interpolation filter 16 sequentially performs an interpolation process using the data of the reference area Ar for each pixel position of the output image, thereby obtaining a stabilized output image.

[0065] As described above, in the stabilization processing method adopted in this embodiment, when determining the reference coordinate CR for each pixel position of the output image, consistency with the output coordinate system is achieved using two-dimensional information in the form of a lattice point mesh, rather than using only one-dimensional information in the form of a virtual line L1 as in the conventional method. This makes it possible to increase the accuracy of the reference coordinates CR and improve the performance of the stabilization process.

[0066] [2-3. Geometric Modulation Processing as an Embodiment] As can be seen by referring to FIG. 15, in this embodiment, the lattice point mesh is subjected to geometric modulation processing such as lens distortion correction processing by the lens distortion corrector 32 (i.e., deformation of the lattice point mesh), rotation by the rotator 34, and affine transformation processing by the scanning controller 36 (i.e., movement and enlargement / reduction of the lattice point mesh). Here, the geometric modulation process means a process of changing at least one of the shape, position, orientation, and size of an object. The object referred to here is one that is made up of a plurality of elements, and a lattice point mesh is one example of such an object. In other words, it can be said that a lattice point mesh is an object that is made up of a plurality of lattice points as its elements. Alternatively, in a lattice point mesh, if one square surrounded by four adjacent lattice points is considered to be a polygon, it can also be said that the lattice point mesh is an object that is made up of a plurality of polygons as its elements.

[0067] As mentioned earlier, geometric modulation processing for an object consisting of multiple elements is generally performed as a coordinate transformation process for each element, but performing coordinate transformation for each element individually increases the computational cost required for the geometric modulation processing, which is undesirable.

[0068] Therefore, in this embodiment, a method is adopted in which a lattice point mesh as an object is approximated as a curved surface by utilizing the self-teaching learning function of a stacked autoencoder (hereinafter referred to as "SAE") used in the field of AI (Artificial Intelligence), and geometric modulation processing is performed on the approximated curved surface data.

[0069] FIG. 16 is a diagram for explaining machine learning for implementing the geometric modulation process on the approximated surface data as described above. First, in the learning environment, the lattice point mesh generating and shaping unit 21 includes a lens distortion corrector 32p, a projector 33p, a rotator 34p, a free curvature perspective projector 35p, a scan controller 36p, a clipper 37p, and a lattice point reference coordinate calculator 38. The lens distortion corrector 32p, the projector 33p, the rotator 34p, the free curvature perspective projector 35p, the scan controller 36p, and the clipper 37p are functionally similar to the lens distortion corrector 32, the projector 33, the rotator 34, the free curvature perspective projector 35, the scan controller 36, and the clipper 37, respectively, but differ in that each process is performed as a coordinate conversion process of each element constituting the lattice point mesh. Specifically, the lens distortion corrector 32p performs lens distortion correction processing of the lattice point mesh as a coordinate conversion process of each lattice point in the lattice point mesh. Also, the projector 33p performs projection processing of the lattice point mesh onto the virtual celestial sphere described above as a coordinate conversion process of each lattice point. For confirmation, this projection processing involves conversion processing of the coordinates of the lattice points from binary coordinates to three-dimensional coordinates. In addition, the rotator 34p performs a rotation process on the lattice point mesh after projection onto the virtual celestial sphere as a coordinate conversion process for each lattice point, and the free curvature perspective projector 35p performs a reprojection process on the rotated lattice point mesh as a coordinate conversion process for each lattice point (in this case, a conversion process from three-dimensional coordinates to two-dimensional coordinates is involved). The scan controller 36p performs an affine transformation process on the reprojected lattice point mesh as a coordinate conversion process for each lattice point, and the clipper 37p performs a clip process on the affine transformed lattice point mesh as a coordinate conversion process for each lattice point.

[0070] In addition, in the learning environment, a lattice point mesh approximation curved surface generation unit 39, a lens distortion correction learner 32b, a projection learner 33b, a rotation learner 34b, a free curvature perspective projection learner 35b, a scanning control learner 36b, and a clip learner 37b are used. A machine learning machine using a DNN (Deep Neural Network) is used as the lattice point mesh approximation curved surface generating unit 39, the lens distortion correction learner 32b, the projection learner 33b, the rotation learner 34b, the free curvature perspective projection learner 35b, the scan control learner 36b, and the clip learner 37b. Specifically, in this example, a machine learning machine using a CNN (Convolutional Neural Network) is used.

[0071] FIG. 17 is an explanatory diagram of CNN. In the following explanation, detailed descriptions of general DNN techniques such as the number of input / output taps and layers in the DNN, firing functions, and DropOut methods are omitted because they are not the essence of this technology. Note that the specific number of layers and the number of taps in the DNN shown below are merely examples for the purpose of explanation and are not limited to these.

[0072] CNN, which is the most widespread DNN technology, has a stacked autoencoder (SAE) that repeats convolution and pooling, followed by a fully connected layer, as shown in Figure 17. For example, a commonly known network structure is AlexNet. In CNN, a pre-training process for SAE is performed. The pre-training process is a type of unsupervised learning (also called semi-supervised learning), and refers to a process of learning so that the output matches the input. Then, a recognition algorithm can be generated by supervised learning (called fine tuning) in the subsequent fully connected layer. However, nowadays, CNN is not limited to this and there are various derivatives of CNN.

[0073] FIG. 18 is a diagram for explaining encoding and decoding of input data in SAE. In SAE, input data given to the input layer is encoded. This encoding process is a dimensionality reduction process for the input data, and dimensionality reduced data of the input data is obtained in the intermediate layer. The encoded data is decoded from the intermediate layer to the output layer.

[0074] Here, SAE is generally intended to reduce dimensions, but by performing pre-training to match the input and output, it has the function of self-teaching and learning the target feature representation. To approximate a surface from a lattice mesh, the self-teaching learning function of SAE is utilized.

[0075] Specifically, the lattice point mesh approximation curved surface generating unit 39 performs pre-training for SAE using the lattice point mesh (coordinate data for each lattice point) generated by the lattice point mesh generator 31 as input data. When a lattice point mesh is input to the SAE of the lattice point mesh approximation unit 39, the dimensionality of the lattice point mesh is reduced in the intermediate layer of the SAE, and by performing the pre-training as described above, data of an approximated surface expressing the shape of the lattice point mesh is obtained by the dimensionality reduction in the intermediate layer. That is, data obtained in the intermediate layer is the approximation of the lattice point mesh to a curved surface.

[0076] In Figure 16, for the lens distortion correction learner 32b, the projection learner 33b, the rotation learner 34b, the free curvature perspective projection learner 35b, the scanning control learner 36b, and the clip learner 37b, machine learning (the aforementioned Fine Tuning) is performed using the outputs (processing results) of the lens distortion corrector 32p, the projector 33p, the rotator 34p, the free curvature perspective projector 35p, the scanning controller 36p, and the clipper 37p, respectively, as teacher data. Specifically, the lens distortion correction learning device 32b performs machine learning using the lattice point mesh approximated surface data obtained in the intermediate layer of SAE in the lattice point mesh approximation curved surface generation unit 39 as learning input data and the output of the lens distortion corrector 32p as teacher data. By performing such machine learning, an algorithm is generated in the learning device serving as lens distortion correction learning device 32b for applying a correction (deformation) to the input approximated surface data similar to the correction process performed by lens distortion corrector 32p. Therefore, when implementing geometric modulation processing as lens distortion correction processing for a lattice point mesh, there is no need to perform coordinate conversion processing on the coordinate data of each lattice point, and the computational costs for the geometric modulation processing can be reduced.

[0077] The projection learning device 33b performs machine learning using the approximate surface data of the lattice point mesh obtained in the intermediate layer of SAE in the lens distortion correction learning device 32b, which is the learning device arranged immediately before, as learning input data and the output of the projector 33p as teacher data. By performing this type of machine learning, an algorithm is generated in the learning device serving as the projection learning device 33b to impart geometric modulation (deformation) to the input approximate surface data in the same manner as when projection processing is performed by the projector 33p. Therefore, when implementing geometric modulation processing as a projection process onto a lattice point mesh, it is no longer necessary to perform coordinate transformation processing on the coordinate data of each lattice point, and the computational costs for the geometric modulation processing can be reduced.

[0078] For the rotation learner 34b, the free curvature perspective projection learner 35b, the scanning control learner 36b, and the clip learner 37b, machine learning is performed using the output of the corresponding one of the rotator 34p, the free curvature perspective projector 35p, the scanning controller 36p, and the clipper 37p as teacher data, and using the approximate surface data obtained in the intermediate layer of the SAE in the learner placed immediately before as learning input data. As a result, in the learning devices such as the rotation learning device 34b, the free curvature perspective projection learning device 35b, the scanning control learning device 36b, and the clip learning device 37b, algorithms are generated for applying geometric modulation to the input approximate surface data similar to that achieved when rotation processing by the rotator 34p, reprojection processing by the free curvature perspective projector 35p, affine transformation processing by the scanning controller 36p, and clipping processing by the clipper 37p are performed. Therefore, it is not necessary to perform coordinate conversion processing for each lattice point in the rotation processing, reprojection processing, affine transformation processing, and clipping processing, and it is possible to reduce the calculation cost.

[0079] Here, in this example, each of the learners, the lens distortion correction learner 32b, the projection learner 33b, the rotation learner 34b, the free curvature perspective projection learner 35b, the scanning control learner 36b, and the clip learner 37b, learns an algorithm for geometric modulation of the input approximate surface data for each different parameter setting for the corresponding geometric modulation process. Specifically, the lens distortion correction learning device 32b learns an algorithm for geometric modulation of the input approximated surface data for each different lens parameter setting. For example, when two types of lens parameter settings, "A" and "B", are possible, learning for the lens parameter A is performed using the output of the lens distortion corrector 32p in the lens parameter A setting state as teacher data, so that an algorithm for the lens parameter A is generated. Learning for the lens parameter B is performed using the output of the lens distortion corrector 32p in the lens parameter B setting state as teacher data, so that an algorithm for the lens parameter B is generated. At this time, the lens distortion correction learning device 32b stores the algorithm generated for each parameter setting in such a way that it can be identified as to which parameter setting the algorithm corresponds to.

[0080] The projection learner 33b, the rotation learner 34b, the free curvature perspective projection learner 35b, the scan control learner 36b, and the clip learner 37b also perform learning for each parameter setting in a similar manner, and the algorithms generated for each parameter setting by the learning are stored in a manner that allows identification of which parameter setting the algorithm corresponds to.

[0081] In this example, the above-mentioned segment matrix and remesh data are also treated as approximate surface data, thereby further reducing the calculation cost. For this purpose, in the learning environment of this example, as shown in Figure 16, a segment matrix generation unit 22, a segment search unit 23, a remesh data generation unit 24, and a pixel coordinate interpolation unit 25 are provided, as well as a lattice point mesh segment matrix conversion learner 26b, a remesh learner 27b, and a remesh extension decoding learner 28b. In this example, a CNN-based learning device is used for each of the lattice point mesh segment matrix conversion learning device 26b, the remesh learning device 27b, and the remesh extension decoding learning device 28b.

[0082] The lattice point mesh segment matrix conversion learner 26b performs machine learning using the approximated surface data (approximated surface data of a lattice point mesh) obtained in the intermediate layer of SAE in the clip learner 37b as learning input data and the segment search processing result by the segment search unit 23 as teacher data. The data of the segment search processing result here is data indicating, for each segment position (segment number) in the segment matrix, the number of a quadrilateral element of the lattice point mesh that contains that segment position.

[0083] FIG. 19 is an explanatory diagram of quadrilateral elements of a lattice point mesh. As shown in the figure, a quadrilateral element of the lattice point mesh refers to a square portion surrounded by four adjacent lattice points (four lattice points that are adjacent in the row, column, and diagonal directions), and the number of the quadrilateral element refers to the number assigned to each square.

[0084] In the lattice point mesh segment matrix transformation learning device 26b, an algorithm is generated by the above-mentioned machine learning to realize a domain transformation from the approximated surface of the lattice point mesh to the approximated surface of the segment matrix. The principle of such domain transformation will be described with reference to Fig. 20 and Fig. 21 using a one-dimensional example. Fig. 20 illustrates an approximate curve of a lattice point mesh when the lattice point mesh is assumed to be one-dimensional. The horizontal axis represents the element number, and the vertical axis represents the coordinate. As can be seen from this one-dimensional example, the approximated curved surface data of the lattice point mesh can be expressed as equivalent to a relational expression between the numbers of the quadrilateral elements of the lattice point mesh and the coordinates. 21 shows an example of an approximation curve of a segment matrix when the segment matrix is ​​assumed to be one-dimensional. The horizontal axis is the segment number, and the vertical axis is the number of a quadrilateral element of a lattice point mesh. The approximation curve of a segment matrix can be expressed as equivalent to a relational expression between the number of a quadrilateral element of a lattice point mesh and the segment number. 20 and 21, it can be seen that the approximated surface of the segment matrix has a deformable correlation with the approximated surface of the lattice point mesh.

[0085] In the lattice point mesh segment matrix conversion learner 26b, machine learning is performed using the approximate surface data from the clip learner 37b as described above as learning input data and the segment search processing result by the segment search unit 23 as teaching data, thereby generating an algorithm that converts the lattice point mesh approximate surface data input from the clip learner 37b into segment matrix approximate surface data (the relationship between the numbers of the quadrangular elements of the lattice point mesh and the segment numbers).

[0086] The remesh learner 27b performs machine learning using the approximate surface data of the segment matrix obtained in the intermediate layer of SAE in the lattice point mesh segment matrix conversion learner 26b and the approximate surface data of the lattice point mesh obtained in the intermediate layer of SAE in the clip learner 37b as learning input data, and the output of the remesh data generation unit 24 (reference coordinates CR at segment granularity) as teacher data. By this machine learning, in the remesh learning device 27b, an algorithm is generated for obtaining approximate surface data corresponding to the relational expression between each segment position (number) in the segment matrix and the reference coordinate CR corresponding to that segment position.

[0087] The remesh extension decoding learner 28b performs machine learning using the approximate surface data obtained in the intermediate layer of the remesh learner 27b and coordinate data indicating each pixel position of the output image (shown as x, y in the figure because the output coordinate system is two-dimensional) as learning input data, and the output of each pixel coordinate interpolation unit 25 (i.e., the reference coordinates CR for each pixel position of the output image) as teacher data.

[0088] FIG. 22 is an explanatory diagram of the extended decoder in SAE that the remesh extended decoding learning device 28b has. The remesh extension decode learning device 28b receives the approximate surface data from the remesh learning device 27b as an input, and performs learning to output the reference coordinates CR of the pixel position specified by the x, y coordinate data.

[0089] By performing such learning, the remesh extension decode learning device 28b generates an algorithm for decoding the reference coordinates CR of the pixel position specified by the x, y coordinate data from the approximate surface data input from the remesh learning device 27b.

[0090] FIG. 23 is a diagram showing an example of the configuration of the reference coordinate calculation unit 11 using each learning device that has already been trained. As shown in the figure, the lattice point mesh generating / shaping unit 21 in the reference coordinate calculation unit 11 is equipped with a lattice point mesh generator 31, and a lattice point reference coordinate calculator 38, as well as a lattice point mesh approximation curved surface unit 39, a lens distortion correction learner 32a, a projection learner 33a, a rotation learner 34a, a free curvature perspective projection learner 35a, a scan control learner 36a, and a clip learner 37a. Here, lens distortion correction learner 32a, projection learner 33a, rotation learner 34a, free curvature perspective projection learner 35a, scanning control learner 36a, and clip learner 37a respectively represent lens distortion correction learner 32b, projection learner 33b, rotation learner 34b, free curvature perspective projection learner 35b, scanning control learner 36b, and clip learner 37b, which have been considered to have completed learning.

[0091] The reference coordinate calculation unit 11 is also provided with a lattice point mesh segment matrix conversion learner 26a, a remesh learner 27a, and a remesh extension decode learner 28a. The lattice point mesh segment matrix conversion learner 26a, the remesh learner 27a, and the remesh extension decode learner 28a represent the trained lattice point mesh segment matrix conversion learner 26b, the remesh learner 27b, and the remesh extension decode learner 28b, respectively.

[0092] In the lattice point mesh generating / shaping unit 21 , the lattice point mesh approximating curved surface unit 39 receives as input coordinate data for each lattice point of the lattice point mesh generated by the lattice point mesh generator 31 . Here, for the lattice point mesh, common data may be used for each frame. Therefore, the process of generating approximated surface data by the lattice point mesh approximation unit 39 may be performed at least once (the generated approximated surface data may be stored in memory and read out sequentially for each frame).

[0093] As shown in the figure, the lens distortion correction learning device 32a is provided with approximated surface data from the lattice point mesh approximation unit 39 as input data, and the projection learning device 33a is provided with approximated surface data that has been subjected to geometric modulation processing as lens distortion correction processing by the lens distortion correction learning device 32a as input data. The rotation learning device 34a is provided with approximated surface data that has been subjected to geometric modulation processing as projection processing by the projection learning device 33a as input data, and the free curvature perspective projection learning device 35a is provided with approximated surface data that has been subjected to geometric modulation processing as rotation processing by the rotation learning device 34a as input data. The scan control learning device 36a is provided with approximated surface data that has been subjected to geometric modulation processing as reprojection processing by the free curvature perspective projection learning device 35a as input data, and the clip learning device 37a is provided with approximated surface data that has been subjected to geometric modulation processing as affine transformation processing by the scan control learning device 36a as input data.

[0094] As can be understood from the above description, in this example, the lens distortion correction learner 32a, the projection learner 33a, the rotation learner 34a, the free curvature perspective projection learner 35a, the scan control learner 36a, and the clip learner 37a learn algorithms for geometric modulation processing of approximated surface data for each different parameter setting. The lens distortion correction learner 32a, the projection learner 33a, the rotation learner 34a, the free curvature perspective projection learner 35a, the scan control learner 36a, and the clip learner 37a have a function of switching algorithms so as to use an algorithm corresponding to the parameter setting from among the multiple learned algorithms.

[0095] The lattice point mesh segment matrix conversion learning device 26a is provided with lattice point mesh approximate surface data that has been subjected to geometric modulation processing as clip processing by the clip learning device 37a as input data.

[0096] The remesh learner 27a is provided with, as input data, the approximate surface data of the segment matrix obtained by the conversion process in the lattice point mesh segment matrix conversion learner 26a, the lattice point reference coordinates from the lattice point reference coordinate calculator 38, and the approximate surface data from the clip learner 37a.

[0097] The remesh extension decode learning device 28a is provided with, as input data, approximate surface data (approximate surface data corresponding to the relational expression between the segment position and the reference coordinate CR) for the remesh data obtained by the remesh learning device 27a and x, y coordinate data for specifying the pixel position of the output image. As a result, the remesh extension decode learning device 28a decodes the reference coordinate CR of the pixel position specified by the x, y coordinate data from the approximate surface data input by the remesh learning device 27a and outputs it.

[0098] <3. Modifications> It should be noted that the present technology is not limited to the specific examples described above, but may take on various modified configurations. For example, in the above, an example was given of generating a lattice point mesh in a fixed mesh mode as shown in Figure 24A. However, to generate the lattice point mesh, modes other than the fixed mesh mode can also be adopted, such as a one-dimensional variable mesh mode, a two-dimensional variable mesh mode, or a finite element mesh mode as exemplified in Figures 24B to 24D.

[0099] Furthermore, in the above, an example has been given in which the present technology is applied to the stabilization processing in the signal processing device 1, but the present technology can be widely and preferably applied to cases in which geometric modulation processing is performed on an object made up of a plurality of elements.

[0100] In addition, in the above, lens distortion correction processing, projection processing onto a virtual celestial sphere, rotation processing on the virtual celestial sphere, reprojection processing, affine transformation processing for changing scale or offset, and clipping processing are given as examples of geometric modulation processing for an object, but the geometric modulation processing to which the present technology can be applied is not limited to these processing.

[0101] <4. Summary of the embodiment> As described above, the signal processing device (same as reference 1) as an embodiment includes a first layered autoencoder that is pre-trained using the coordinate data of each element of an object consisting of a plurality of elements as learning input data, an approximation unit (lattice point mesh approximation unit 39) that uses the coordinate data of each element as input data to obtain approximate surface data by approximating the object in an intermediate layer of the first layered autoencoder, and a second layered autoencoder that is machine-trained using the approximate surface data as learning input data and the result of performing geometric modulation processing on the object by coordinate transformation of each element as teacher data, and a geometric modulation processing unit (e.g., lens distortion correction learning device 32a, etc.) that performs geometric modulation processing on the approximate surface data using the second layered autoencoder. According to the above configuration, geometric modulation processing such as deformation, movement, and rotation of an object is performed on approximate surface data obtained by the first stacked autoencoder by dimensionally compressing the coordinate data of each element. Therefore, compared to performing coordinate transformation for each element of an object, it is possible to reduce the computational cost involved in the geometric modulation processing of an object.

[0102] In addition, in the signal processing device as an embodiment, a stabilization process of an image captured by an imaging device is performed using a lattice point mesh having multiple lattice points arranged in the horizontal and vertical directions in the output coordinate system of the captured image, and in which movement information of the imaging device is associated with each lattice point, and the object is treated as the lattice point mesh. As a result, the geometric modulation processing such as deformation and rotation of the lattice point mesh, which is performed in the stabilization processing using the lattice point mesh, is performed on the approximate curved surface data of the lattice point mesh. Therefore, the computational costs associated with the geometric modulation of the lattice point mesh can be reduced, and the computational costs required for the stabilization process can be reduced.

[0103] Furthermore, in the signal processing device as an embodiment, the result of a segment search process that identifies in which of the squares in the lattice point mesh that has been subjected to geometric modulation processing each segment in a segment matrix formed by dividing the image frame of the output image by stabilization processing into segment units is located is used as teacher data, and a third layered autoencoder that has been machine-learned using the approximated surface data that has been subjected to geometric modulation processing as learning input data is provided, and the third layered autoencoder is provided with a lattice point mesh segment matrix conversion unit (lattice point mesh segment matrix conversion learning device 26a) that uses the approximated surface data that has been subjected to geometric modulation processing as input data to obtain approximated surface data that indicates the correspondence between each segment and the squares in the lattice point mesh in the intermediate layer of the third layered autoencoder. This allows the approximated surface data of the lattice point mesh to be input, and the approximated surface data (approximated surface data obtained by converting the lattice point mesh into the domain of the segment matrix) showing the correspondence between each segment and the cells in the lattice point mesh to be obtained. In other words, it becomes possible to eliminate the need for relatively heavy processing such as generating approximated surface data corresponding to the segment matrix in obtaining approximated surface data corresponding to the segment search result. Therefore, the calculation cost for the stabilization process can be reduced.

[0104] Furthermore, in the signal processing device as an embodiment, the remesh data obtained by the remesh data generation unit (24) generates remesh data indicating motion information at the granularity of a segment based on the result of the segment search processing and the motion information associated with each lattice point, and the fourth layered autoencoder is machine-learned using the approximated surface data that has been subjected to geometric modulation processing and the approximated surface data obtained by the lattice point mesh segment matrix conversion unit as learning input data, and is equipped with a remesh approximated surface data generation unit (remesh learner 27a) that obtains the approximated surface data of the remesh data in the intermediate layer of the fourth layered autoencoder using the approximated surface data that has been subjected to geometric modulation processing, the approximated surface data obtained by the lattice point mesh segment matrix conversion unit, and the motion information associated with each lattice point as input data. This makes it possible to consistently perform processes from the geometric modulation process of the lattice point mesh to the generation of remesh data by using approximate surface data. Therefore, the calculation cost for the stabilization process can be reduced.

[0105] In addition, in the signal processing device as an embodiment, the motion information for each pixel position obtained by each pixel motion information calculation unit (each lattice point reference coordinate calculator 38) that calculates motion information for each pixel position within the image frame of the output image from the remesh data obtained by the remesh data generation unit is used as teacher data, and a fifth stacked autoencoder that has been machine-learned using the approximate surface data obtained by the remesh approximate surface data generation unit and coordinate data specifying the pixel position as learning input data is provided, and a motion information decoding unit (remesh extended decoding learner 28a) that receives as input the approximate surface data obtained by the remesh approximate surface data generation unit and the coordinate data specifying the pixel position, and outputs motion information at specified coordinates within the image frame of the output image. This makes it possible to appropriately decode motion information for each output pixel position required in the stabilization process from the approximated surface data equivalent to the remesh data.

[0106] Furthermore, in a signal processing device as an embodiment, the second stacked autoencoder possessed by the geometric modulation processing unit learns an algorithm for geometric modulation of approximate surface data for each different parameter setting for the geometric modulation processing, and the geometric modulation processing unit switches the algorithm depending on the parameter setting. This makes it possible to allow parameter settings to be changed in the geometric modulation process of the approximated surface data. Therefore, a single signal processing device can handle geometric modulation processes using different parameter settings, eliminating the need to prepare different signal processing devices for each parameter setting. In addition, it is possible to deal with cases where dynamic switching of parameter settings is permitted. For example, in a camera system in which the geometric modulation process is a deformation process as a lens distortion correction process and the lens is interchangeable, it is necessary to dynamically change the parameter settings of the lens distortion correction process in response to the lens exchange, but such a case can be dealt with by a single signal processing device.

[0107] Moreover, a signal processing method as an embodiment is a signal processing method in which a first layered autoencoder is pre-trained using coordinate data for each element of an object consisting of a plurality of elements as learning input data, and approximate surface data is obtained by approximating an object in an intermediate layer of the first layered autoencoder using the coordinate data for each element as input data, and geometric modulation processing is performed on the approximate surface data by a second layered autoencoder that has been machine-trained using the approximate surface data as learning input data and the result of performing geometric modulation processing on the object by coordinate transformation for each element as training data. With this first signal processing method as well, it is possible to obtain the same functions and effects as with the first signal processing device described above.

[0108] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0109] <5. This Technology> The present technology can also be configured as follows. (1) a first layered autoencoder that is pre-trained using coordinate data for each element of an object consisting of a plurality of elements as learning input data, and an approximation surface generating unit that obtains approximated surface data by approximating the object to a middle layer of the first layered autoencoder using the coordinate data for each element as input data; a second layered autoencoder that is machine-trained using the approximated surface data as learning input data and a result of performing a geometric modulation process on the object by coordinate transformation for each element as training data, and a geometric modulation processing unit that performs the geometric modulation process on the approximated surface data using the second layered autoencoder. Signal processing device. (2) performing a stabilization process for an image captured by an imaging device using a lattice point mesh having a plurality of lattice points arranged in a horizontal direction and a plurality of lattice points arranged in a vertical direction in an output coordinate system of the captured image, the stabilization process being performed using the lattice point mesh in which motion information of the imaging device is associated with each of the lattice points; The object is the lattice point mesh. The signal processing device according to (1) above. (3) a third layered autoencoder that uses as training data a result of a segment search process that identifies in which of the squares in the lattice point mesh that has been subjected to the geometric modulation process each segment in a segment matrix formed by dividing an image frame of an output image by the stabilization process is located, and that performs machine learning using the approximated curved surface data that has been subjected to the geometric modulation process as learning input data; and a lattice point mesh segment matrix conversion unit that obtains, in an intermediate layer of the third layered autoencoder, approximated curved surface data that indicates a correspondence between each of the segments and the squares in the lattice point mesh, using the approximated curved surface data that has been subjected to the geometric modulation process as input data. The signal processing device according to (2) above. (4) The fourth layered autoencoder includes a remesh data generating unit that generates remesh data indicating the motion information at the granularity of the segment based on the result of the segment search process and the motion information associated with each lattice point, and is machine-learned using the approximated surface data that has been subjected to the geometric modulation process and the approximated surface data obtained by the lattice point mesh segment matrix conversion unit as training input data, and includes a remesh approximated surface data generating unit that obtains the approximated surface data of the remesh data in an intermediate layer of the fourth layered autoencoder using the approximated surface data that has been subjected to the geometric modulation process, the approximated surface data obtained by the lattice point mesh segment matrix conversion unit, and the motion information associated with each lattice point as input data. The signal processing device according to (3) above. (5) a fifth layered autoencoder that uses as training data the motion information for each pixel position obtained by a pixel motion information calculation unit that calculates motion information for each pixel position within the image frame of the output image from the remesh data obtained by the remesh data generation unit, and that is machine-learned using the approximated surface data obtained by the remesh approximated surface data generation unit and coordinate data specifying the pixel position as learning input data; and a motion information decoding unit that receives as input the approximated surface data obtained by the remesh approximated surface data generation unit and the coordinate data specifying the pixel position, and outputs the motion information at specified coordinates within the image frame of the output image. The signal processing device according to (4) above. (6) the second layered autoencoder included in the geometric modulation processing unit learns an algorithm for geometric modulation of the approximation surface data for each different parameter setting for the geometric modulation processing; The geometric modulation processing unit switches the algorithm in response to the parameter setting. A signal processing device according to any one of (1) to (5). (7) For a first layered autoencoder that has been pre-trained using coordinate data for each element of an object consisting of a plurality of elements as learning input data, the coordinate data for each element is used as input data to obtain approximate surface data that approximates the object in an intermediate layer of the first layered autoencoder, The approximated surface data is used as learning input data, and a result of performing a geometric modulation process on the object by coordinate transformation for each element is used as training data to perform the geometric modulation process on the approximated surface data by a second stacked autoencoder that has been machine-trained. Signal processing methods. [Explanation of symbols]

[0110] 1. Signal Processing Device 2 IMU Sensor 3. Image Sensor 6 Quaternion Calculation Section 7 Stabilization processing section 11 Reference coordinate calculation section 12 Buffer control section 13 Buffer Memory 14 Memory control section 15 Cache Memory 16 Interpolation Filters CR Reference Coordinates L1 Virtual Line Pr Reference pixel Ar reference area 21 Grid point mesh generation and shaping section 22 Segment matrix generator 23 Segment Search Unit 24 Remesh data generation section 25 Each pixel coordinate interpolation section 31 Grid Point Mesh Generator 32 Lens Distortion Corrector 33 Projector 34 Rotating Device 35 Free curvature perspective projector 36 Scan Controller 37 Clipper 38 Each grid point reference coordinate calculator 32b, 32a Lens distortion correction learning machine 33b, 33a Projection learner 34b, 34a Rotational Learner 35b, 35a Free curvature perspective projection learning device 36b, 36a Scanning control learning device 37b, 37a Clip Learner 26b, 26a Lattice point mesh segment matrix transformation learner 27b, 27a Remesh learner 28b,28a Remesh extended decoding learner 39 Lattice point mesh approximation curved surface generation section

Claims

1. a first layered autoencoder that is pre-trained using coordinate data for each element of an object consisting of a plurality of elements as learning input data, and an approximation surface generating unit that obtains approximated surface data by approximating the object to a middle layer of the first layered autoencoder using the coordinate data for each element as input data; a second layered autoencoder that is machine-trained using the approximated surface data as learning input data and a result of performing a geometric modulation process on the object by coordinate transformation for each element as training data, and a geometric modulation processing unit that performs the geometric modulation process on the approximated surface data using the second layered autoencoder. Signal processing device.

2. performing a stabilization process for an image captured by an imaging device using a lattice point mesh having a plurality of lattice points arranged in a horizontal direction and a plurality of lattice points arranged in a vertical direction in an output coordinate system of the captured image, the stabilization process being performed using the lattice point mesh in which motion information of the imaging device is associated with each of the lattice points; The object is the lattice point mesh. The signal processing device according to claim 1 .

3. a third layered autoencoder that uses as training data a result of a segment search process that identifies in which of the squares in the lattice point mesh that has been subjected to the geometric modulation process each segment in a segment matrix formed by dividing an image frame of an output image by the stabilization process is located, and that performs machine learning using the approximated curved surface data that has been subjected to the geometric modulation process as learning input data; and a lattice point mesh segment matrix conversion unit that obtains, in an intermediate layer of the third layered autoencoder, approximated curved surface data that indicates a correspondence between each of the segments and the squares in the lattice point mesh, using the approximated curved surface data that has been subjected to the geometric modulation process as input data. The signal processing device according to claim 2 .

4. The fourth layered autoencoder includes a remesh data generating unit that generates remesh data indicating the motion information at the granularity of the segment based on the result of the segment search process and the motion information associated with each lattice point, and is machine-learned using the approximated surface data that has been subjected to the geometric modulation process and the approximated surface data obtained by the lattice point mesh segment matrix conversion unit as training input data, and includes a remesh approximated surface data generating unit that obtains the approximated surface data of the remesh data in an intermediate layer of the fourth layered autoencoder using the approximated surface data that has been subjected to the geometric modulation process, the approximated surface data obtained by the lattice point mesh segment matrix conversion unit, and the motion information associated with each lattice point as input data. The signal processing device according to claim 3 .

5. a fifth layered autoencoder that uses as training data the motion information for each pixel position obtained by a pixel motion information calculation unit that calculates motion information for each pixel position within the image frame of the output image from the remesh data obtained by the remesh data generation unit, and that is machine-learned using the approximated surface data obtained by the remesh approximated surface data generation unit and coordinate data specifying the pixel position as learning input data; and a motion information decoding unit that receives as input the approximated surface data obtained by the remesh approximated surface data generation unit and the coordinate data specifying the pixel position, and outputs the motion information at specified coordinates within the image frame of the output image. The signal processing device according to claim 4.

6. the second layered autoencoder included in the geometric modulation processing unit learns an algorithm for geometric modulation of the approximation surface data for each different parameter setting for the geometric modulation processing; The geometric modulation processing unit switches the algorithm in response to the parameter setting. The signal processing device according to claim 1 .

7. For a first layered autoencoder that has been pre-trained using coordinate data for each element of an object consisting of a plurality of elements as learning input data, the coordinate data for each element is used as input data to obtain approximate surface data that approximates the object in an intermediate layer of the first layered autoencoder, The approximated surface data is used as learning input data, and a result of performing a geometric modulation process on the object by coordinate transformation for each element is used as training data to perform the geometric modulation process on the approximated surface data by a second stacked autoencoder that has been machine-trained. Signal processing methods.

Images