Image processing device and program

Abstract

An image processing device according to an aspect of the present invention comprises an image acquisition unit, a control point recognition unit, a region dividing unit, a displacement amount storage unit, a displacement destination calculation unit, an interpolation unit, and an image deformation unit. The image acquisition unit acquires image data containing a target region comprising a plurality of pixels. The control point recognition unit sets control points in the target region. The region dividing unit divides the target region into a plurality of sub-regions. The displacement amount storage unit stores a displacement amount set for each control point. The displacement destination calculation unit sets at least one target pixel for each sub-region and calculates a displacement amount of the target pixel on the basis of the positions and displacement amounts of the control points. The interpolation unit calculates displacement amounts of pixels other than the target pixel by interpolation based on the positions and displacement amounts of the control points and the position and displacement amount of the target pixel. The image deformation unit deforms the target region on the basis of the displacement amount of the target pixel and the displacement amounts of pixels other than the target pixel.

Description

Image processing device and program

[0001] One aspect of the present invention relates to an image processing device and a program.

[0002] For example, facial images of people are transformed to generate facial images of various emotions. There are known techniques that can obtain natural facial expressions through relatively simple image processing, even if they are not as large-scale as so-called generative AI (artificial intelligence). Rigid MLS, disclosed in Non-Patent Document 1, is one such technique.

[0003] Scott Schaefer, et al., “Image Deformation Using Moving Least Squares”, SIGGRAPH, 2006.

[0004] Existing technologies calculate the amount of pixel movement between control points based on the amount of movement of several control points, and then deform the image based on the results. However, because the exact amount of movement is calculated using the same algorithm for all pixels within the calculation range, the amount of calculation increases depending on the number of pixels. In particular, with high-resolution images such as 4K and 8K, the amount of calculation increases dramatically along with the number of pixels to be calculated, resulting in long processing times.

[0005] Therefore, the present invention aims to provide a technique that enables images to be transformed at high speed.

[0006] An image processing device according to one aspect of the present invention includes an image acquisition unit, a control point recognition unit, a region division unit, a movement amount storage unit, a movement destination calculation unit, an interpolation unit, and an image deformation unit. The image acquisition unit acquires image data including a target region consisting of a plurality of pixels. The control point recognition unit sets control points in the target region. The region division unit divides the target region into a plurality of small regions. The movement amount storage unit stores the movement amount set for each control point. The movement destination calculation unit sets at least one target pixel for each small region and calculates the movement amount of the target pixel from the position and movement amount of the control point. The interpolation unit calculates the movement amount of pixels other than the target pixel by interpolating based on the position and movement amount of the control point and the position and movement amount of the target pixel. The image deformation unit deforms the target region from the movement amount of the target pixel and the movement amount of pixels other than the target pixel.

[0007] According to one aspect of the present invention, it becomes possible to transform an image at high speed.

[0008] FIG. 1 is a functional block diagram showing an example of an image processing apparatus according to an embodiment. FIG. 2 is a diagram showing an example of an original image to be processed. FIG. 3 is a diagram showing an example of an image after processing. FIG. 4 is a diagram showing an example of set control points. FIG. 5 is a diagram showing an example of coordinates of control points. FIG. 6 is a diagram showing an example of area division by rectangles. FIG. 7 is a diagram showing an example of area division by Voronoi division. FIG. 8 is a diagram showing an example of movement amounts of control points. FIG. 9 is a diagram for explaining interpolation calculations for areas divided by rectangles. FIG. 10 is a diagram for explaining interpolation calculations in Voronoi division. FIG. 11 is a block diagram showing an example of the hardware configuration of an image processing apparatus 10.

[0009] 1 is a functional block diagram showing an example of an image processing apparatus according to an embodiment. The image processing apparatus 10 as a computer includes an image acquisition unit 21, a control point recognition unit 22, an area division unit 23, a movement amount storage unit 31, a movement destination calculation unit 24, an interpolation unit 25, and an image deformation unit 26.

[0010] The image acquisition unit 21 acquires image data of an image to be processed from an image captured by a web camera or an existing database. As shown in FIG. 2 , the image includes a target region to be processed, such as a person's face. The image and the target region, which is a part of the image, include multiple pixels. The acquired image data is stored in a storage device such as a memory or HDD (Hard Disk Drive). In this embodiment, it is considered to deform the target region to generate a deformed image (smile) as shown in FIG. 3 from the facial expression (smile) in FIG. 2 .

[0011] In Fig. 1, the control point recognition unit 22 receives an image as input, recognizes the positions of control points, and outputs them. That is, the control point recognition unit 22 sets control points in the target area of ​​the image. Fig. 4 shows an example of the set control points. For example, facial features can be extracted from facial image data using an existing library, and control points can be set based on the results.

[0012] 5 is a diagram showing an example of the coordinates of control points. Each control point is identified by an identifier (ID) and is associated with x-coordinate and y-coordinate values. Transforming an image corresponds to transforming the positions of the control points and their surrounding pixels.

[0013] The region dividing unit 23 receives the positions of the control points and outputs the region division results. That is, the region dividing unit 23 divides the target region into multiple small regions. For example, the region dividing unit 23 may divide the target region into rectangular regions as shown in Fig. 6, or may use Voronoi division as shown in Fig. 7.

[0014] 1, the movement amount storage unit 31 stores the movement amount for each control point. The movement amount for each control point is calculated and set in advance for each type of image transformation, for example. The movement amount storage unit 31 stores the movement amount for each control point in a storage device such as a memory or HDD. FIG. 8 is a diagram showing an example of the movement amount of a control point. The movement amount in image transformation is calculated for each control point.

[0015] The destination calculation unit 24 calculates and outputs the movement amount of the main pixel (pixel) in each small region based on the positions of the control points, the movement amount for each control point, and the settings of the divided regions. That is, the destination calculation unit 24 sets at least one target pixel for each small region and calculates the movement amount of the target pixel from the positions and movement amounts of the control points. Here, the movement amount of the main pixel (target pixel) can be calculated by applying Rigid MLS described in Non-Patent Document 1.

[0016] The interpolation unit 25 additionally calculates and outputs the movement amounts of the remaining pixels based on the movement amounts of the main pixels. That is, the interpolation unit 25 calculates the movement amounts of pixels other than the target pixel by interpolating based on the positions and movement amounts of the control points and the position and movement amount of the target pixel. In this way, the positions of all pixels after deformation are calculated by interpolation. For example, linear interpolation or spline interpolation can be used as the interpolation method. In short, any algorithm can be used as long as it requires less calculation than Rigid MLS.

[0017] The image transformation unit 26 processes the image data of the image before transformation, and transforms and outputs the image based on the movement amounts of all pixels. That is, the image transformation unit 26 transforms the target area by image processing based on the movement amounts of the target pixel and the movement amounts of pixels other than the target pixel (the movement amounts of all pixels).

[0018] 9 is a diagram illustrating an interpolation operation for an area divided by a rectangle. The example illustrates a case where an image is divided into rectangles of N×N pixels with a width N=40, and linear interpolation is performed using the intersections of the line segments dividing the areas as major pixels.

[0019] First, calculate the displacement (Δx, Δy) for pixels p1 to p4, which are the intersections of the line segments, using Rigid MLS. Next, calculate the displacement (Δx) for a region within the pixel p4, for example, pixel p5 (coordinates (10, 30)). ｐ５ , Δy ｐ５ ) can be obtained as (+8, +16) by the linear interpolation formula of formula (1).

[0020] 10 is a diagram for explaining the interpolation calculation in Voronoi division. A case will be described in which an image is subjected to Voronoi division and linear interpolation is applied with the midpoints of the line segments dividing the regions as the main pixels.

[0021] First, calculate the displacement (Δx, Δy) for pixels p1 to p3, which are the midpoints of the line segments, using Rigid MLS. Next, calculate the displacement (Δx) for a region within that, for example, pixel p4 (coordinates (30, 30)). ｐ４ , Δy ｐ４ ) can be calculated as (+12, +16) using the following linear interpolation formula:

[0022] That is, coefficient A １ , A ２ , B １ , B ２ , C １ , C ２ Then, the three simultaneous equations of equation (3) are set up for p1 to p3, and the coefficient A is calculated as in equation (4). １ , A ２ , B １ , B ２ , C １ , C ２ By obtaining (Δx ｐ４ , Δy ｐ４ ) can be obtained.

[0023] As described above, according to the embodiment, the range including the control points is divided into multiple small regions, and the destination of the key pixel in each region is calculated using, for example, Rigid MLS. Meanwhile, the amount of movement of pixels other than the key pixel is calculated simply by interpolating the destinations of the surrounding pixels. The region division method may be any method, such as rectangular division or Voronoi division. Furthermore, the key pixel for each small region can be defined in any way, such as the midpoint of the region, or the midpoint or intersection of the line segments dividing the region.

[0024] Furthermore, the interpolation algorithm may be any algorithm, such as linear interpolation or spline interpolation. In other words, any algorithm that is different from the algorithm used to calculate the destination of the primary pixel and requires less computation can be applied. As a result, according to the embodiment, the amount of computation required for image transformation can be reduced, thereby enabling images to be transformed at high speed.

[0025] The functions of the image processing device 10 can be realized by installing a program on a computer. For example, the functions of the image processing device 10 can be implemented by causing a computer to execute a program provided as package software or online software.

[0026] Fig. 11 is a block diagram showing an example of the hardware configuration of the image processing device 10. As shown in Fig. 15, the image processing device 10 includes a CPU (Central Processing Unit) 20A, a bridge circuit 102, a memory 30, a GPU (Graphics Processing Unit) 20B connected to a display 104, a BIOS-ROM 107, a storage 109, a USB connector 110, and an input unit 112.

[0027] The storage 109 is a non-volatile storage medium (block device), such as a hard disk drive (HDD) or a solid state drive (SSD). The storage 109 stores basic programs such as an operating system (OS) 35 and device drivers, as well as a program 34 for implementing the functions of the image processing device 10.

[0028] The memory 30 includes a ROM (Read Only Memory) and a RAM (Random Access Memory). The CPU 20A and GPU 20B are arithmetic elements related to the processor 20. The CPU 20A mainly controls the image processing device 10. The GPU 20B mainly performs calculations related to image processing (such as multiply-accumulate operations) at high speed. The CPU 20A executes a BIOS (basic input / output system) stored in the BIOS-ROM 107. The CPU 20A also loads a program 34 from the storage 109 into the memory 30 and executes it. The same is true for the GPU 20B.

[0029] The bridge circuit 102 relays data transmission between the CPU 20A and the GPU 20B and each component. For example, the bridge circuit 102 is interposed between the CPU 20A and the GPU 20B and hardware devices connected to a PCI (Peripheral Component Interconnect) bus or a PCIe bus (not shown), relaying communication between them.

[0030] The USB connector 110 connects a USB device or the like. For example, the program 34 may be installed in the image processing device 10 via the USB device. The input unit 112 connects the image processing device 10 to a communication network such as a LAN (Local Area Network) or a WAN (Wide Area Network).

[0031] The program 34 and various data may be stored in a removable storage medium other than the storage 109, and may be read by the CPU 20A from a disk drive or the like. Alternatively, the program 34 and various data may be stored in another computer connected via a communication network, and may be read by the CPU 20A via the input unit 112.

[0032] It should be noted that the present invention is not limited to the above-described embodiment. For example, the image processing device 10 may be a notebook, desktop, or server computer. Alternatively, it may be a portable terminal such as a smartphone or tablet. Furthermore, it is also possible to make computer resources virtualized on the cloud function as the image processing device 10.

[0033] That is, this invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0034] 10...Image processing device 20...Processor 20A...CPU 21...Image acquisition unit 22...Control point recognition unit 23...Area division unit 24...Movement destination calculation unit 25...Interpolation unit 26...Image deformation unit 30...Memory 31...Movement amount storage unit 34...Program 102...Bridge circuit 104...Display 109...Storage 110...USB connector 112...Input unit.

Claims

1. An image processing device comprising: an image acquisition unit that acquires image data of an image including a target region consisting of a plurality of pixels; a control point recognition unit that sets control points in the target region; a region division unit that divides the target region into a plurality of small regions; a movement amount storage unit that stores a movement amount set for each of the control points; a movement destination calculation unit that sets at least one target pixel for each small region and calculates the movement amount of the target pixel from the position and movement amount of the control point; an interpolation unit that calculates the movement amounts of pixels other than the target pixel by interpolating based on the position and movement amount of the control point and the position and movement amount of the target pixel; and an image transformation unit that transforms the target region from the movement amount of the target pixel and the movement amount of pixels other than the target pixel.

2. The image processing device according to claim 1, wherein the interpolation unit calculates the amount of movement of pixels other than the target pixel by linearly interpolating the position and amount of movement of the control point and the position and amount of movement of the target pixel.

3. The image processing device according to claim 1, wherein the interpolation unit calculates the movement amounts of pixels other than the target pixel by performing spline interpolation between the positions and movement amounts of the control points and the positions and movement amounts of the target pixel.

4. A program including instructions for causing a computer to function as the image processing device according to any one of claims 1 to 3.

Images