Aspect ratio conversion processing device and program

Abstract

To convert an aspect ratio of a video content so as to maintain creator's intentions put into the video content.SOLUTION: An object detection unit 10 of an aspect ratio conversion processing device 1 trims a video I having an aspect ratio AS at a predetermined position from an input video, detects an object, extracts a rectangular area K, and generates additional information including the aspect ratio AS of the video I, etc. A creation intention estimation unit 11 estimates a creator's intention for the video I based on the additional information, and generates intention information including a size and a position of the object in the video I. An aspect ratio conversion unit 12 trims a video I1' from the input video based on the intention information such that the size and the position of the object in the video I are similar to a size and a position of the object in the video I1, and enlarges or reduces the video I1' to the video I1.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an aspect ratio conversion processing device and a program for converting the aspect ratio of a video material or video content into a different aspect ratio. [Background technology]

[0002] Video content, such as broadcast program content, is produced as a rectangular image with a certain aspect ratio. This aspect ratio is called the width:height ratio and is usually expressed as the width:height ratio.

[0003] Traditionally, the most commonly used aspect ratio in video production is the 16:9 ratio defined by the broadcasting standard. This is because many display devices that have come into widespread use since the advent of digital broadcasting were manufactured with a 16:9 ratio.

[0004] However, there are cases where images with ratios other than 16:9 are used. For example, in movies, which are a different genre of video content, images with a higher horizontal ratio than 16:9 are conventionally used. In broadcast program content, images with various ratios are used, such as movies with a higher horizontal ratio due to the creator's intention, and images with a lower horizontal ratio of 4:3 used in past analog broadcasts.

[0005] When analog broadcasting was replaced by digital broadcasting, the aspect ratio of the majority of display devices owned by viewers was 4:3. For this reason, broadcast program content with an aspect ratio of 16:9 and movie content with a wider aspect ratio than 16:9 had to have the aspect ratio converted on the display device side, losing the creator's intention.

[0006] To solve this problem, aspect ratio retargeting or reframing techniques have been disclosed (see, for example, Patent Documents 1 to 4). The technique of Patent Document 1 adds a background created by scaling the input image itself to the area where the image aspect ratio and the display aspect ratio do not match when converting the image to be displayed.

[0007] The technology of Patent Document 2 involves an encoding device that performs letterbox or pillarbox display to prevent undesirable conversion when content is viewed by users on various devices, thereby providing content that retains the creator's intention.

[0008] The technology in Patent Document 3 relates to reframing and encoding of video signals, and involves extracting an area called a feature map and expanding that area vertically or horizontally, thereby enabling processing by existing encoding devices.

[0009] The technology of Patent Document 4 provides an optimal viewing experience for each of various devices in a viewing system in response to user interaction.

[0010] In this way, the video aspect ratio and the display aspect ratio do not match, and as mentioned above, content created with one video aspect ratio is processed to be converted to a different display aspect ratio. The background to this is the spread of a new viewing style in which content is viewed on mobile devices.

[0011] Traditionally, the aspect ratio of most display devices was 16:9 or close to 16:9, but in recent years, there has been an increase in display devices with aspect ratios of 3:4, such as tablets, and closer to 9:16, such as smartphones.

[0012] When using such a mobile device to watch video content with an aspect ratio of 16:9, if the mobile device is held vertically, there is an issue that black margins appear above and below the image due to the difference between the image aspect ratio of the video content and the display aspect ratio of the display device, making the image appear smaller in comparison.

[0013] As a method for solving the problems caused by the difference between the video aspect ratio and the display aspect ratio, a process for converting the aspect ratio when viewing using a mobile terminal is known. Examples of aspect ratio conversion technologies installed in editing machines and the like include "AutoFlip" and "AutoReframe" (see, for example, Non-Patent Documents 1 and 2).

[0014] These techniques mainly detect an object in a video and trim the video to a rectangular area surrounding the object. If the object is a person, the facial area is extracted and the video is trimmed so that the facial area is displayed large. This makes it possible to display only the specific object large.

[0015] On the other hand, much of the video content is produced by the creator adding embellishments to make it more appealing, and then the cameraman uses his or her camerawork to shoot each cut in a way that recreates that direction.

[0016] Examples of camera work include long shots and close-ups. Long shots can convey the location and time of an object from the image, while close-ups can convey the details of an object. Shooting that is intermediate between long shots and close-ups can simultaneously convey the details of an object, as well as the location and time of the object.

[0017] Examples of cut changes in video content include a change from a long shot to a shot between a long shot and a close-up, or a change to a close-up shot. If the aspect ratio of such video content is converted using the above-mentioned technology, there is a drawback in that all cuts will end up being close-up shots.

[0018] Also, when shooting a person, the degree of close-up shots varies greatly depending on the purpose, such as a bust shot, a face-up shot, etc. If the above-mentioned technology is used for an image such as a face-up shot, the aspect ratio may not be converted appropriately.

[0019] In this case, if the aspect ratio is converted to make only a specific object in a video appear larger, the creator's intention behind the video content will be lost. [Prior art documents] [Patent documents]

[0020] [Patent Document 1] Patent No. 4864038 [Patent Document 2] International Publication No. 2013 / 057938 [Patent Document 3] Patent No. 5974089 [Patent Document 4] Patent No. 6891667 [Non-patent literature]

[0021] [Non-Patent Document 1] “AutoFlip: An Open Source Framework for Intelligent Video Reframing”, [online], Google, [searched July 26, 2023], Internet<URL:https: / / ai.googleblog.com / 2020 / 02 / autoflip-open-source-framework-for.html> [Non-Patent Document 2] “Automatically reframe videos for social media channels”, [online], Adobe, [Retrieved July 26, 2023], Internet<URL:https: / / helpx.adobe.com / jp / premiere-pro / using / auto-reframe.html> Summary of the Invention [Problem to be solved by the invention]

[0022] In this way, when the aspect ratio of the video content created by the creator (image aspect ratio) differs from the aspect ratio of the display device used by the viewer (display aspect ratio), if the creator converts the image aspect ratio to the display aspect ratio, the creator's intentions conveyed in the video content will be lost.

[0023] Therefore, the present invention has been made to solve the above-mentioned problems, and its object is to provide an aspect ratio conversion processing device and program capable of converting the aspect ratio of video content so as to preserve the creator's intentions conveyed in the video content. [Means for solving the problem]

[0024] In order to solve the above problem, an aspect ratio conversion processing device according to claim 1 converts an aspect ratio AS of an image I included in an input image into an aspect ratio AS1 different from the aspect ratio AS, and generates an image I1 with the aspect ratio AS1, the aspect ratio conversion processing device further comprising an object detection unit that trims the image I from the input image, detects an object from the image I, and determines the position of the object within the image I, and for the object detected by the object detection unit, determines a size of the object within the image I, and compares the size of the object within the image I and the position of the object within the image I determined by the object detection unit. the aspect ratio conversion unit generating the image I1 by trimming an image I1' from the input image and enlarging or reducing the image I1' to the image I1 so that a difference between the size and position of the object in the image I contained in the intention information estimated by the production intention estimation unit and the size and position of the corresponding object in the image I1 are each equal to or smaller than a predetermined threshold value, using an object included in the image I1 that corresponds to the object included in the image I as a corresponding object, and

[0025] The aspect ratio conversion processing device of claim 2 is the aspect ratio conversion processing device of claim 1, wherein the object detection unit detects the object from the video I to extract a rectangular area K including the object, and calculates the number of horizontal pixels w and the number of vertical pixels h of the rectangular area K and the relative position p of the rectangular area K within the video I, and the creative intention estimation unit calculates an area ratio R of the rectangular area K within the video I based on the number of horizontal pixels W and the number of vertical pixels H of the video I and the number of horizontal pixels w and the number of vertical pixels h of the rectangular area K calculated by the object detection unit, and calculates the area ratio R of the rectangular area K within the video I and the relative position p of the rectangular area K within the video I calculated by the object detection unit. p is estimated as the intention information, and the aspect ratio conversion unit generates the image I1 by trimming the image I1' from the input image and enlarging or reducing the image I1' to the image I1 so that a difference between an area ratio R of the rectangular area K in the image I contained in the intention information and an area ratio R1 of the rectangular area K1 in the image I1 is equal to or smaller than a predetermined threshold, and a difference between a relative position p of the rectangular area K in the image I contained in the intention information and a relative position p1 of the rectangular area K1 in the image I1 is equal to or smaller than a predetermined threshold.

[0026] The aspect ratio conversion processing device of claim 3 is the aspect ratio conversion processing device of claim 2, wherein the aspect ratio conversion unit sets a frame A corresponding to the image I1' such that a difference between an area ratio R of the rectangular area K in the image I and an area ratio R1' of the rectangular area K in the frame A is equal to or smaller than a predetermined threshold, and a difference between a relative position p of the rectangular area K in the image I and a relative position p1' of the rectangular area K in the frame A is equal to or smaller than a predetermined threshold, trims the image of the frame A from the input image as the image I1', and enlarges or reduces the image I1' to the image I1, thereby generating the image I1.

[0027] Moreover, the aspect ratio conversion processing device of claim 4 is the aspect ratio conversion processing device of claim 2 or 3, characterized in that the relative position p of the rectangular area K in the image I determined by the object detection unit is set to the center position, upper left position, lower left position, upper right position or lower right position of the rectangular area K, or, when the object is a person, the center position of the head of the person.

[0028] Furthermore, a program according to claim 5 causes a computer to function as the aspect ratio conversion processing device according to claim 1. Effect of the Invention

[0029] As described above, according to the present invention, the aspect ratio of video content can be converted so as to preserve the intent of the creator of the video content. [Brief description of the drawings]

[0030] [Figure 1] 1 is a block diagram showing an example of the configuration of an aspect ratio conversion processing device according to an embodiment of the present invention. [Diagram 2] 2A to 2C are diagrams illustrating examples of an input video, a video I with an aspect ratio AS, etc. FIG. [Diagram 3] 1A to 1C are diagrams illustrating examples of images I1 and the like having a target aspect ratio AS1. [Figure 4] 10 is a flowchart illustrating an example of processing by an aspect ratio conversion processing device. [Diagram 5] 13 is a flowchart showing an example of processing by a creative intention estimation unit (step S407). [Figure 6] 13 is a flowchart showing an example of processing (steps S408 and S409) of the aspect ratio conversion unit. [Figure 7] FIG. 1(1) is a diagram illustrating an example of an image I that is trimmed from an input image. FIG. 1(2) is a diagram illustrating an example of an image I1' of a frame A that is trimmed from an input image. [Figure 8]Fig. 3(3) is a diagram illustrating an example of an image I1' of the trimmed frame A. Fig. 3(4) is a diagram illustrating an example of an image I1 after enlargement or reduction. [Figure 9] 1A and 1B are diagrams showing examples of video I captured in a loose shot of a person, and videos I1 and I2 after aspect ratio conversion. [Figure 10] 1A and 1B are diagrams showing examples of an image I of two people captured in a loose shot, and images I1 and I2 after aspect ratio conversion. [Figure 11] 1A and 1B are diagrams showing examples of an image I captured in full figure of a person, and images I1 and I2 after aspect ratio conversion. [Figure 12] 1A and 1B are diagrams showing examples of an image I of two people captured in full figure, and images I1 and I2 after aspect ratio conversion. [Figure 13] 1 is a diagram showing an example of video I in which a person is shot in a bust shot, and videos I1 and I2 after aspect ratio conversion. [Figure 14] 1A and 1B are diagrams showing examples of video I, which is a knee shot of two people, and videos I1 and I2 after aspect ratio conversion. [Figure 15] FIG. 13 is a diagram showing an example in which a tiling technique is used when converting the aspect ratio of video I in which two people are shot in knee shot. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. [Configuration of aspect ratio conversion processing device] 1 is a block diagram showing an example of the configuration of an aspect ratio conversion processing device according to an embodiment of the present invention. The aspect ratio conversion processing device 1 includes an object detection unit 10, a creative intention estimation unit 11, and an aspect ratio conversion unit 12.

[0032] The object detection unit 10 inputs an image (input image), trims an image I with an aspect ratio (image aspect ratio) AS, horizontal pixel count W, and vertical pixel count H at a predetermined position from the input image, and outputs the image I with aspect ratio AS. In the input image, a partial area (image I) of wide shooting information is framed with a specific aspect ratio AS, and the original wide shooting information is retained as a video signal. The object detection unit 10 then detects an object from the image I, and extracts a rectangular area K including the object.

[0033] 2 is a diagram illustrating an example of an input video, video I with aspect ratio AS, etc. In this input video, video I with a predetermined aspect ratio AS, horizontal pixel count W, and vertical pixel count H is framed at a predetermined position. An object detection unit 10 trims the video I from the input video, incorporating the creator's intention, detects an object (a person in the example of FIG. 2) from video I, and extracts a rectangular region K with horizontal pixel count w and vertical pixel count h that includes the object.

[0034] For example, video I is produced by trimming a predetermined position (central region) of video I having an aspect ratio AS of 16:9 and a resolution of 2K from an input video captured by a camera having an aspect ratio of 16:9 and a resolution of 8K. The predetermined position of video I to be trimmed is set in advance or by the cameraman.

[0035] Returning to Fig. 1, object detection unit 10 generates additional information including the aspect ratio AS of video I, the number of horizontal pixels W and vertical pixels H of video I, the number of horizontal pixels w and vertical pixels h of rectangular area K, and the absolute position P(X,Y) and relative position p(x,y) of rectangular area K. Object detection unit 10 outputs the input video and the additional information to creative intention estimation unit 11. Details of the processing by object detection unit 10 will be described later.

[0036] 2, point P indicates, for example, the central position of rectangular area K or an object contained in rectangular area K. The absolute position P(X,Y) of rectangular area K indicates the coordinates of point P in plane coordinates of X and Y axes representing the position on the input video. Also, the relative position p(x,y) of rectangular area K indicates the coordinates of point P in plane coordinates of x and y axes representing the position on the video I.

[0037] In addition, the absolute position P(X,Y) and relative position p(x,y) of point P may indicate the coordinates of the top left, bottom left, top right or bottom right position of the rectangular area K, or, if the object contained in the rectangular area K is a person, may indicate the coordinates of the center position of the person's head.

[0038] Returning to Fig. 1, creative intention estimation unit 11 receives input video and additional information from object detection unit 10, and generates intention information indicating the intention of the creator contained in video I including an object by estimating the intention of the creator regarding video I based on the additional information. Then, creative intention estimation unit 11 outputs the input video, additional information, and intention information to aspect ratio conversion unit 12. Details of the processing by creative intention estimation unit 11 will be described later.

[0039] The intention information is information reflecting the size and position of an object in the image I, indicating how large and where the object is to be displayed within the entire image I. The intention information includes, for example, the area ratio R of the rectangular area K in the image I, and the position of the rectangular area K in the image I (relative position p(x, y)).

[0040] The relative position p(x, y) of the rectangular area K in the image I represents the composition of the image I including the object. There are various compositions, such as the Japanese flag composition, the rule of thirds composition, and the golden ratio composition, and each of them reflects the position of the object included in the image I captured by the photographer. For example, in the case of a single object, when it is located in the center of the image I, it is the Japanese flag composition, and when it is close to one of the short sides of the rectangular image I, it is the rule of thirds composition. In other words, the position of the object included in the intention information is information about the composition, and the aspect ratio is converted by the aspect ratio conversion unit 12, which will be described later, so that the composition of the image I is maintained.

[0041] Aspect ratio conversion unit 12 inputs the input video, additional information, and intention information from creative intention estimation unit 11, as well as a target aspect ratio (display aspect ratio) AS1, horizontal pixel count W1, and vertical pixel count H1 from an external source. The user can specify a target aspect ratio AS1, etc., different from the aspect ratio AS of video I, by external input.

[0042] The aspect ratio conversion unit 12 extracts an image I at a predetermined position from the input image based on the additional information. Then, the aspect ratio conversion unit 12 converts the aspect ratio AS into a target aspect ratio AS1 by trimming an image I1' from the input image and enlarging or reducing the image I1' to the image I1 so as to maintain the intention of the creator indicated by the intention information, and generates an image I1 with the target aspect ratio AS1, horizontal pixel count W1, and vertical pixel count H1. Details of the image I1' will be described later.

[0043] In this case, when converting the aspect ratio, the aspect ratio conversion unit 12 trims the image I1' from the input image and enlarges or reduces the image I1' to the image I1 so that the size and position of the rectangular area K in the image I approximates the size and position of the rectangular area K1 in the converted image I1. Here, approximation means that the difference between the size and position of the rectangular area K in the image I and the size and position of the rectangular area K1 in the image I1 is equal to or smaller than a predetermined threshold.

[0044] In other words, the aspect ratio conversion unit 12 converts the aspect ratio of an image I having aspect ratio AS by reproducing what the producer intended in a target aspect ratio AS1 that is different from the aspect ratio AS and generating an image I1 having the target aspect ratio AS1.

[0045] The aspect ratio conversion unit 12 outputs the trimmed and enlarged or reduced image I1 as an image I1 with a target aspect ratio AS1. The process of the aspect ratio conversion unit 12 will be described in detail later.

[0046] 3 is a diagram illustrating an example of an image I1 with a target aspect ratio AS1. This image I1 includes a rectangular area K1 with a horizontal pixel count w1 and a vertical pixel count h1. The aspect ratio conversion unit 12 generates an image I1 with a target aspect ratio AS1, a horizontal pixel count W1, and a vertical pixel count H1 so as to maintain the intention (size and position of an object (person) in the image I) that the creator put into the image I shown in FIG. 2.

[0047] [Processing by aspect ratio conversion processing device] Next, a description will be given of the processing of the aspect ratio conversion processing device 1 shown in Fig. 1. Fig. 4 is a flowchart showing an example of processing of the aspect ratio conversion processing device 1. The following description will be given with reference to the configuration diagram shown in Fig. 1 and the flowchart shown in Fig. 4.

[0048] (Object detection unit 10) The object detection unit 10 receives an input video (step S401), and trims a video I having an aspect ratio AS, horizontal pixel count W, and vertical pixel count H at a predetermined position from a frame of the input video (step S402). The object detection unit 10 then detects an object from the video I, and extracts a rectangular region K including the object (step S403).

[0049] The detection of an object may be performed using the technologies described in the above-mentioned Non-Patent Documents 1 and 2. In addition, existing technologies may be used, such as a technology for detecting a gaze area based on the characteristic that the viewer's gaze tends to be focused on a fine-grained area (an area where high-frequency components are concentrated), and a technology for detecting an area of ​​high interest based on information such as brightness and color reproduction, which is called a saliency map. Machine learning and AI technologies may also be used.

[0050] Here, a rectangular area K including the object is extracted, but no trimming or the like is performed. For example, as shown on the left side of Figs. 9 to 14 described later, in each frame constituting video I, when the subject, which is the object, is a person, a rectangular area K including the person is extracted.

[0051] The object detection unit 10 determines whether the number of objects detected in step S403 is one, or two or more (step S404).

[0052] If the object detection unit 10 determines in step S404 that the number of objects is 1 (step S404:=1), the process proceeds to step S406.

[0053] On the other hand, if the object detection unit 10 determines in step S404 that the number of objects is two or more (step S404: ≧2), it extracts a new rectangular area K that contains two or more objects, replacing the rectangular area K extracted in step S403 (step S405), and proceeds to step S406.

[0054] Moving on from step S404 (=1) or step S405, object detection unit 10 obtains the number of horizontal pixels w and the number of vertical pixels h of rectangular area K, and the absolute position P(X,Y) and relative position p(x,y) of rectangular area K. Then, object detection unit 10 generates additional information including the aspect ratio AS of video I, the number of horizontal pixels W and the number of vertical pixels H of video I, the number of horizontal pixels w and the number of vertical pixels h of rectangular area K, and the absolute position P(X,Y) and relative position p(x,y) of rectangular area K (step S406).

[0055] (Production Intention Estimation Unit 11) The creative intention estimation unit 11 receives the input video and the additional information from the object detection unit 10, and generates intention information by estimating the intention of the creator regarding the video I based on the additional information (step S407).

[0056] 5 is a flowchart showing an example of processing (step S407) by the creative intention estimation unit 11. The creative intention estimation unit 11 receives an input video and additional information from the object detection unit 10 (step S501), and generates intention information by estimating intention information indicating the size and position of a rectangular area K in video I based on the additional information (step S502).

[0057] Specifically, the creative intention estimation unit 11 calculates an area ratio R = (w × h) / (W × H), which indicates the size of rectangular area K within image I, using the number of horizontal pixels W and the number of vertical pixels H of image I contained in the additional information, and the number of horizontal pixels w and the number of vertical pixels h of rectangular area K.

[0058] For example, if one wishes to mainly photograph an object through camera framing, the area ratio R approaches 1, whereas if one wishes to mainly photograph the spatial elements in which the object exists, the area ratio R approaches 0.

[0059] Furthermore, the creative intention estimation unit 11 treats the relative position p(x, y) of the rectangular area K in the video I included in the additional information as the position of the rectangular area K in the video I included in the intention information.

[0060] Then, the creative intention estimation unit 11 generates intention information including the area ratio R of the rectangular region K in the video I and the relative position p(x, y).

[0061] The creative intention estimation unit 11 outputs the input video, the additional information, and the intention information to the aspect ratio conversion unit 12 (step S503).

[0062] Furthermore, the production intention estimation unit 11 may generate intention information indicating the position of rectangular area K within image I by determining, on a rule-based basis, where the top and bottom of rectangular area K are located in terms of vertical pixels in image I, and where the sides of rectangular area K are located in terms of horizontal pixels in image I.

[0063] Generally, in most video content, the subject that the creator wants the viewer to see is rarely positioned at the top, bottom, left, or right edge of image I. When framing image I from the input video, the cameraman determines the position of the object in image I by referring to typical photographic compositions such as the Hinomaru composition, the rule of thirds, and the golden ratio composition. In other words, there is a high probability that the object (main subject) that the creator wants the viewer to see will be concentrated near the center of image I.

[0064] The camerawork makes full use of close-up and long-distance shots to change the size of the object in image I. In other words, by determining the size and position of rectangular area K in image I, it is possible to reproduce what the creator intended.

[0065] (Aspect ratio conversion section 12) Returning to Figures 1 and 4, aspect ratio conversion unit 12 inputs the input video, additional information, and intention information from creative intention estimation unit 11, as well as the target aspect ratio AS1, horizontal pixel count W1, and vertical pixel count H1 from the outside (step S408).

[0066] The aspect ratio conversion unit 12 extracts the image I at a predetermined position from the input image based on the additional information. Then, the aspect ratio conversion unit 12 converts the aspect ratio AS to the target aspect ratio AS1 by trimming the image I1' from the input image and enlarging or reducing the image I1' to the image I1 based on the intention information so as to maintain the intention of the creator, and generates the image I1 with the target aspect ratio AS1, the horizontal pixel count W1 and the vertical pixel count H1, and outputs this as the image I1 with the target aspect ratio AS1 (step S409).

[0067] Fig. 6 is a flow chart showing an example of processing (steps S408, S409) of the aspect ratio conversion unit 12. Fig. 7(1) is a diagram for explaining an example of an image I trimmed from an input image, and Fig. 7(2) is a diagram for explaining an example of an image I1' of a frame A trimmed from the input image. Fig. 8(3) is a diagram for explaining an example of an image I1' of a trimmed frame A, and Fig. 8(4) is a diagram for explaining an example of an image I1 after enlargement or reduction.

[0068] An example of processing by the aspect ratio conversion unit 12 shown in Figure 6, referring to Figures 7(1)(2) and 8(3)(4), involves trimming an image I1' from the input image and enlarging or reducing the image I1' to the image I1 so that the area ratio R of the rectangular area K in the image I is approximate to the area ratio R1 of the rectangular area K1 in the image I1 and the relative position p(x, y) of the rectangular area K in the image I is approximate to the relative position p1(x1, y1) of the rectangular area K1 in the image I1.

[0069] As mentioned above, the area ratio R of the rectangular region K in the image I is calculated as R = (w × h) / (W × H), and the area ratio R1 of the rectangular region K1 in the image I1 is calculated as R1 = (w1 × h1) / (W1 × H1).

[0070] Returning to Fig. 6, the aspect ratio conversion unit 12 inputs the input video, additional information, and intention information from the creative intention estimation unit 11 (step S601). As shown in Fig. 7(1), the input video includes an image I, and the image I at a predetermined position is extracted from the input video based on the additional information.

[0071] The aspect ratio conversion unit 12 receives a target aspect ratio AS1, a horizontal pixel count W1, and a vertical pixel count H1 from the outside (step S602).

[0072] 7(2), a frame A with a target aspect ratio AS1 is assumed within the input video so as to include a rectangular area K. The number of horizontal pixels of frame A is W1', the number of vertical pixels is H1', ​​and the area of ​​frame A is the same as the area of ​​video I (or the same as the area of ​​video I plus a predetermined value).

[0073] 7(1)(2), the aspect ratio conversion unit 12 sets a frame A of a target aspect ratio AS1 in the input video so that the area ratio R of the rectangular region K in the video I included in the intention information is similar to the area ratio R1' of the rectangular region K in the frame A, and the relative position p(x, y) of the rectangular region K in the video I included in the intention information is similar to the relative position p1'(x1', y1') of the rectangular region K in the frame A (step S603). Setting the frame A means setting the horizontal pixel count W1' and vertical pixel count H1' of the frame A, and the position of the frame A in the input video.

[0074] Here, the area ratio R of the rectangular area K in the image I is calculated as R = (w × h) / (W × H) as described above, and the area ratio R1' of the rectangular area K in the frame A is calculated as R1' = (w × h) / (W1' × H1').

[0075] In addition, when the area ratio R of the rectangular region K in the image I and the area ratio R1' of the rectangular region K in the frame A are approximated to each other, it means that the difference between the area ratio R and the area ratio R1' is less than or equal to a predetermined threshold value.

[0076] In addition, the relative position p(x, y) of rectangular area K in image I and the relative position p1'(x1', y1') of rectangular area K in frame A being approximate means that the difference between the position of point P in the horizontal direction (number of horizontal pixels W) and the position of point P in the vertical direction (number of vertical pixels H) of image I indicated by relative position p(x, y) and the position of point P in the horizontal direction (number of horizontal pixels W1') and the position of point P in the vertical direction (number of vertical pixels H1') of frame A indicated by relative position p1'(x1', y1') is each below a predetermined threshold value.

[0077] The aspect ratio conversion unit 12 trims the image I1' of the frame A from the input image, as shown in FIG. 7(2) and FIG. 8(3) (step S604).

[0078] This results in an image I1' with a target aspect ratio AS1, horizontal pixel count W1', and vertical pixel count H1', ​​as shown in Fig. 8(3). Image I1' includes a rectangular area K, and image I1' is an image that has been trimmed taking into account the size and position of rectangular area K within image I, so as to preserve the creator's intention as indicated by the intention information.

[0079] The aspect ratio conversion unit 12 enlarges or reduces the image I1' having horizontal pixel count W1' and vertical pixel count H1' to an image I1 having horizontal pixel count W1 and vertical pixel count H1 (step S605), and outputs it as an image I1 having a target aspect ratio AS1 (step S606).

[0080] This results in an image I1 with target aspect ratio AS1, horizontal pixel count W1, and vertical pixel count H1, as shown in Fig. 8(4). Image I1 includes rectangular area K1 with horizontal pixel count w1 and vertical pixel count h1, as in Fig. 3, and image I1 was obtained by taking into consideration the size and position of rectangular area K within image I, so as to preserve the intention of the creator as indicated by the intention information. In other words, it is possible to obtain an image I1 that reproduces the creator's intention in a target aspect ratio AS1 different from aspect ratio AS for image I with aspect ratio AS.

[0081] Next, examples of images before and after aspect ratio conversion will be described. Fig. 9 is a diagram showing an example of an image I taken with a loose shot of one person, and images I1 and I2 after aspect ratio conversion, and Fig. 10 is a diagram showing an example of an image I taken with a loose shot of two people, and images I1 and I2 after aspect ratio conversion.

[0082] FIG. 11 is a diagram showing an example of an image I captured in full figure of one person, and images I1 and I2 after aspect ratio conversion, and FIG. 12 is a diagram showing an example of an image I captured in full figure of two people, and images I1 and I2 after aspect ratio conversion.

[0083] FIG. 13 is a diagram showing an example of video I in which one person is shot in a bust shot, and images I1 and I2 after aspect ratio conversion, and FIG. 14 is a diagram showing an example of video I in which two people are shot in a knee shot, and images I1 and I2 after aspect ratio conversion.

[0084] For example, when converting the aspect ratio AS of a horizontally long image I shown on the left side of Figures 9 to 14, the aspect ratio conversion processing device 1 generates images I1 and I2 by trimming and enlarging or reducing processing using the input image based on intention information indicating the creator's intention, so that the size and position of an object in image I approximates the size and position of an object in portrait image I1 shown in the center of Figures 9 to 14, or so that the size and position of an object in landscape image I2 shown on the right side of Figures 9 to 14.

[0085] In this case, rectangular area K with horizontal pixel count w and vertical pixel count h is also enlarged or reduced to rectangular area K1 with horizontal pixel count w1 and vertical pixel count h1. The same is true for rectangular area K2 with horizontal pixel count w2 and vertical pixel count h2.

[0086] The area ratio R2 of a rectangular region K2 indicating the size of an object in a landscape image I2 shown on the right side of Figures 9 to 14 is calculated as R2 = (w2 x h2) / (W2 x H2). The number of horizontal pixels of image I2 is W2, and the number of vertical pixels is H2.

[0087] 11 to 14, the rectangular area K2 in the horizontally long image I2 does not correspond to the entire area of ​​the rectangular area K, but corresponds to a part of it. This is because the aspect ratio conversion unit 12 trims the image I2' including a part of the rectangular area K from the input image, and enlarges or reduces it to the image I2.

[0088] 13 does not correspond to the entire area of ​​rectangular area K, but corresponds to an area larger than rectangular area K (an area including rectangular area K and larger than rectangular area K). This is because aspect ratio conversion unit 12 trims image I1' including an area larger than rectangular area K from the input image, and enlarges or reduces it to image I1.

[0089] Also, referring to Fig. 14, which shows a knee-shot of two people, the number of horizontal pixels W1 of the image I1 and the number of horizontal pixels w1 of the rectangular area K1 may be close to each other, as in the vertically long image I1 shown in the center of Fig. 14. In this case, it can be said that the creator's intention is not faithfully reproduced, so a tiling technique that expresses individual objects (people) may be used.

[0090] Fig. 15 is a diagram showing an example in which a tiling technique is used when converting the aspect ratio of video I in which two people are shot in knee-shot (as in the case of Fig. 14). When the creator's intention cannot be faithfully reproduced, such as in the vertically long video I1 shown in the center of Fig. 14, the aspect ratio conversion unit 12 of the aspect ratio conversion processing device 1 divides the rectangular area K1 into the same number of areas as the number of objects (people), and tiles the objects for each divided area. The same applies when the number of horizontal pixels W2 of video I2 and the number of horizontal pixels w2 of rectangular area K2 are close to each other, making it impossible to faithfully reproduce the creator's intention.

[0091] Furthermore, the aspect ratio conversion unit 12 may provide blank spaces at the top, bottom, left, and right positions of the images I1 and I2 after the aspect ratio conversion, and may superimpose predetermined CG data at the blank positions.

[0092] As described above, according to the aspect ratio conversion processing device 1 of the embodiment of the present invention, the object detection unit 10 trims an image I having aspect ratio AS, horizontal pixel count W, and vertical pixel count H at a predetermined position from the input image, detects an object from the image I, and extracts a rectangular area K containing the object. Then, the object detection unit 10 generates additional information including the aspect ratio AS of the image I, the horizontal pixel count W and vertical pixel count H of the image I, the horizontal pixel count w and vertical pixel count h of the rectangular area K, and the absolute position P(X,Y) and relative position p(x,y) of the rectangular area K.

[0093] The creative intention estimation unit 11 estimates the creator's intention regarding the video I based on the additional information, and generates intention information including the area proportion R and relative position p(x, y) of a rectangular region K in the video I, which indicate the size and position of an object in the video I.

[0094] The aspect ratio conversion unit 12 converts the aspect ratio AS of the image I to the target aspect ratio AS1 of the image I1 by trimming the image I1' from the input image and enlarging or reducing the image I1' to the image I1 so as to retain the creator's intention indicated by the intention information, and generates an image I1 with the target aspect ratio AS1, horizontal pixel count W1 and vertical pixel count H1.

[0095] Specifically, based on the intention information, the aspect ratio conversion unit 12 trims the image I1' from the input image and enlarges or reduces the image I1' to the image I1 so that the area ratio R of the rectangular area K in the image I is approximate to the area ratio R1 of the rectangular area K1 in the image I1, and the relative position p(x, y) of the rectangular area K in the image I is approximate to the relative position p1(x1, y1) of the rectangular area K1 in the image I1.

[0096] In this way, the aspect ratio AS of video I is converted to the target aspect ratio AS1 of video I1, and video I1 is generated with the target aspect ratio AS1, horizontal pixel count W1, and vertical pixel count H1. This allows the creator's intention to be estimated from video I, and to be reproduced in video I1 with the converted target aspect ratio AS1. For example, a composition created by long shots, close-ups, etc. in video I can also be reproduced in video I1.

[0097] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the technical concept thereof.

[0098] For example, the aspect ratio conversion processing device 1 shown in Fig. 1 processes frames (still images) of an input video to convert the aspect ratio. The present invention is applicable not only to still images of an input video, but also to moving images. In other words, the aspect ratio conversion processing device 1 converts the aspect ratio AS of a moving image I into the target aspect ratio AS1 of a moving image I1 by continuously processing each frame constituting the input video.

[0099] A typical computer can be used as the hardware configuration of the aspect ratio conversion processing device 1 according to the embodiment of the present invention. The aspect ratio conversion processing device 1 is configured by a computer equipped with a CPU, a volatile storage medium such as a RAM, a non-volatile storage medium such as a ROM, an interface, etc.

[0100] The functions of the object detection unit 10, the creative intention estimation unit 11, and the aspect ratio conversion unit 12 provided in the aspect ratio conversion processing device 1 are each realized by causing a CPU to execute a program in which these functions are written.

[0101] These programs are stored in the storage medium and are read and executed by the CPU. These programs can also be distributed by storing them in a storage medium such as a magnetic disk (floppy disk, hard disk, etc.), an optical disk (CD-ROM, DVD, etc.), or a semiconductor memory, and can also be transmitted and received via a network. [Industrial Applicability]

[0102] The aspect ratio conversion processing device 1 according to the present invention is used in a wide range of fields, such as imaging devices, video signal conversion devices, video monitoring devices, and editing devices. [Explanation of symbols]

[0103] 1. Aspect ratio conversion processing device 10 Object detection unit 11. Creative Intention Estimation Department 12 Aspect ratio conversion section I, I1, I1', I2, I2' Video AS Aspect Ratio (Image Aspect Ratio) AS1 target aspect ratio (display aspect ratio) W, W1, W1', W2, w, w1, w2 horizontal pixel count H, H1, H1', ​​H2h, h1, h2 vertical pixel count K,K1,K2 rectangular area P(X,Y) absolute position p(x,y),p1(x1,y1),p1'(x1',y1') relative position R, R1, R1', R2 area ratio A frame

Claims

1. The aspect ratio AS of the image I included in the input image is changed to an aspect ratio AS different from the aspect ratio AS. 1 and converting the aspect ratio AS 1 Video I 1 In an aspect ratio conversion processing device which generates an object detection unit that trims the image I from the input image, detects an object from the image I, and determines a position of the object within the image I; a production intention estimation unit that calculates a size of the object in the video I for the object detected by the object detection unit, and estimates the size of the object in the video I and the position of the object in the video I calculated by the object detection unit as intention information indicating the intention of a creator who produced the video I; The image I corresponding to the object included in the image I 1 The objects included in are treated as corresponding objects, The size and position of the object in the image I included in the intention information estimated by the production intention estimation unit, and 1 and the difference between the size and the position of the corresponding object in the image is equal to or smaller than a predetermined threshold value. Image I from the input image 1 ' and trim the image I 1 ' to the image I 1 By enlarging or reducing the image I 1 an aspect ratio conversion unit that generates An aspect ratio conversion processing device comprising:

2. 2. The aspect ratio conversion processing device according to claim 1, The object detection unit is By detecting the object from the image I, a rectangular region K including the object is extracted, and a horizontal pixel count w and a vertical pixel count h of the rectangular region K and a relative position p of the rectangular region K within the image I are calculated; The production intention estimation unit calculating an area ratio R of the rectangular region K in the image I based on a horizontal pixel count W and a vertical pixel count H of the image I and a horizontal pixel count w and a vertical pixel count h of the rectangular region K calculated by the object detection unit; an area ratio R of the rectangular region K in the image I and a relative position p of the rectangular region K in the image I obtained by the object detection unit are estimated as the intention information; The aspect ratio conversion unit is The image I corresponding to the rectangular area K included in the image I 1 Let K be the rectangular area included in 1 As, The area ratio R of the rectangular region K in the image I included in the intention information and 1 The rectangular area K 1 Area ratio R 1 is equal to or smaller than a predetermined threshold value, and the difference between the relative position p of the rectangular area K in the image I included in the intention information and the relative position p of the rectangular area K in the image I 1 The rectangular area K 1 Relative position p 1 so that the difference between The input image is converted into the image I 1 ' and trim the image I 1 ' to the image I 1 By enlarging or reducing the image I 1 4. An aspect ratio conversion processing device comprising:

3. 3. The aspect ratio conversion processing device according to claim 2, The aspect ratio conversion unit is Video I 1 Let A be the frame corresponding to ', The area ratio R of the rectangular region K in the image I and the area ratio R of the rectangular region K in the frame A 1 ' is equal to or smaller than a predetermined threshold, and the difference between the relative position p of the rectangular area K in the image I and the relative position p of the rectangular area K in the frame A is equal to or smaller than a predetermined threshold. 1 The frame A is set so that the difference between the The image of the frame A is extracted from the input image to the image I 1 ' and trim the video I 1 ' to the image I 1 By enlarging or reducing the image I 1 4. An aspect ratio conversion processing device comprising:

4. 4. The aspect ratio conversion processing device according to claim 2, an aspect ratio conversion processing device, characterized in that the relative position p of the rectangular area K in the image I determined by the object detection unit is the center position, upper left position, lower left position, upper right position or lower right position of the rectangular area K, or, if the object is a person, the center position of the person's head.

5. A program for causing a computer to function as the aspect ratio conversion processing device according to claim 1.

Images