Information processing device, method, program, and system

Abstract

To promote the utilization of three-dimensional information of a subject as visual content of an unset medium.SOLUTION: A program causes a computer to function as the means of: acquiring a first medium being a two-dimensional still picture or moving image; specifying a target subject being one of subjects of the first medium; specifying a reference size being a size in a real space from a first edge to a second edge in the reference direction of the target subject; specifying a first edge position being the position of the first edge of the target subject in a coordinate system of the first medium and a second edge position being the position of the second edge in the coordinate system of the first medium; and setting a scale of a display area for displaying a second medium being the two-dimensional still picture or moving image including the target subject on the basis of the first medium, on the basis of the reference size, the first edge position and the second edge position.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, a method, a program, and a system. [Background technology]

[0002] XR (Extended Reality) technologies such as AR (Augmented Reality) and VR (Virtual Reality) technologies are used in various fields. The impression of a viewer on visual content such as XR content can change depending on the content itself as well as how the content is displayed.

[0003] Patent Document 1 discloses a technical idea of ​​registering the size of a virtual information 3D model in advance and superimposing and displaying it on a real space image according to that size. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2011-118834 A Summary of the Invention [Problem to be solved by the invention]

[0005] To utilize the technical idea of ​​Patent Document 1, it is necessary to create a 3D model of the virtual information and further to register its size.

[0006] An object of the present disclosure is to provide a technology that encourages the use of media in which three-dimensional information of a subject is not set as visual content. [Means for solving the problem]

[0007] A program according to one embodiment of the present disclosure causes a computer to function as: means for acquiring a first media which is a two-dimensional still image or video; means for identifying a target subject which is one of the subjects of the first media; means for identifying a reference size which is the size in real space from a first end to a second end of the reference direction of the target subject; means for identifying a first end position which is the position of the first end of the target subject in the coordinate system of the first media and a second end position which is the position of the second end in the coordinate system of the first media; and means for setting the scale of a display area for displaying a second media which is a two-dimensional still image or video based on the first media and which includes the target subject, based on the reference size, the first end position, and the second end position. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a configuration of an information processing system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a block diagram showing a configuration of a creation terminal according to the present embodiment. [Diagram 3] FIG. 2 is a block diagram showing a configuration of a server according to the present embodiment. [Figure 4] FIG. 2 is a block diagram showing a configuration of a playback terminal according to the present embodiment. [Diagram 5] FIG. 1 is an explanatory diagram of one aspect of the present embodiment. [Figure 6] FIG. 2 is a diagram showing a data structure of a content database according to the present embodiment. [Figure 7] 4 is a flowchart of a content creation process according to the present embodiment. [Figure 8] 11A to 11C are diagrams showing examples of screens displayed in the content creation process of the present embodiment. [Figure 9] 4 is a flowchart of a content playback process according to the present embodiment. [Figure 10] 11A to 11C are diagrams showing examples of screens displayed in the content playback process of the present embodiment. [Figure 11] 11A to 11C are diagrams showing examples of screens displayed in the content playback process of the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings for explaining the embodiment, the same components are generally designated by the same reference numerals, and the repeated description will be omitted.

[0010] (1) Information Processing System Configuration The configuration of the information processing system will be described below. Fig. 1 is a block diagram showing the configuration of the information processing system according to the present embodiment.

[0011] As shown in FIG. 1, the information processing system 1 includes a creation terminal 10, a server 30, and a playback terminal 50. The creation terminal 10 and the server 30 are connected via a network (eg, the Internet or an intranet) NW. The playback terminal 50 and the server 30 are connected via a network NW.

[0012] The creation terminal 10 is an example of an information processing device. The creation terminal 10 is, for example, a smartphone, a tablet terminal, or a personal computer. The creation terminal 10 is mainly used to create media constituting visual content based on media (for example, two-dimensional still images or videos). The visual content corresponds to two-dimensional still image content or two-dimensional video content displayed in real space or virtual space. The visual content may correspond to XR content, in which case the visual content may include AR content, VR content, MR (Mixed Reality) content, SR (Substitutional Reality) content, etc. Alternatively, the visual content may constitute a part of two-dimensional or three-dimensional CG (Computer Graphics). In addition, the visual content may further include audio recorded when the original media was shot, or audio based on the audio.

[0013] The playback terminal 50 is, for example, a smartphone, a tablet terminal, or a personal computer, and is used to play back visual content.

[0014] The server 30 is an example of an information processing device. The server 30 is, for example, a server computer. The server 30 accumulates information necessary for playing back the visual content created by the creation terminal 10. The server 30 also provides the information necessary for playing back the visual content to the playback terminal 50.

[0015] (1-1) Configuration of the creation terminal The configuration of the creation terminal will be described below with reference to Fig. 2, which is a block diagram showing the configuration of the creation terminal of this embodiment.

[0016] 2, the creation terminal 10 includes a storage device 11, a processor 12, an input / output interface 13, and a communication interface 14. The creation terminal 10 is connected to a display 21.

[0017] The storage device 11 is configured to store programs and data. The storage device 11 is, for example, a combination of a Read Only Memory (ROM), a Random Access Memory (RAM), and a storage (for example, a flash memory or a hard disk).

[0018] The programs include, for example, the following programs: ·OS (Operating System) programs Programs for applications that process information (e.g. web browsers, applications for creating visual content)

[0019] The data includes, for example, the following data: Databases referenced in information processing Data obtained by performing information processing (i.e., the results of performing information processing)

[0020] The processor 12 is a computer that realizes the functions of the production terminal 10 by starting a program stored in the storage device 11. The processor 12 is, for example, at least one of the following: ·CPU(Central Processing Unit) ·GPU(Graphic Processing Unit) ·ASIC(Application Specific Integrated Circuit) ·FPGA(Field Programmable Gate Array)

[0021] The input / output interface 13 is configured to obtain information (e.g., user instructions) from an input device connected to the creation terminal 10, and to output information (e.g., image signals) to an output device connected to the creation terminal 10.

[0022] The input device is, for example, a microphone, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display 21, a speaker, or a combination thereof.

[0023] The communication interface 14 is configured to control communication between the production terminal 10 and an external device (eg, the server 30).

[0024] The display 21 is configured to display an image (a still image or a moving image). The display 21 is, for example, a liquid crystal display or an organic EL display.

[0025] (1-2) Server configuration The configuration of the server will now be described with reference to Fig. 3, which is a block diagram showing the configuration of the server according to this embodiment.

[0026] As shown in FIG. 3, the server 30 includes a storage device 31, a processor 32, an input / output interface 33, and a communication interface .

[0027] The storage device 31 is configured to store programs and data, and is, for example, a combination of a ROM, a RAM, and a storage (for example, a flash memory or a hard disk).

[0028] The programs include, for example, the following programs: -OS programs Application programs that perform information processing

[0029] The data includes, for example, the following data: Databases referenced in information processing Results of information processing

[0030] The processor 32 is a computer that realizes the functions of the server 30 by starting a program stored in the storage device 31. The processor 32 is, for example, at least one of the following. ·CPU GPU ·ASIC FPGA

[0031] The input / output interface 33 is configured to obtain information (eg, a user's instruction) from an input device connected to the server 30, and to output information (eg, an image signal) to an output device connected to the server 30. The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display.

[0032] The communication interface 34 is configured to control communication between the server 30 and an external device (eg, the production terminal 10 or the playback terminal 50).

[0033] (1-3) Configuration of playback device The configuration of the playback terminal will now be described with reference to Fig. 4, which is a block diagram showing the configuration of the playback terminal of this embodiment.

[0034] 4, the playback terminal 50 includes a storage device 51, a processor 52, an input / output interface 53, and a communication interface 54. The playback terminal 50 is connected to a display 61, a camera 62, and a measurement unit 63.

[0035] The storage device 51 is configured to store programs and data, and is, for example, a combination of a ROM, a RAM, and a storage (for example, a flash memory or a hard disk).

[0036] The programs include, for example, the following programs: -OS programs Programs for applications that process information (e.g. web browsers, applications for playing visual content)

[0037] The data includes, for example, the following data: Databases referenced in information processing Results of information processing

[0038] The processor 52 is a computer that realizes the functions of the playback terminal 50 by starting a program stored in the storage device 51. The processor 52 is, for example, at least one of the following: ·CPU GPU ·ASIC FPGA

[0039] The input / output interface 53 is configured to acquire information (e.g., user instructions, sensing signals, image signals) from an input device connected to the playback terminal 50, and to output information (e.g., image signals, control signals, audio signals) to an output device connected to the playback terminal 50.

[0040] The input device is, for example, a camera 62, a measuring unit 63, a microphone, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display 61, a speaker, or a combination thereof.

[0041] The communication interface 54 is configured to control communication between the playback terminal 50 and an external device (eg, the server 30).

[0042] The display 61 is configured to display an image (still image or moving image). The display 61 is, for example, a liquid crystal display or an organic EL display.

[0043] The camera 62 is configured to capture an image (still image or video) in response to control from the playback terminal 50. The camera 62 typically acquires color information of the subject. Furthermore, the camera 62 (e.g., a stereo camera) or another sensor (not shown, e.g., Lidar (light detection and ranging)) may acquire depth information of the subject.

[0044] The measuring unit 63 performs a predetermined measurement. As an example, the measuring unit 63 includes a GPS (Global Positioning System) module. The playback terminal 50 may specify the position of the camera 62 based on the position information measured by the GPS module.

[0045] (2) One aspect of the embodiment An embodiment of the present invention will now be described with reference to Fig. 5, which is an explanatory diagram of an embodiment of the present invention.

[0046] As shown in FIG. 5, the creation terminal 10 acquires media MD11 (an example of a "first media") that is the basis of the visual content. The media MD11 is a two-dimensional still image or video. It is generated by photographing a subject OBJ10 with a camera CM12. The subject OBJ10 is typically a person, but may be an animal, a robot, a building, a machine, a tree, a rock, or the like. The camera CM12 may be a digital camera product, or may be a camera mounted on a multi-purpose electronic device (which may include the creation terminal 10) such as a smartphone. The camera CM12 may be equipped with a microphone (not shown). In this case, the creation terminal 10 may further acquire audio received by the microphone. In this case, the creation terminal 10 may use the acquired audio, together with the media MD11, as a material for creating the visual content.

[0047] The creation terminal 10 identifies a subject OBJ10 as one of the subjects of the acquired media MD1 to be processed (hereinafter, referred to as a "target subject").

[0048] The creation terminal 10 specifies the vertical size of the subject OBJ10. The vertical size of the subject OBJ10 is the size in real space from the top to the bottom of the subject OBJ10. Since the subject OBJ10 is a person, the size in real space from the top of the head to the soles of the feet of the subject OBJ10, which is usually the height, corresponds to the vertical size. If the real space is defined by an XYZ Cartesian coordinate system, the top and bottom ends are both ends in the Y-axis direction (the vertical direction or height direction in the real space). On the other hand, the X-axis direction corresponds to the left-right direction or width direction in the real space, and the Z-axis direction corresponds to the depth direction or front-back direction in the real space.

[0049] Creation terminal 10 identifies the top end position of subject OBJ10 in the coordinate system (xy coordinate system) of medium MD11 and the bottom end position of subject OBJ10 in the coordinate system of medium MD11. The top end position is the coordinate value of the top end of subject OBJ10, for example, in the y axis direction (the up-down direction or height direction in image space). On the other hand, the bottom end position is the coordinate value of the bottom end of subject OBJ10, for example, in the y axis direction.

[0050] The creation terminal 10 sets the scale of a display area for displaying visual content based on the media MD11 (an example of a "second media"). The visual content is a two-dimensional still image or video that includes a subject OBJ10 common to the media MD11. The display area is a two-dimensional area for displaying the visual content. The display area is defined as a plane that is approximately perpendicular to the horizontal plane, for example. The scale of the display area is the physical size of real space that one pixel of the display area occupies.

[0051] The creation terminal 10 sets the scale of the display area of ​​the visual content based on the vertical size of the subject OBJ10 and the top and bottom end positions of the subject OBJ10. Specifically, the creation terminal 10 determines the size of the display area so that the size of the subject OBJ10 represented by the visual content reproduced in the display area reproduces the actual size (vertical size) of the subject OBJ10. As an example, the creation terminal 10 may set the scale of the display area so that when the visual content is displayed in the display area, the size (height) in real space from the top end display position (pixels) of the subject OBJ10 to the bottom end display position (pixels) matches the vertical size of the subject OBJ10.

[0052] In this way, according to the creation terminal 10 of this embodiment, even if the three-dimensional information of the subject OBJ10 is unknown, the media MD11 in which the subject OBJ10 is captured can be easily used as visual content with a sense of realism.

[0053] The vertical size and the upper and lower end positions may be set for a specific part of the subject (such as the face) rather than for the entire body of the subject.

[0054] (3) Database The databases of this embodiment are stored in the storage device 31.

[0055] (3-1) Content database The content database of this embodiment will now be described with reference to Fig. 6, which is a diagram showing the data structure of the content database of this embodiment.

[0056] The content database stores content information, which is information about visual content created by the server 30.

[0057] 6, the content database includes a "content ID" field, a "content file" field, a "position" field, a "orientation" field, and a "scale" field. Each field is associated with the others.

[0058] The "Content ID" field stores a content ID. The content ID is information for identifying visual content.

[0059] The "content file" field stores content file information. The content file information is information about a content file for playing back visual content. As an example, the content file information may be address information for accessing the content file, or may be data of the content file.

[0060] Here, the content file indicated by the content file information may be original media, or may be media that has been processed as visual content (for example, by cropping, image correction (transparency as one example), addition of visual effects, etc.). When the content file indicated by the content file information is original media, the content information may further include information capable of identifying the processing to be applied to the media. The information capable of identifying the processing may include, for example, information indicating cropping parameters (for example, cropping position), information indicating image correction parameters, information indicating visual effect parameters, etc.

[0061] The "Location" field stores location information. The location information is information about the location of a display area for playing back visual content. The location information may be, for example, geographic coordinate information indicating the installation location of the display area in real space or virtual space.

[0062] The "orientation" field stores orientation information. The orientation information is information about the orientation of the display area for playing back visual content. The orientation information may be, for example, angle information (e.g., roll angle, yaw angle, pitch angle, or a combination thereof) indicating the front direction of the display area in real space or virtual space (i.e., the normal direction of the display area).

[0063] The "scale" field stores scale information. The scale information is information about the correspondence between the resolution of a visual content and the size of a display area for playing back the visual content. The scale information may be, for example, a ratio indicating the size of real space occupied by the display area per pixel of the visual content (for example, 1.26 [mm / pixel], etc.).

[0064] It should be noted that the content information may further include information not shown. As an example, the content information may include playback start condition information. The playback start condition information is information relating to the conditions (playback start conditions) for starting playback of visual content. The playback start condition information defines, for example, the position and orientation that the real camera or virtual camera on the playback side should satisfy. As an example, the playback start condition information may include information capable of identifying a range of position information permitted for the real camera or virtual camera on the playback side, and information capable of identifying a range of spatial features permitted for an image captured by the real camera or virtual camera on the playback side.

[0065] (4) Information processing The information processing of this embodiment will be described.

[0066] (4-1) Content creation process The content creation process of this embodiment will be described below with reference to Fig. 7, which is a flowchart of the content creation process of this embodiment. Fig. 8 is a diagram showing an example of a screen displayed in the content creation process of this embodiment.

[0067] The content creation process of this embodiment may be started, for example, by a user launching a content creation application on the creation terminal 10 and selecting a specific object (for example, an object that accepts user instructions to begin creating visual content).

[0068] As shown in FIG. 7, the production terminal 10 executes acquisition of media (S110). Specifically, the creation terminal 10 acquires media that will be the basis of the visual content to be created. As an example, the user may select desired media from a library in which multiple media are registered, and the creation terminal 10 may acquire the media corresponding to the received selection input from the storage device 11 or an external device. The media is a two-dimensional still image or video, and may also include audio. Furthermore, the creation terminal 10 may acquire the following information together with the media: Information that can identify the position of the camera when the media to be acquired was captured (for example, at least one of the following: the position information of the camera detected by a sensor, or spatial features obtained by analyzing the media) Information that can identify the camera orientation when the media to be acquired was captured (for example, at least one of the following: camera orientation information detected by a sensor, or spatial features obtained by analyzing the media)

[0069] After step S110, the production terminal 10 executes identification of a target subject (S111). Specifically, production terminal 10 identifies a target subject that is any of the subjects of the media acquired in step S110.

[0070] As a first example of specifying the target subject (S111), the creation terminal 10 specifies a subject selected by the user as the target subject.

[0071] As a second example of identifying the target subject (S111), the media acquired in step S110 is image-analyzed, and the target subject is identified based on the analysis results. As an example, the creation terminal 10 may identify, as the target subject, the subject that has the largest area appearing in the media (image) or the subject that is located closest to the center of the media (image) among the subjects that correspond to a predetermined type (e.g., a person), as the target subject.

[0072] After step S111, the production terminal 10 specifies the vertical size (S112). Specifically, the creation terminal 10 specifies the vertical size of the target subject specified in step S111 (the size of the target subject in real space from the top to the bottom).

[0073] As a first example of specifying the vertical size (S112), the creation terminal 10 specifies the vertical size specified by the user as the vertical size of the target subject. For example, the user may specify the vertical size by referring to knowledge about the target subject (e.g., profile or specifications).

[0074] As a second example of specifying the vertical size (S112), the creation terminal 10 may estimate the vertical size based on the type of the target subject. For example, the creation terminal 10 may estimate the average vertical size of individuals belonging to the same type as the target subject as the vertical size of the target subject. Alternatively, the vertical size of the target subject may be determined randomly according to the distribution of the vertical sizes of individuals belonging to the same type as the target subject.

[0075] After step S112, the production terminal 10 executes the specification of the upper end position and the lower end position (S113). Specifically, production terminal 10 identifies the upper and lower end positions of the target subject identified in step S111 (the upper and lower end positions of the target subject in the coordinate system of the media acquired in step S110).

[0076] As a first example of specifying the upper end position and the lower end position (S113), the production terminal 10 specifies the upper end position and the lower end position of the target subject in response to an instruction from the user.

[0077] As an example, the creation terminal 10 displays a user interface including a first object displaying the media acquired in step S110 and a second object that is movable vertically on the media displayed on the first object in response to an instruction from the user on the display 21. Then, the creation terminal 10 may identify at least one of the top end position and the bottom end position based on the position of the second object designated by the user (the vertical coordinate in the coordinate system of the media).

[0078] For example, the creation terminal 10 may display a screen shown in Fig. 8 as an example of such a user interface on the display 21. The screen in Fig. 8 includes objects J20 to J26. Object J20 accepts a designation of the vertical size (e.g., height) of the target subject. Object J20 may be used to accept user instructions in step S112 described above.

[0079] The object J21 accepts the designation of the top end (e.g., head) position of the target subject. When the vertical position of the object J24 (described later) is changed, the production terminal 10 changes the value stored in the object J21 accordingly.

[0080] The object J22 accepts the designation of the bottom end (for example, the feet) position of the target subject. When the vertical position of the object J25 described later is changed, the production terminal 10 changes the numerical value stored in the object J22 accordingly.

[0081] The object J23 is an example of the first object. That is, the object J23 displays the media acquired in step S110. If the media corresponds to a video, the object J23 displays a frame corresponding to any one of the playback times.

[0082] The object J24 is an example of the second object. That is, the object J24 can be moved vertically (in the y-axis direction in the coordinate system of the media) on the media displayed on the object J23 in response to an instruction from the user. The object J24 is used to specify the top position of the target subject. The display position of the object J24 is linked to the numerical value stored in the object J21. When the numerical value stored in the object J21 is changed, the creation terminal 10 changes the display position of the object J24.

[0083] The object J25 is an example of the second object. That is, the object J25 can be moved vertically (in the y-axis direction in the coordinate system of the media) on the media displayed in the object J23 in response to an instruction from the user. The object J25 is used to specify the bottom position of the target subject. The display position of the object J25 is linked to the numerical value stored in the object J22. When the numerical value stored in the object J22 is changed, the creation terminal 10 changes the display position of the object J25.

[0084] The object J26 accepts a user instruction to determine the top and bottom positions of the target subject. When the object J26 is selected, the creation terminal 10 specifies the display position of the object J24 (the x coordinate in the coordinate system of the media displayed in the object J23, which is equal to the value stored in the object J21) as the top position of the target subject, and specifies the display position of the object J25 (the x coordinate in the coordinate system of the media displayed in the object J23, which is equal to the value stored in the object J22) as the bottom position of the target subject.

[0085] As a second example of specifying the upper end position and the lower end position (S113), production terminal 10 specifies the upper end position and the lower end position of the target subject by image analysis of the media acquired in step S110.

[0086] As an example, the creation terminal 10 calculates image features of the media, and recognizes an area of ​​the media corresponding to the target subject based on the image features. The creation terminal 10 may then identify the top and bottom positions based on the vertical coordinates of this area in the media's coordinate system. For example, if the minimum y coordinate of the recognized area is 380 [pixels] and the maximum y coordinate is 1730 [pixels], the creation terminal 10 may identify the top position as 380 [pixels] and the bottom position as 1730 [pixels].

[0087] As a third example of specifying the upper end position and the lower end position (S113), the first and second examples may be combined. For example, the creation terminal 10 may specify one of the upper end position and the lower end position according to the first example, and specify the other according to the second example. Alternatively, the creation terminal 10 may determine the initial position of the second object (e.g., object J24 or object J25) shown in the first example according to the result of specifying according to the second example.

[0088] After step S113, the production terminal 10 executes setting of the display area (S114). Specifically, the creation terminal 10 sets a display area for displaying the visual content to be created. This visual content is based on the media acquired in step S110 and is media that includes the target subject identified in step S111. The creation terminal 10 sets the scale of the display area. Furthermore, the creation terminal 10 may set the position and orientation of the display area as necessary.

[0089] The creation terminal 10 sets the scale of the display area based on the vertical size identified in step S112 and the top and bottom end positions identified in step S113. As an example, the creation terminal 10 may set the scale of the display area so that the size in real space from the bottom end display position to the top end display position of the target subject matches the vertical size when the visual content to be created is displayed in the display area. That is, the creation terminal 10 calculates the number of pixels corresponding to the distance from the top end position to the bottom end position in the vertical direction of the media's coordinate system (i.e., the vertical component of the line segment connecting the top end position and the bottom end position), and divides the vertical size of the target subject in real space by the number of pixels to determine how many mm square of the display area should be used to represent one pixel of the visual content.

[0090] An example of setting the position and orientation of the display area will also be described. Specifically, when information capable of identifying the position and orientation of the camera at the time of shooting the media to be acquired in step S110 has been acquired, the creation terminal 10 may determine the position and orientation of the display area by converting the position and orientation of the target plane in the camera coordinate system into the world coordinate system based on the external parameters of the camera (i.e., the position and orientation of the camera). The target plane is a plane that is perpendicular to the optical axis of the camera and passes through a reference point of the target subject (for example, a predetermined part such as the waist).

[0091] As another example, the creation terminal 10 may identify the center position (i.e., x coordinate) of the target subject in the horizontal direction (i.e., x-axis direction) of the coordinate system of the media acquired in step S110. Then, the creation terminal 10 may set the horizontal arrangement of the display area based on the center position of the target subject. This makes it possible to display the target subject in an appropriate position even if the target subject is captured near the left or right edge of the media acquired in step S110.

[0092] Although not shown, the production terminal 10 may also perform processing of the media, as described below, although processing of the media is not required and the unprocessed media may be displayed in the display area as visual content. For example, the creation terminal 10 extracts an area in which the target subject (or a subject of the same type as the target subject (e.g., multiple people)) appears from the media acquired in step S110, and discards (i.e., trims) or makes transparent the remaining area (i.e., background portion). This allows the creation terminal 10 to create visual content (still image or video) that does not substantially include visual elements other than the target subject (or a subject of the same type as the target subject). Furthermore, the creation terminal 10 may discard the media below the bottom end identified in step S113. This causes the bottom end of the processed media to match the bottom end of the target subject, so the target subject can be displayed with its bottom end in contact with the ground simply by setting the bottom end of the display area to match the ground (which may include the floor). In other words, it is possible to prevent the target subject from being displayed as if it is floating in the air, which would give the viewer a sense of incongruity. As a first option, the creation terminal 10 can make the contour appear smooth by adding at least one of a curve or a straight line of a predetermined color and thickness along the contour of the extracted area. As a second option, the creation terminal 10 can enhance the reality by adding an effect equivalent to a shadow to the lower part of the extracted area.

[0093] Although not shown, the creation terminal 10 may set a playback start condition. However, setting the playback start condition is not essential. As a first example, the creation terminal 10 sets a predetermined range based on the position (coordinates) in the world coordinate system of the camera that captured the media acquired in step S110 as a condition that the position of the real camera on the playback side should satisfy. As a second example, the creation terminal 10 sets a predetermined range based on the coordinates of the virtual space corresponding to the position (coordinates) in the world coordinate system of the camera that captured the media acquired in step S110 as a condition that the position of the virtual camera on the playback side should satisfy. As a third example, the creation terminal 10 sets a predetermined range based on the spatial feature amount obtained by image analysis of the media as a condition that the attitude of the real camera or virtual camera on the playback side should satisfy.

[0094] After step S114, the creation terminal 10 transmits various information about the created visual content to the server 30, and the content creation process of this embodiment ends. Specifically, creation terminal 10 transmits information regarding the scale of the display area determined in step S114 to server 30. Furthermore, creation terminal 10 may transmit to server 30 the media acquired in step S110, or media obtained by processing the media, or information capable of identifying these media (e.g., information indicating a storage destination of the media). Furthermore, creation terminal 10 may transmit to server 30 information regarding the target subject (e.g., information for identifying the target subject from the media, vertical size, top end position, or bottom end position), information regarding the display area (e.g., position or orientation of the display area), or information regarding playback start conditions. Server 30 registers a record of the visual content in the content database (FIG. 6) based on the received information.

[0095] (4-2) Content playback processing The content playback process of this embodiment will be described. Fig. 9 is a flowchart of the content playback process of this embodiment. Fig. 10 is a diagram showing an example of a screen displayed in the content playback process of this embodiment. Fig. 11 is a diagram showing an example of a screen displayed in the content playback process of this embodiment.

[0096] The content playback process of this embodiment can be started, for example, when a user starts a content playback application (which may include a web browser if the content is played back on a web basis) on the playback terminal 50. While the content playback application is running, the playback terminal 50 displays an image captured by the camera 62 on the display 61. When visual content is played back in a virtual space, the camera 62 in the following description can be appropriately interpreted as a virtual camera.

[0097] As shown in FIG. 9, the playback terminal 50 identifies the position and orientation of the camera (S150). Specifically, the playback terminal 50 identifies the position and attitude of the camera 62. For example, the playback terminal 50 acquires position information from a GPS module included in the measurement unit 63, and identifies the position of the camera 62 based on the position information. In addition, the playback terminal 50 analyzes an image captured by the camera 62 to obtain a spatial feature amount (an example of information that can identify the attitude of the camera 62), thereby identifying the attitude of the camera 62.

[0098] After step S150, the playback terminal 50 executes a judgment of a playback start condition (S151). Specifically, the playback terminal 50 judges whether or not the position and orientation identified in step S150 satisfy the playback start condition of any of the visual contents. The playback terminal 50 may acquire in advance playback start condition information registered in, for example, a content database (FIG. 6) from the server 30, and judge whether or not each condition is satisfied. Alternatively, the playback terminal 50 may transmit information capable of identifying the position and orientation of the camera 62 to the server 30, and have the server 30 make the judgment.

[0099] If it is determined in step S151 that the playback start condition corresponding to any visual content is satisfied, playback terminal 50 executes playback of the visual content (S152). Specifically, the playback terminal 50 automatically starts playback of the visual content corresponding to the playback start condition determined to be met, without requiring any additional operation from the user. As an example, the playback terminal 50 acquires a content file of the visual content corresponding to the playback start condition determined to be met, position information, orientation information, and scale information from the server 30. Note that this information may also be acquired in advance. Based on the position information, orientation information, scale information, and the position and attitude of the camera 62, the playback terminal 50 calculates how to superimpose a display area for playing back the visual content on the image captured by the camera 62 (for example, the position, orientation, size, and shape of the display area on the screen of the display 61). Then, the playback terminal 50 starts playing back the visual content in the display area superimposed on the captured image.

[0100] As an example, assume that a playback start condition for visual content corresponding to display area DA1 is met while camera 62 is capturing an image including display area DA1 set around picture PIC2 hung on the wall, as shown in Fig. 10. In this case, the visual content is played back in display area DA1, superimposed on the image captured by camera 62, as shown in Fig. 11. In this visual content, subject OBJ3 is displayed in life-size, so that the user of playback terminal 50 can experience the sensation of actually having subject OBJ3 in the location.

[0101] After step S152, the playback terminal 50 re-executes the determination of the position and attitude of the camera (S150). Note that, when the playback terminal 50 detects a change in the position and attitude of the camera in step S150, it can perform at least one of the following processes. Re-determining how the display area of ​​the visual content being played is superimposed on the image captured by the camera 62 (i.e., changing at least one of the position, orientation, size, or shape of the display area on the screen of the display 61) The playback of the visual content is interrupted or terminated (for example, when the display area of ​​the visual content is no longer within the range of the camera 62).

[0102] If it is determined in step S151 that the playback start condition corresponding to any visual content is not satisfied, the playback terminal 50 may re-execute the process of identifying the position and orientation of the camera (S150).

[0103] (5) Summary As described above, the creation terminal 10 of the present embodiment acquires a medium (hereinafter, for convenience, referred to as a "first medium") that is a two-dimensional still image or video, and identifies a target subject that is one of the subjects of the first medium. The creation terminal 10 identifies a vertical size that is the size in real space from the top end to the bottom end of the target subject, and identifies an upper end position that is the position of the top end of the target subject in the coordinate system of the first medium, and a lower end position that is the position of the bottom end of the target subject in the coordinate system of the first medium. The creation terminal 10 sets the scale of a display area for displaying a medium (hereinafter, for convenience, referred to as a "second medium") that is a two-dimensional still image or video that is based on the first medium and includes the target subject, based on the vertical size, the upper end position, and the lower end position. This makes it possible to easily utilize a medium that captures the target subject as a visual content with a sense of realism, even if the three-dimensional information of the target subject is unknown.

[0104] The creation terminal 10 may set the scale of the display area so that the size in real space from the bottom display position to the top display position of the target subject matches the vertical size when the second media is displayed in the display area. This makes it possible to efficiently create visual content that displays the target subject in life-size.

[0105] The creation terminal 10 may specify at least one of the upper end position and the lower end position in response to an instruction from the user, thereby making it possible to specify the upper end and the lower end positions intended by the user.

[0106] The creation terminal 10 may display on the display unit a first object that displays a first medium and a second object that can be moved vertically on the first medium displayed on the first object in response to an instruction from the user, and may specify at least one of the top end position and the bottom end position based on the position of the second object. This allows the user to easily specify the top end and the bottom end by an intuitive operation.

[0107] The production terminal 10 may identify at least one of the top and bottom end positions by image analysis of the first medium, thereby further reducing the burden on the user.

[0108] The creation terminal 10 may recognize an area corresponding to a target subject in the first medium based on the image feature amount of the first medium, and specify at least one of the top end position and the bottom end position based on the vertical coordinate of the area in the coordinate system of the first medium. This allows the top end position and the bottom end position to be appropriately specified.

[0109] The creation terminal 10 may identify the center position of the target subject in the horizontal direction of the coordinate system of the first medium, and set the horizontal arrangement of the display area based on the center position of the target subject. This allows the horizontal arrangement of the display area to be set based on the center position of the target subject, not the center position of the second medium, so that even if the center position of the second medium and the center position of the target subject are misaligned, the target subject is displayed in an appropriate position and is less likely to give a sense of incongruity to the viewer.

[0110] (6) Other modifications The storage device 11 may be connected to the creation terminal 10 via a network NW. The display 21 may be integrated with the creation terminal 10. The storage device 31 may be connected to the server 30 via the network NW. The storage device 51 may be connected to the reproduction terminal 50 via the network NW. At least one of the display 61, the camera 62, and the measurement unit 63 may be integrated with the reproduction terminal 50.

[0111] In the above description, an example was given in which each step in each process is executed in a specific order, but the execution order of each step is not limited to the example described above, as long as there is no dependency. Each step in the above process can be executed by any of the creation terminal 10, the server 30, and the playback terminal 50. For example, the playback terminal 50 may cause the server 30 to determine the playback start condition (S151). Also, in the present embodiment, an example was given in which the creation terminal 10 executes the content creation process and uploads the execution result to the server 30, but the server 30 may execute a part or all of the process. The division of the processing between the creation terminal 10 and the server 30, or the division of the processing between the server 30 and the playback terminal 50, can be determined arbitrarily.

[0112] In the above description, an example was given in which the scale of the display area is set based on the vertical size of the target subject and the top and bottom positions of the target subject. However, the bottom of the target subject may not appear in the media, such as in a video in which only the upper body of a person is shown. In this case, the creation terminal 10 may operate as described below. That is, the creation terminal 10 may determine whether or not the bottom of the target subject appears in the first media acquired in step S110. The determination may be performed using image analysis, or the result of the user's judgment may be used as the determination result. Then, when it is determined that the bottom of the target subject does not appear in the media, the creation terminal 10 may determine a part vertical size, which is the size in real space from the top to the bottom of a specific part of the target subject, in the same manner as in determining the vertical size (S112). The specific part is, for example, the face of the target subject. Furthermore, when it is determined that the bottom end of the target subject is not shown in the media, the creation terminal 10 may specify a part upper end position, which is the position of the top end of the specific part in the coordinate system of the media, and a part lower end position, which is the position of the bottom end of the specific part in the coordinate system of the media, in the same manner as in the determination of the top end position and the bottom end position (S113). Then, when it is determined that the bottom end of the target subject is not shown in the media, the creation terminal 10 may set the scale of the display area based on the part vertical size, the part upper end position, and the part lower end position in the display area setting (S114). As an example, the creation terminal 10 may set the scale of the display area so that when the visual content to be created is displayed in the display area, the size in real space from the display position of the top end to the display position of the bottom end of the specific part matches the part vertical size. By such processing, it is possible to efficiently create life-size visual content based on media in which the bottom end of the target subject is not shown. Note that a similar process may be performed when the top end of the target subject, rather than the bottom end, is not shown in the media.

[0113] In the above description, an example is shown in which the scale of the display area is set based on the vertical size of the target subject and the upper and lower end positions of the target subject. However, when the target subject has movable parts, the vertical size of the target subject may be changed according to the posture of the target subject. For example, when the target subject is a person, the creation terminal 10 may determine the posture of the target subject. The determination may be performed using image analysis, or the result of a user's judgment may be used as the determination result. When it is determined that the target subject is sitting on a chair, the creation terminal 10 may specify the sitting height of the target subject as the vertical size in the determination of the vertical size (S112). Then, when it is determined that the target subject is sitting on a chair, the creation terminal 10 may set the scale of the display area based on the sitting height, the upper end position, and the lower end position in the setting of the display area (S114). As an example, the creation terminal 10 may set the scale of the display area so that when the visual content to be created is displayed in the display area, the size in real space from the lower end display position to the upper end display position of the target subject matches the sitting height. This process allows for an appropriate scale to be set for media that shows a target subject sitting in a chair, making it possible to efficiently create life-size visual content.

[0114] As another example, the creation terminal 10 may use, for example, body tracking technology to predict the top end position when the target subject is made to assume a reference posture (for example, a standing posture such as a T-pose) regardless of the posture of the target subject. Then, the creation terminal 10 may set the scale of the display area based on the predicted top end position.

[0115] In the above description, an example has been shown in which the playback terminal 50 calculates how to superimpose a display area for playing back visual content on an image captured by the camera 62 based on the position information, orientation information, and size information of the visual content, as well as the position and attitude of the camera 62. However, the playback terminal 50 may control the visual content so that the visual content faces a predetermined direction (for example, the front) regardless of the position or attitude of the camera 62 (or the virtual camera) throughout the playback of the visual content. This allows the user to view the visual content facing a predetermined direction even if the position or attitude of the camera 62 (or the virtual camera) is slightly shifted. In addition, the amount of calculation related to the control of the direction of the display area can be reduced. Note that, when the position or attitude of the camera 62 (or the virtual camera) is shifted beyond an allowable range (for example, when the playback start condition of the visual content is no longer satisfied), the playback terminal 50 may interrupt or end the playback of the visual content. This can prevent, for example, a deviation between the direction in which a person points with their line of sight, finger, or hand during shooting and the direction in which a person depicted in the visual content points, causing a sense of incongruity to the user.

[0116] In the above description, an example has been shown in which, when it is determined that the playback start condition is satisfied, playback of the corresponding visual content is automatically started without requiring any additional operation from the user. However, on the condition that the playback start condition is satisfied and the user performs a predetermined operation (e.g., a tap operation), the playback terminal 50 may play the corresponding visual content.

[0117] In the above description, an example in which the measuring unit 63 includes a GPS module has been described. However, the measuring unit 63 may include other components in addition to or instead of the GPS module. For example, the measuring unit 63 may include at least one of an acceleration sensor, a geomagnetic sensor, or a gyro sensor. In this case, the attitude of the camera 62 can be specified based on a sensing signal from such a sensor, or the attitude of the camera 62 can be specified based on a combination of the sensing signal and the spatial feature amount of the image. Alternatively, the position or attitude of the camera 62 can be specified based only on the spatial feature amount of the image, as in the case of a VPS (Visual Positioning Service / System). In this case, the measuring unit 63 can be omitted.

[0118] In the above description, an example has been shown in which the playback start condition that the position and attitude of the playback camera must satisfy to start playback of visual content is set based on the position and attitude of the recording camera. However, the playback start condition is not limited to this example, and for example, a trigger installed in the real world or virtual world may be set. The trigger may be, for example, an object such as a product or an exhibit, or a code such as a QR code (registered trademark). When the trigger is an object, the playback start condition may be determined to be that the object is recognized from the captured image by an object recognition model, and when the trigger is a code, the playback start condition may be determined to be that the information of the code is decoded based on the captured image. Furthermore, multiple types of playback start conditions may be set for the same visual content, and playback of the visual content may be started when any of the playback start conditions is satisfied.

[0119] In the above description, an example of playing back visual content has been shown. However, visual content may be presented to a user in combination with various auxiliary information. As a first example, a link to a specific web page (for example, a top page of a specific EC (Electronic Commerce) site, a page on which a specific product is posted, or an explanatory web page on which detailed information on an exhibit is posted) may be embedded in the visual content. When a user selects visual content (for example, by tapping, etc.), the playback terminal 50 may display the linked web page automatically or subject to the user's approval. As a second example, an object that displays link information for accessing a specific web page may be arranged on the screen together with the visual content or after the playback of the visual content is completed. When a user selects this object (for example, by tapping, etc.), the playback terminal 50 may display the linked web page automatically or subject to the user's approval. As a third example, an object that accepts an operation related to the purchase of a specific product on a specific EC site may be arranged on the screen together with the visual content or after the playback of the visual content is completed. When the user selects this object (e.g., by tapping, etc.), the playback terminal 50 may automatically, or subject to the user's approval, send a request to purchase a specific product (e.g., a request to add a specific product to a cart linked to the user account, or a request to open a payment screen for the specific product) to a server of a specific EC site.

[0120] In the above description, an example in which the display area is set in real space has been shown. However, the display area may be set in virtual space. In this case, the creation terminal 10 may set the scale of the display area so that the size in the virtual space from the display position of the lower end to the display position of the upper end of the target subject corresponds to the vertical size in the real space. Also, when the display area is set in virtual space, the establishment / non-establishment of the playback start condition is determined for the position and attitude in the virtual space of the virtual camera instead of the real camera.

[0121] In the above description, an example was given in which the scale of the display area is set based on the "vertical size," "top end position," and "bottom end position" of the target subject. However, instead of vertical, the horizontal direction, the diagonal direction, or other directions may be used as the reference direction. That is, the "vertical size" can be read as the "reference size." The reference size is the size in real space from the first end to the second end of the reference direction of the target subject. In addition, the "top end position" can be read as the "first end position," and the "bottom end position" can be read as the "second end position." The first end position is the position of the first end of the target subject in the coordinate system of the first medium (for example, the coordinate value of the direction corresponding to the reference direction in the coordinate system of the first medium), and the second end position is the position of the second end in the coordinate system of the first medium (for example, the coordinate value of the direction corresponding to the reference direction in the coordinate system of the first medium). The second object described above may be configured to be movable along a direction corresponding to the reference direction in the coordinate system of the eleventh medium. Any direction can be used as the reference direction, but it is necessary that the size of the target subject in real space between both ends in that direction can be specified. Also, the distance between the first end and the second end in the coordinate system of the first medium must be larger than the image resolution of the first medium. From the viewpoint of setting the scale of the display area with higher accuracy, the longitudinal direction of the target subject is suitable as the reference direction.

[0122] Although the embodiment of the present invention has been described in detail above, the scope of the present invention is not limited to the above embodiment. Furthermore, the above embodiment can be improved or modified in various ways without departing from the spirit of the present invention. Furthermore, the above embodiment and the modified examples can be combined. [Explanation of symbols]

[0123] 1: Information processing system 3D: Virtual Information 10: Creation terminal 11:Storage device 12: Processor 13: Input / Output Interface 14: Communication interface 21: Display 30: Server 31:Storage device 32 : Processor 33: Input / Output Interface 34: Communication interface 50: Playback terminal 51: Storage device 52: Processor 53: Input / Output Interface 54: Communication interface 61: Display 62: Camera 63: Measurement section

Claims

1. Computer, means for acquiring a first media, the first media being a two-dimensional still or video image; means for identifying a target subject, the target subject being one of the subjects of the first medium; a means for specifying a reference size, which is a size in real space from a first end to a second end in a reference direction of the target subject; a means for identifying a first end position, which is a position of a first end of the target subject in a coordinate system of the first medium, and a second end position, which is a position of a second end of the target subject in the coordinate system of the first medium; a means for setting a scale of a display area for displaying a second medium, which is a two-dimensional still image or video based on the first medium and includes the target subject, based on the reference size, the first end position, and the second end position; A program that functions as a

2. the setting means sets a scale of the display area so that, when the second medium is displayed in the display area, a size in real space from a display position of a second end to a display position of a first end of the target subject matches the reference size. The program according to claim 1.

3. the means for identifying the first end position and the second end position identifies at least one of the first end position and the second end position in response to an instruction from a user. The program according to claim 1 or 2.

4. the means for identifying the first end position and the second end position displays, on a display unit, a first object displaying the first medium and a second object movable on the first medium displayed on the first object in a direction corresponding to the reference direction in a coordinate system of the first medium in response to an instruction from the user, and identifies at least one of the first end position and the second end position based on a position of the second object. The program according to claim 3.

5. the means for identifying the first end position and the second end position identifies at least one of the first end position and the second end position by image analysis of the first medium. The program according to claim 1 or 2.

6. the means for identifying the first end position and the second end position recognizes an area in the first medium corresponding to the target subject based on an image feature amount of the first medium, and identifies at least one of the first end position and the second end position based on a coordinate of the area in a direction corresponding to the reference direction in a coordinate system of the first medium; The program according to claim 5.

7. The computer, means for determining whether a second edge of the target subject is captured in the first medium; a means for specifying a part reference size, which is a size in real space from a first end to a second end in a reference direction of a specific part of the target subject, when it is determined that the second end of the target subject is not captured in the first medium; a means for specifying a first part position, which is the position of the first end of the specific part in a coordinate system of the first medium, and a second part position, which is the position of the second end of the specific part in the coordinate system of the first medium, when it is determined that the second end of the target subject is not included in the first medium; Function as a the setting means sets a scale of the display area based on the part reference size, the first part position, and the second part position when it is determined that the second edge of the target subject is not included in the first medium. The program according to claim 1.

8. the setting means sets a scale of the display area so that, when the second medium is displayed in the display area, a size in real space from a display position of a first end to a display position of a second end of the specific part matches the part reference size. The program according to claim 7.

9. the reference direction is a vertical direction, causing the computer to further function as a means for identifying a center position of the target subject in a horizontal direction of a coordinate system of the first medium; the setting means sets the horizontal arrangement of the display area based on a center position of the target subject. The program according to claim 1.

10. the target subject is a person, the reference direction is a vertical direction, causing the computer to function as a means for determining a posture of the target subject; When it is determined that the target subject is sitting on a chair, the means for specifying the reference size specifies a sitting height of the target subject as the reference size, When it is determined that the target subject is sitting on a chair, the setting means sets the scale of the display area based on the sitting height, the first end position, and the second end position. The program according to claim 1.

11. the setting means sets a scale of the display area so that, when the second medium is displayed in the display area, a size in real space from a display position of a second end of the target subject to a display position of a first end of the target subject matches the sitting height. The program according to claim 10.

12. The computer acquiring a first media, the first media being a two-dimensional still or video image; identifying a target subject, the target subject being one of the subjects of the first media; determining a reference size, which is a size in real space from a first end to a second end in a reference direction of the target subject; identifying a first end position that is a position of a first end of the target subject in a coordinate system of the first medium and a second end position that is a position of a second end of the target subject in the coordinate system of the first medium; setting a scale of a display area for displaying a second medium, which is a two-dimensional still image or video based on the first medium and includes the target subject, based on the reference size, the first end position, and the second end position; A method for performing.

13. means for acquiring a first media, the first media being a two-dimensional still or video image; means for identifying a target subject, the target subject being one of the subjects of the first medium; a means for determining a reference size, which is a size in real space from a first end to a second end in a reference direction of the target subject; a means for identifying a first end position, which is a position of a first end of the target subject in a coordinate system of the first medium, and a second end position, which is a position of a second end of the target subject in the coordinate system of the first medium; a means for setting a scale of a display area for displaying a second medium, which is a two-dimensional still image or video based on the first medium and includes the target subject, based on the reference size, the first end position, and the second end position; An information processing device comprising:

14. A system including a plurality of information processing devices, means for acquiring a first media, the first media being a two-dimensional still or video image; means for identifying a target subject, the target subject being one of the subjects of the first medium; a means for determining a reference size, which is a size in real space from a first end to a second end in a reference direction of the target subject; a means for identifying a first end position, which is a position of a first end of the target subject in a coordinate system of the first medium, and a second end position, which is a position of a second end of the target subject in the coordinate system of the first medium; a means for setting a scale of a display area for displaying a second medium, which is a two-dimensional still image or video based on the first medium and includes the target subject, based on the reference size, the first end position, and the second end position; A system comprising:

Images