Information processing apparatus, information processing method, and program

Abstract

To provide an information processing apparatus, an information processing method, and a program for reducing the workload of a user for virtual fitting.SOLUTION: In an information processing apparatus, when a subject included in a captured image obtained by a camera satisfies a predetermined determination condition, an image processing unit defines the captured image as a saved image, synthesizes an image of at least one object selected from among a plurality of objects with the subject included in the captured image, and displays the synthesized image.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an information processing technique for combining an object image, such as an image of a product, with a subject included in an image. [Background technology]

[0002] Virtual try-on services using augmented reality have been introduced on e-commerce sites and the like. For example, a system is known that synthesizes an object image such as a product onto an image of the user's body taken with a built-in camera on a smartphone or the like, and displays the synthesized image on a built-in display, providing the user with the experience of actually wearing the product. A product selection UI (user interface) is displayed on the display, and when a user selects a new product via the product selection UI, a new image is displayed with the selected product synthesized onto it. This allows the user to virtually try on various products in a short period of time.

[0003] Patent document 1 discloses a technology that detects the orientation of a user's body from an image and displays a composite image generated during a period when the user cannot see the composite image on the display with a predetermined delay. Patent Document 2 discloses a technology in which a captured image is saved using a first trigger detected from the image as a user action such as closing the eyes, and the saved image is played back using a second trigger detected as a user action such as directing the user's gaze toward the camera. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2010-152556 A [Patent Document 2] US Patent Application Publication No. 2021 / 0258534 Summary of the Invention [Problem to be solved by the invention]

[0005] Users who use virtual try-on services often check composite images of themselves and the product in multiple poses to determine whether the product meets their needs. Users also often select products from multiple products to compare and consider. In this case, the user must repeatedly select products to virtually try on from the multiple products displayed in the product selection UI and take photos of themselves in multiple poses for each selected product, which is a very burdensome task for the user.

[0006] Therefore, an object of the present invention is to make it possible to reduce the workload of a user. [Means for solving the problem]

[0007] The information processing device of the present invention is characterized by having a storage means for storing a captured image as a saved image when a subject contained in the captured image satisfies predetermined judgment conditions, a selection means for selecting at least one of a plurality of objects, a synthesis means for synthesizing an image of the selected object with the subject contained in the saved image, and a display processing means for displaying a synthetic image synthesized by the synthesis means. Effect of the Invention

[0008] According to the present invention, it is possible to reduce the workload of a user. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a virtual try-on system. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a mobile terminal. [Diagram 3] FIG. 2 illustrates an example of a hardware configuration of a server. [Figure 4] FIG. 4 is a block diagram showing the functions of a virtual try-on processing unit. [Diagram 5] FIG. 13 is a diagram showing an example of a virtual try-on UI. [Figure 6] 13 is a flowchart showing the operation of the virtual try-on processing unit. [Figure 7] FIG. 2 is a block diagram showing functions of an image processing unit. [Figure 8] 13 is a flowchart showing the operation of an image processing unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The following embodiment does not limit the present invention, and not all of the combinations of features described in the present embodiment are necessarily essential to the solution of the present invention. The configuration of the embodiment may be appropriately modified or changed depending on the specifications of the device to which the present invention is applied and various conditions (such as the conditions of use and the environment in which the device is used). In the following embodiments, the same configurations and functions are given the same reference symbols, and duplicated descriptions will be omitted.

[0011] <System configuration example> The information processing device according to this embodiment is a device capable of generating and displaying an image in which a predetermined object is virtually worn on a subject. In this embodiment, a predetermined part of the user (subject part) is taken as an example of the subject, and a product is taken as an example of the predetermined object. In this embodiment, a virtual wearing is taken in which a user photographs a specific part of his / her body with a photographing device (camera), and an image of the product is synthesized and displayed on the image of the subject part, thereby making it appear as if the product is virtually worn on the specific part. In this manner, this embodiment will be described by taking an example of application to a service that provides an experience of the user actually wearing the product. In the following description, virtually wearing a product on a specific part of the user's body will be described as "virtual try-on".

[0012] Fig. 1 is a diagram showing a schematic configuration of a virtual try-on system 1 according to this embodiment. Fig. 1 shows, as an example, a configuration example of the virtual try-on system 1 in which a mobile terminal 3 used by a user 2 and a server 5 that constructs an e-commerce site are connected via the Internet 4. The mobile terminal 3 used by the user 2 is an information processing device according to this embodiment, and is, for example, a smartphone equipped with a camera and a display. The server 5 is, for example, a computer.

[0013] In this embodiment, a wristwatch is given as an example of a product handled on an electronic commerce site. The product that the user 2 virtually tries on is not limited to a wristwatch, and may be other types of products, such as clothes, hats, glasses, accessories, and shoes. The product that the user 2 virtually tries on is not limited to one or one type, and the user 2 may virtually try on multiple products or multiple types of products, or the products and types may be appropriately selected and switched. When the user 2 virtually tries on a product, a virtual try-on user interface screen (hereinafter referred to as a virtual try-on UI), which will be described later, is displayed on the display of the mobile terminal 3. Therefore, the user 3 can select a product, a product type, and a saved image, which will be described later, by operating the virtual try-on UI.

[0014] FIG. 2 is a diagram showing an example of the main hardware configuration of the mobile terminal 3, which is an example of an information processing device according to this embodiment. A central processing unit 301 uses a main memory device 302 as a work memory, executes an operating system (OS) and various programs stored in a solid state drive (SSD) that is an auxiliary memory device 303, and controls each component via a system bus 304. In addition to the central processing unit 301, main memory device 302, and auxiliary memory device 303, a communication device 305, a camera 306, a display 307, a touch panel 308, etc. are connected to the system bus 304.

[0015] The communication device 305 is a communication unit that can connect to the Internet or the like by wireless communication such as a mobile phone network. The display 307 is a display unit equipped with a liquid crystal display, an organic electroluminescence display, or the like, and the touch panel 308 is a user input device that allows a user to perform an operation input by touching it with a finger or the like. The touch panel 308 is transparent and is attached to the display surface of the display 307. The central processing unit 301 causes the display 307 to display various UIs such as a virtual fitting UI, and when the user performs an operation instruction such as touching the UI with a finger, the touch panel 308 acquires the operation instruction and sends it to the central processing unit 301. As a result, the central processing unit 301 executes a process according to the operation instruction from the user.

[0016] The camera 306 is an imaging unit (photographing device) having a CMOS image sensor and a lens. The mobile terminal 3 of this embodiment is provided with two imaging units, a so-called outer camera and inner camera. The outer camera is a camera arranged on the side of the housing opposite to the side on which the display 307 is arranged, and the inner camera is a camera arranged on the same side of the housing as the display 307. In this embodiment, the outer camera is used to photograph, for example, the hands and feet of the user 2, and the inner camera is used to photograph, for example, the face and chest of the user 2. In this embodiment, an example is given of virtually trying on a wristwatch, and the user photographs his or her hand (for example, the area from the hand to the elbow), so the camera 306 is considered to be an outer camera.

[0017] FIG. 3 is a diagram showing an example of the main hardware configuration of the server 5 according to this embodiment. A central processing unit 501 uses a main memory device 502 as a work memory, executes an operating system (OS) and various programs stored in a hard disk drive (HDD) that is an auxiliary memory device 503, and controls each component via a system bus 504. A communication device 505 that can be connected to the Internet, etc. is also connected to the system bus 504.

[0018] An outline of the operation of the virtual try-on system 1 of this embodiment when the user 2 operates the mobile terminal 3 to virtually try on an item will be described below. First, when a command to execute a web browser program is input from user 2 through operation of touch panel 308, central processing unit 301 of mobile terminal 3 starts the web browser and controls communication device 305 to connect to the Internet 4. Furthermore, when a command to access an e-commerce site is input from user 2 through operation of the web browser via touch panel 308, central processing unit 301 accesses server 5 that operates the e-commerce site via communication device 305.

[0019] When the central processing unit 501 of the server 5 receives access from the mobile terminal 3 via the communication device 505, it reads out a virtual try-on program and a product information list prepared and stored in advance in the auxiliary storage device 503, and transmits them from the communication device 505 to the mobile terminal 3. The virtual try-on program is an example of an information processing program according to this embodiment. When the central processing unit 301 of the mobile terminal 3 receives the virtual try-on program and the product information list from the server 5 via the communication device 305, it stores the received information in the auxiliary storage device 303, and then reads out and starts the virtual try-on program.

[0020] Here, the product information list includes a three-dimensional product shape model, a product icon, a product representative direction vector, and product detail information for each of a plurality of products (watches in this embodiment). The three-dimensional product shape model is data including three-dimensional model data and textures used to generate a CG (computer graphics) image of the product. When the product is a wristwatch, the three-dimensional product shape model is expressed in a three-dimensional model coordinate system in which the center position of the back of the wristwatch case is the product's origin coordinate, the axis perpendicular to the dial surface is the Z axis, and the axes perpendicular to the dial surface on a two-dimensional plane parallel to the dial are the X axis and the Y axis. In this embodiment, the Y axis is set as the axis in the 3 o'clock direction on the dial surface (i.e., the direction in which the crown is arranged). Although details will be described later, the product's origin coordinate, which is the center position of the back of the wristwatch case, is used as the product synthesis position for the shooting space of the camera 306 of the mobile terminal 3.

[0021] The product icon is an icon image representing a product, and the central processing unit 301 of the mobile terminal 3 displays the product icon in a product selection UI area in a virtual fitting UI, which will be described later. The detailed product information is information indicating product specifications, price, and the like.

[0022] The product representative direction vector is a vector that represents the observation direction of the product so that the user 2 can well grasp the features of the product when the product is virtually tried on. In this embodiment, the product representative direction vector is set as a line of sight vector that represents an observation direction so that the user 2 can well grasp the features of the product virtually tried on, assuming that the optical axis of the camera 306 (the viewpoint of the user 2) is on the Z axis passing through the origin coordinate of the model coordinate system. Since the user 2 may observe the product from various directions, multiple product representative direction vectors may be set for one product. In other words, multiple observation directions may be set for one product, and the product representative direction vector may include multiple line of sight vectors corresponding to each of the multiple observation directions. For example, in the case of a wristwatch, there may be a line of sight vector that represents the observation direction when the user 2 looks at the dial, a line of sight vector that represents the observation direction when the user 2 operates the crown, and a line of sight vector that represents the observation direction when the user 2 looks at (operates) the buckle of the belt. In the case of the wristwatch illustrated in this embodiment, it is assumed that these three line of sight vectors are included in the product representative direction vector.

[0023] Fig. 4 is a functional block diagram showing each functional unit formed by the central processing unit 301 of the mobile terminal 3 by executing a virtual fitting program, which is an information processing program according to this embodiment. Note that Fig. 4 also shows the main components shown in Fig. 2 other than the central processing unit 301. The central processing unit 301 of the mobile terminal 3 executes the virtual fitting program to operate as a virtual fitting processing unit 6 including the functions of an image processing unit 8, a state management unit 9, and a display processing unit 10 as shown in FIG.

[0024] The virtual fitting processing unit 6 operates in one of two operation modes, a first operation mode and a second operation mode. The first operation mode and the second operation mode differ in the screen layout of the display 307 and the processing in the image processing unit 8. The first operation mode is a real-time mode in which a product image is synthesized with an image captured by the camera 306 and displayed in an image display area provided on the screen of the display 307. The second operation mode is a playback mode in which, when an image captured by the camera 306 is stored in the auxiliary storage device 303 through a storage determination process described later, a product image is synthesized with the stored image and displayed in the image display area of ​​the display 307. In the following embodiment, the first operation mode is described as a "real-time mode" and the second operation mode is described as a "playback mode". These real-time mode and playback mode are managed by the state management unit 9, and are set to the real-time mode when the virtual fitting program is started, and are appropriately switched according to an instruction from the user.

[0025] 5(a) and 5(b) are diagrams showing an example of the screen layout of the display 307. The central processing unit 301 displays a virtual dressing UI 7 as shown in Fig. 5(a) or Fig. 5(b) on the display 307 depending on whether the operating mode is the real-time mode or the playback mode. Fig. 5(a) is a diagram showing the screen layout of the virtual dressing UI 7 displayed in the real-time mode, and Fig. 5(b) is a diagram showing the screen layout of the virtual dressing UI 7 displayed in the playback mode.

[0026] The virtual try-on UI 7 displayed in the real-time mode shown in FIG. 5(a) includes an image display area 701, a product selection UI area 702, and a navigator UI area 703. The product selection UI area 702 is an area in which a plurality of product icons are displayed. If there are product icons that cannot be displayed in the product selection UI area 702, for example, left and right scroll buttons 725 are also displayed in the product selection UI area 702. When the left and right scroll buttons 725 are operated, the product icons that were not displayed are displayed in the product selection UI area 702 in response to the operation. The user 2 can select a desired product icon from the plurality of product icons displayed in the product selection UI area 702 through operation of the touch panel 308. The image display area 701 in the real-time mode is an area in which a composite image obtained by combining an image captured by the camera 306 with a product image corresponding to a product icon selected from a product selection UI area 702 is displayed.

[0027] The navigator UI area 703 is an area where a purchase button 731, a product details button 732, a mode change button 733, an end button 734, and the like are arranged and displayed. The product details button 732 is a button used when the user 2 instructs to display detailed information of a product corresponding to a product icon selected by the user 2 from within the product selection UI area 702. When the user 2 touches the product details button 732 via the touch panel 308, a product details information panel in which detailed information of the product is described is displayed on the screen of the virtual dressing UI 7. The mode change button 733 is a button used when the user 2 instructs to change the operation mode. In the case of the virtual dressing UI 7 in the real-time mode shown in FIG. 5(a), the mode change button 733 is displayed as a button representing a change to the playback mode. When the user 2 touches the mode change button 733 via the touch panel 308 to instruct a change to the playback mode, the screen of the virtual dressing UI 7 transitions to the screen of the playback mode shown in FIG. 5(b). The purchase button 731 is a button used when purchasing a product corresponding to a product icon selected from the product selection UI area 702. When the user 2 touches the purchase button 731 via the touch panel 308, the screen of the display 307 transitions to a product purchase web page. The end button 734 is a button used when the user 2 instructs to end the virtual try-on program. When the user 2 touches the end button 734 via the touch panel 308, the mobile terminal 3 ends the virtual try-on program.

[0028] The virtual fitting UI 7 displayed in playback mode shown in Figure 5(b) displays an image display area 701, product selection UI area 702, and navigator UI area 703, which are generally similar to the example in Figure 5(a), as well as a saved image selection UI area 704. The saved image selection UI area 704 is displayed in a direction perpendicular to the longitudinal direction of the product selection UI area 702. The saved image selection UI area 704 is an area in which the saved images 741, 742, and 743 photographed by the camera 306 and stored in the auxiliary storage device 303 are displayed as preview images. Details will be described later, but the saved images in the auxiliary storage device 303 are the photographed images acquired by the camera 306 that are stored by being determined to satisfy a predetermined determination condition described later. However, the saved images 741, 742, and 743 displayed in the saved image selection UI area 704 are preview images in which a product image is synthesized in order to clearly show the pose of the user when the photographed image is saved as a saved image. In addition, if there are undisplayed saved images that cannot be displayed in the saved image selection UI area 704, for example, up and down scroll buttons 745 are also displayed in the saved image selection UI area 704. When the up / down scroll button 745 is operated, a preview image of an undisplayed saved image is displayed in the saved image selection UI area 704. User 2 can select a desired saved image from among the multiple saved images displayed in the saved image selection UI area 704 by operating the touch panel 308.

[0029] In the playback mode, the image display area 701 displays a composite image in which a saved image selected from the saved image selection UI area 704 is combined with a product image corresponding to a product icon selected from the product selection UI area 702. Also, in the case of playback mode, the mode change button 733 is displayed as a button representing a change to real-time mode.

[0030] FIG. 6 is a flowchart showing the flow of operations in the virtual try-on processing section 6 of the mobile terminal 3 shown in FIG. As part of the startup process of step S601, the status management unit 9 of the virtual try-on processing unit 6 reads out and retains the product information list sent from the server 5 together with the virtual try-on program and stored in the auxiliary storage device 303 as described above. Furthermore, the status management unit 9 sends screen layout information of the virtual try-on UI 7 corresponding to the operation mode and the product information list to the display processing unit 10. The display processing unit 10 renders the screen layout information and drawing data of each product in the product information list to generate a display image for the virtual try-on UI 7, and displays it on the display 307. Note that the operation mode immediately after startup of the virtual try-on program is set to the real-time mode as described above.

[0031] Next, in the process of step S602, the image processing unit 8 of the virtual try-on processing unit 6 acquires a captured image from the camera 306. In this embodiment, since an example is given of the user 2 virtually trying on a wristwatch, the user 2 will capture an image of his or her own hand (particularly from the hand to the elbow, etc.) on which the wristwatch is virtually tried on, with the camera 306. Therefore, the image processing unit 8 acquires a captured image of the user 2's hand from the camera 306.

[0032] Next, in step S603, the image processing unit 8 executes a predetermined image processing on the captured image acquired from the camera 306. At this time, the image processing unit 8 also receives information indicating the operation mode and a product information list from the state management unit 9. The operation of the image processing unit 8 differs depending on the operation mode.

[0033] When the operation mode is the real-time mode, the image processing unit 8 performs each process such as image recognition process, product synthesis position estimation process, save determination process, image saving process, and product synthesis process as a predetermined image process. The image recognition process is a process of recognizing a subject included in a captured image, and outputs the recognition result. The product synthesis position estimation process is a process of estimating a product synthesis position where a product image is synthesized with respect to a subject included in a captured image. In the example of this embodiment, in the product synthesis position estimation process, the area of ​​the hand of the user 2 is detected from the captured image, and the product synthesis position is estimated based on the area of ​​the hand. The save determination process is a process of determining whether or not a subject included in a captured image satisfies a predetermined judgment condition, and in this embodiment, whether or not a product synthesis position estimated by the product synthesis position estimation process satisfies a predetermined judgment condition. The image saving process is a process of saving the captured image in the auxiliary storage device 303 as a saved image to be used in the playback mode when it is determined in the save determination process that a subject included in a captured image satisfies a predetermined judgment condition. The product synthesis process is a process of generating a product image based on a three-dimensional product shape model of the product corresponding to the product icon selected by the user 2 from the product selection UI area 702, and generating a synthetic image by synthesizing the product image at the product synthesis position of the subject in the captured image. Each of these processes will be described in detail later.

[0034] On the other hand, when the operation mode is the playback mode, the image processing unit 8 performs a saved image acquisition process and a product synthesis process. The saved image acquisition process is a process for acquiring a saved image selected by, for example, user 2 via the saved image selection UI area 704 from among a plurality of saved images saved in the auxiliary storage device 303. The product synthesis process in the playback mode is a process for synthesizing a product image generated from a 3D product shape model of the product selected by user 2 in the product selection UI area 702 with the saved image selected by user 2 in the saved image acquisition process. Details of each of these processes will be described later. Whether the operation mode is the real-time mode or the playback mode, the composite image generated by the image processing unit 8 is sent to the display processing unit 10.

[0035] In step S604, the display processing unit 10 generates drawing data for the virtual try-on UI 7 based on the composite image input from the image processing unit 8 and the screen layout information and product information list corresponding to the operation mode received from the state management unit 9. Then, a display image based on the drawing data for the virtual try-on UI 7 is displayed on the display 307, that is, the virtual try-on UI 7 corresponding to the operation mode is displayed on the display 307. This allows the user 2 to look at the composite image displayed in the image display area 701 and consider whether to purchase the product. In addition, the user can perform inputs such as changing the product to be virtually tried on, switching the operation mode, and moving to a product purchase screen from the product selection UI area 702 and the navigator UI area 703.

[0036] Then, the state management unit 9 manages the operation of the image processing unit 8 and the display processing unit 10 as the process of step S605 in response to the input from the user via the touch panel 308. That is, the state management unit 9 performs processes such as switching the operation mode, changing the screen layout of the virtual try-on UI 7 according to the operation mode, changing the product, changing the image to be used in the composite image, transitioning to a product purchase screen, etc. In addition, in the case of the playback mode, as shown in FIG. 5(b), a saved image selection UI area 704 is also displayed on the screen of the virtual try-on UI 7. Therefore, in the playback mode, the user 2 can also select a saved image to be used in the composite image.

[0037] Thereafter, in the process of step S606, the state management unit 9 determines whether or not an end instruction has been detected from the user 2. An end instruction from the user 2 is generated when the end button 734 arranged in the above-mentioned product selection UI area 702 is touched, or when the purchase button 731 is touched. Then, if the state management unit 9 has not detected an end instruction, it returns the process to step S602, and if an end instruction has been detected, it proceeds to step S607.

[0038] When the process proceeds to step S607, the virtual try-on processing unit 6 ends the virtual try-on program. However, if the purchase button 731 is touched, the virtual try-on processing unit 6 ends the virtual try-on program and then transfers control to the web browser program. The central processing unit 301 of the mobile terminal 3 then displays on the display 307 a product purchase web page linked to the product selected when the purchase button 731 was touched.

[0039] Fig. 7 is a functional block diagram showing each functional unit included in the image processing unit 8 described above. As shown in Fig. 7, the image processing unit 8 includes each functional unit of an image recognition unit 801, an image storage unit 802, a switching unit 803, and a synthesis unit 804. Note that Fig. 7 also shows the configuration shown in Fig. 2 or Fig. 4 other than the image processing unit 8. Fig. 8 is also a flowchart showing a detailed operation flow of the image processing unit 8 in step S603 in Fig. 4. Hereinafter, the operation of each functional unit of the image processing unit 8 shown in Fig. 7 will be described with reference to the flowchart in Fig. 8.

[0040] As described above, the image processing unit 8 performs different processing operations depending on the operation mode. Therefore, in step S801, the image processing unit 8 first determines whether the operation mode is the real-time mode or the playback mode. If the operation mode is the real-time mode, the image processing unit 8 proceeds to step S802, whereas if the operation mode is the playback mode, the image processing unit 8 proceeds to step S807.

[0041] Since the operation mode is the real-time mode, when the process proceeds to step S802, the image recognition unit 801 of the image processing unit 8 performs a predetermined image recognition process using a predefined detection model on the captured image input from the camera 306. In the case of this embodiment, a virtual try-on of a wristwatch is taken as an example, and the user 2 photographs his / her hand (for example, the area from the hand to the elbow) with the camera 306, and image recognition process is performed to determine whether the hand is reflected in the captured image. For this reason, a detection model for detecting the area of ​​the hand from an image is used as the detection model predefined in the image recognition process.

[0042] In this embodiment, a hand detector is used as a detection model for the image recognition unit 801, in which an SSD (Single Shot Multibox Detector), which is an object detector using deep learning, is trained with images of human hands. Unlike face images, hand images often do not include high-contrast patterns, so when training the hand detector, images of the hand including the forearm and fingers as well as the wrist on which the wristwatch is worn are trained. The hand detector outputs the coordinates of the candidate region of the hand and the reliability indicating the likelihood of the detection. The reliability is a value normalized from 0 to 1. When the reliability is high, the detected coordinates are likely to be a hand, and conversely, when the reliability is low, there is a high possibility of a false detection. In this embodiment, 0.6 is set as a threshold value for the reliability. Then, when the reliability is smaller than the threshold, it is determined to be a false detection, and the product synthesis position is set to null. When the product synthesis position is null, the synthesis unit 804 described later does not synthesize the product image with the captured image.

[0043] Furthermore, in step S802, the image recognition unit 801 estimates a product composition position for the hand area detected using the detection model, using a predefined estimation model. Since a wristwatch is generally worn on the wrist, the image recognition unit 801 estimates a product composition position corresponding to the position of the wrist, using a predefined estimation model. That is, the process performed in step S802 is the product composition position estimation process described above.

[0044] In this embodiment, as an estimation model used by the image recognition unit 801, an estimator that has been trained to estimate the three-dimensional positions of landmarks set from a two-dimensional image of the hand using deep learning is used. The landmarks are the wrist, each part of the five fingers, and the elbow. The estimation result is expressed in a camera coordinate system in which the optical axis direction of the camera 306 is the Z axis, the horizontal direction of the image sensor of the camera 306 perpendicular to the two-dimensional plane parallel to the Z axis is the X axis, and the vertical direction of the image sensor is the Y axis. Note that the origin of the XY plane is the center of the image.

[0045] In this embodiment, the product synthesis position is information represented by one point and two vectors that define the coordinate system required for product synthesis. In the case of a wristwatch exemplified in this embodiment, one point defined in the coordinate system required for product synthesis is the position on the outside of the wrist (defined as the product synthesis origin) that comes into contact with the center position of the back of the case, which is the origin coordinate of the product. In addition, the two vectors defined in the coordinate system required for product synthesis are the normal vector of the plane that includes the product synthesis position (defined as the product synthesis surface normal vector), and a vector from the elbow to the wrist that defines the rotation direction of the product (defined as the product synthesis direction vector). Then, the image recognition unit 801 passes the captured image input from the camera 306 and the information on the product superimposition position obtained in the process of step S802 to the image storage unit 802 and the switching unit 803.

[0046] The image storage unit 802 caches the captured image and product synthesis position passed from the image recognition unit 801 for a predetermined period of time. In this embodiment, the cache time is set to 3 seconds. The cache is implemented with a first-in, first-out data structure, and the image storage unit 802 holds information on the captured image and product synthesis position from the current time up to a predetermined time ago. Furthermore, in step S803, the image storage unit 802 performs storage determination processing to determine whether or not to store the image captured by the camera 306, based on the product superimposition position estimated in step S802 from the image captured by the camera 306.

[0047] Here, in the save determination process for determining whether or not to save an image, at least two determination conditions, a first determination condition and a second determination condition, are used as the predetermined determination conditions. The first judgment condition is whether the product composition origin (the position on the outside of the wrist that comes into contact with the center position of the back of the watch case) estimated by the image recognition unit 801 is in the center vicinity area of ​​the captured image. For example, the image storage unit 802 divides the captured image vertically and horizontally into thirds, and if the product composition origin is in the central area of ​​the three vertically and horizontally divided areas, it judges that it is in the center vicinity area, that is, it judges that the first judgment condition is satisfied.

[0048] The second judgment condition is a condition that uses a line of sight deviation angle formed by the optical axis direction of the camera 306, which corresponds to the line of sight direction of the user 2, when the optical axis direction is assumed to be substantially parallel to the Z axis on the product synthesis origin, and one of the line of sight vectors of the product representative direction vectors. For example, the image storage unit 802 judges that the second judgment condition is satisfied when the angle (referred to as the line of sight deviation angle) formed by the optical axis direction of the camera 306 and the line of sight vector of the product representative direction vector is less than a preset angle threshold value.

[0049] The angle threshold for the gaze deviation angle may be a preset fixed value, or may be arbitrarily set by the user 2. The angle threshold for the gaze deviation angle may be set to a different value for each gaze vector included in the product representative direction vector. As an example, the angle threshold for the gaze deviation angle between the optical axis direction of the camera 306, which is approximately parallel to the Z axis, and the gaze vector representing the observation direction when the user 2 looks at the dial, may be 15 degrees, for example.

[0050] If the image storage unit 802 determines that both the first and second judgment conditions are met in the storage judgment process of step S803, it stores the currently cached captured image as a saved image in the auxiliary storage device 303. In other words, if both the first and second judgment conditions are met, the image storage unit 802 determines that the currently cached captured image, product composition position, and line of sight deviation angle are to be saved, and proceeds to step S804. Then, when proceeding to step S804, the image storage unit 802 saves the currently cached captured image, product composition position, and line of sight deviation angle in the auxiliary storage device 303.

[0051] Here, if a saved image already exists in the auxiliary storage device 303, the image storage unit 802 further performs a judgment process using a third judgment condition. The third judgment condition is a condition that uses the line of sight misalignment angle calculated for the captured image as described above and the line of sight misalignment angle calculated when the already existing saved image was captured. If a saved image already exists in the auxiliary storage device 303, the image storage unit 802 judges that the third judgment condition is satisfied if the line of sight misalignment angle calculated for the captured image is less than the line of sight misalignment angle calculated when the already existing saved image was captured. That is, if the first and second judgment conditions are satisfied in step S803 and the third judgment condition is also satisfied, the image storage unit 802 overwrites the saved image existing in the auxiliary storage device 303 with the currently cached captured image. Note that when overwriting the saved image with the captured image, the product synthesis position and the line of sight misalignment angle are also overwritten and saved.

[0052] If there are multiple product representative direction vectors, the image storage unit 802 performs the above-described storage determination process for each product representative direction vector and stores them in separate storage areas of the auxiliary storage device 303. After that, the image processing unit 8 advances the process to step S805.

[0053] On the other hand, if either the first or second judgment condition is not satisfied in step S803, the image storage unit 802 judges not to store the currently cached captured image, product composite position, and line of sight deviation angle. After that, the image processing unit 8 advances the process to step S805.

[0054] Furthermore, when the operation mode is the real-time mode, the switching unit 803 always outputs the captured image input via the image recognition unit 801 and the information on the product synthesis position input from the state management unit 9 to the synthesis unit 804. For this reason, in step S805, the synthesis unit 804 in the real-time mode performs synthesis processing to synthesize the captured image acquired by the camera 306 with a product image based on the three-dimensional product shape model of the product selected by the user 2. Then, this synthesized image is output to the display processing unit 10, and an image of the watch virtually worn on the wrist of the user 2 is displayed in the image display area 701 of the virtual try-on UI 7.

[0055] When it is determined in step S801 that the mode is the playback mode and the process proceeds to step S807, the state management unit 9 changes the image to be sent to the synthesis unit 804 to the saved image selected by the user. In more detail, the state management unit 9 reads out the saved image selected by the user 2 in the saved image selection UI area 704 from the auxiliary storage device 303 and passes it to the image storage unit 802. The image storage unit 802 then inputs the saved image to the switching unit 803. In the playback mode, the switching unit 803 outputs to the synthesis unit 804 the saved image input from the image storage unit 802, rather than the captured image input from the image recognition unit 801. After that, the image processing unit 8 advances the process to step S805.

[0056] In the playback mode, the synthesis unit 804 performs processing to synthesize the saved image input from the switching unit 803 with a product image based on the product synthesis position input from the state management unit 9, that is, a product image based on the 3D product shape model of the product selected by the user 2. In other words, in the playback mode, the synthesized image is an image in which the product image is synthesized with the saved image selected by the user 2 in the saved image selection UI area 704 from among the saved images stored in the auxiliary storage device 303.

[0057] In this embodiment, when the synthesis unit 804 synthesizes images, synthesis is performed so that the model coordinate system of the product shape model coincides with the product synthesis position estimated by the image recognition unit 801. In other words, the synthesis unit 804 aligns the origin position of the model coordinate system with the product synthesis origin, aligns the Z axis of the model coordinate system with the product synthesis surface normal vector, and aligns the Y axis of the model coordinate system with the product synthesis direction vector. The product image is subjected to hidden surface processing for the arms, and is rendered as an image seen from the position of the camera 306.

[0058] As described above, when the operation mode of the mobile terminal 3 of this embodiment is the real-time mode, the mobile terminal 3 presents the user with a composite image in which a product image is composited with an image captured by the camera 306. In addition, when the mobile terminal 3 is in the real-time mode, if a subject included in an image captured by the camera 306 satisfies a predetermined judgment condition, the mobile terminal 3 saves the captured image. On the other hand, when the operation mode is the playback mode, the mobile terminal 3 of this embodiment presents the user with a composite image in which a product image is composited with a saved image. Therefore, the mobile terminal 3 of this embodiment can reduce the workload of a user performing virtual fitting.

[0059] Here, when a user compares and considers multiple products while virtually trying on the watch, the user needs to repeatedly perform product selection operations and pose changes while operating a mobile terminal with one hand, which places a large workload on the user. For example, when selecting a watch, the user is expected to check the watch in multiple poses, such as a pose to look at the watch face, a pose to operate the crown, and a pose to look at the buckle side of the watch band, in order to determine whether the watch he or she has virtually tried on suits him or her. In order to perform such a checking operation for each of the multiple watches, it was necessary to repeat each of these poses as many times as the number of watches to be compared and considered. Conversely, when changing watches while fixing the pose, it is necessary to maintain the pose and perform the operation to change the product while comparing and considering.

[0060] In contrast, according to this embodiment, when the user 2 virtually tries on a wristwatch, a captured image that approximately matches the product representative direction vector of that product is saved. The product representative direction vector is set to a product observation direction that is useful for the user 2 to compare and consider products. When the user 2 virtually tries on the wristwatch and observes the wristwatch from various directions, if the product observation direction approximately matches one of the product representative direction vectors, an image of the pose at that time is automatically saved. Therefore, when the user 2 subsequently virtually tries on another wristwatch, the already saved image can be used as a playback image. Therefore, the user 2 only needs to perform the task of instructing to switch to the playback mode as the operation mode, and the user 2 does not need to repeatedly change the pose again, and can change both the product and the pose to compare and consider the wristwatch.

[0061] In this embodiment, a wristwatch is used as an example of a product, but of course the product is not limited to a wristwatch. Furthermore, it is possible to virtually try on products of different product types, such as rings. However, when the product type is changed, the product representative direction vector corresponding to the saved image used in the playback mode is not the product representative direction vector of the current product after the type change, so there is a possibility that the image will not allow the user to clearly grasp the product features. For this reason, when the product type is changed, the image saving unit 802 erases (clears) the saved image, or saves the image for each product type (for each product representative surface direction vector corresponding to the product type). In the above-described embodiment, the user himself / herself selects the saved image and the product, but these may be performed by, for example, a recommendation function or an AI function included in the virtual try-on program.

[0062] The present invention can also be realized by supplying a program that realizes one or more functions of the above-mentioned embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. The above-mentioned embodiments are merely examples of concrete implementations of the present invention, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0063] The disclosure of each embodiment includes the following configurations, methods, and programs. (Configuration 1) a storage means for storing the photographed image as a stored image when a subject included in the photographed image satisfies a predetermined determination condition; a selection means for selecting at least one object from among a plurality of objects; a synthesis means for synthesizing an image of the selected object with the subject included in the stored image; a display processing means for displaying a composite image synthesized by the synthesis means; 13. An information processing device comprising: (Configuration 2) a recognition means for recognizing the subject included in the captured image, 2. The information processing apparatus according to configuration 1, wherein the storage means determines whether or not a result of the subject recognition by the recognition means satisfies the predetermined determination condition. (Configuration 3) the recognition means acquires, as a result of the recognition of the subject, a synthesis position of the subject at which an image of the object is synthesized; The predetermined judgment conditions include a first judgment condition as to whether or not the synthesis position of the subject included in the captured image is within a region that has been set in advance for the captured image, and a second judgment condition as to whether or not an angle formed between an optical axis direction of a photographing device that has acquired the captured image and an observation direction that has been set in advance for the synthesis position of the subject is less than a predetermined angle threshold value, 3. The information processing apparatus according to configuration 2, wherein the storage means stores the captured image as a stored image when both the first and second determination conditions are satisfied. (Configuration 4) the predetermined judgment condition further includes a third judgment condition as to whether or not an angle between an optical axis direction of the photographing device and an observation direction that is set in advance with respect to a composition position of the subject is smaller than the angle formed when an already existing saved image was photographed; The information processing device according to configuration 3, wherein the storage means overwrites the already existing stored image with the captured image when the first judgment condition, the second judgment condition, and the third judgment condition are satisfied. (Configuration 5) a plurality of observation directions are set for the subject in advance with respect to the synthesis position of the subject; 5. The information processing device according to configuration 3 or 4, wherein the storage means sets one stored image corresponding to one observation direction. (Configuration 6) An information processing device described in any one of configurations 3 to 5, characterized in that the synthesis position includes a position where the subject touches a position of an origin that is preset on the object, a normal vector of a plane that includes the synthesis position, and a direction vector that determines a rotation direction. (Configuration 7) A plurality of types of the object are set, 7. The information processing device according to any one of configurations 1 to 6, wherein the storage means deletes the saved image before the type of the object was changed when the type of the object is changed. (Configuration 8) The information processing device according to any one of configurations 1 to 7, wherein the synthesis means, in a first mode, synthesizes an image of the selected object with the subject included in the captured image, and, in a second mode, synthesizes the image of the selected object with the subject included in the saved image. (Configuration 9) 9. The information processing apparatus according to configuration 8, further comprising a management means for managing switching between the first mode and the second mode based on an instruction from a user. (Configuration 10) 10. The information processing device according to configuration 8 or 9, further comprising an image selection means for selecting, from among the plurality of stored images, the stored image onto which the image of the object is to be composited in the second mode, based on an instruction from a user. (Configuration 11) the display processing means displays a user interface screen on which the plurality of stored images are arranged, 11. The information processing apparatus according to configuration 10, wherein the image selection means selects a saved image designated by a user from among the plurality of saved images displayed on the user interface screen. (Configuration 12) 12. The information processing apparatus according to configuration 11, wherein the user interface screen displays a preview image in which an image of the object is synthesized with the subject included in the saved image. (Configuration 13) the object is an object representing a product, 13. The information processing device according to any one of configurations 1 to 12, wherein the subject is a specific part of the user's body. (Method 1) a storing step of storing the photographed image as a stored image when a subject included in the photographed image satisfies a predetermined determination condition; a selection step of selecting at least one object from among a plurality of objects; a synthesis step of synthesizing an image of the selected object with the subject included in the stored image; a display processing step of displaying a composite image obtained by the composition step; 13. An information processing method comprising: (Program 1) A program that causes a computer to function as the information processing device according to any one of configurations 1 to 13. [Explanation of symbols]

[0064] 3: mobile terminal, 6: virtual fitting processing unit, 8: image processing unit, 9: state management unit, 10: display processing unit, 801: image recognition unit, 802: image storage unit, 803: switching unit, 804: synthesis unit

Claims

1. a storage means for storing the photographed image as a stored image when a subject included in the photographed image satisfies a predetermined determination condition; a selection means for selecting at least one object from among a plurality of objects; a synthesis means for synthesizing an image of the selected object with the subject included in the stored image; a display processing means for displaying a composite image synthesized by the synthesis means; 13. An information processing device comprising:

2. a recognition means for recognizing the subject included in the captured image, 2 . The information processing apparatus according to claim 1 , wherein the storage means determines whether or not the subject recognition result obtained by the recognition means satisfies the predetermined determination condition.

3. the recognition means acquires, as a result of the recognition of the subject, a synthesis position of the subject at which an image of the object is synthesized; The predetermined judgment conditions include a first judgment condition as to whether or not the synthesis position of the subject included in the captured image is within a region that has been set in advance for the captured image, and a second judgment condition as to whether or not an angle formed between an optical axis direction of a photographing device that has acquired the captured image and an observation direction that has been set in advance for the synthesis position of the subject is less than a predetermined angle threshold value, 3. The information processing apparatus according to claim 2, wherein the storage means stores the photographed image as a stored image when both the first and second determination conditions are satisfied.

4. the predetermined judgment condition further includes a third judgment condition as to whether or not an angle between an optical axis direction of the photographing device and an observation direction that is set in advance with respect to a composition position of the subject is smaller than the angle formed when an already existing saved image was photographed; 4. The information processing apparatus according to claim 3, wherein the storage means overwrites the already existing stored image with the captured image when the first, second and third determination conditions are satisfied.

5. a plurality of observation directions are set for the subject in advance with respect to the synthesis position of the subject; 4. The information processing apparatus according to claim 3, wherein said storage means sets one stored image corresponding to one observation direction.

6. 4. The information processing device according to claim 3, wherein the synthesis position includes a position where the subject is in contact with a position of an origin preset on the object, a normal vector of a plane including the synthesis position, and a direction vector that determines a rotation direction.

7. A plurality of types of the object are set, 2. The information processing apparatus according to claim 1, wherein said storage means deletes the stored image before the type of said object was changed when the type of said object is changed.

8. The information processing device according to claim 1, characterized in that, in a first mode, the combining means combines an image of the selected object with the subject included in the captured image, and, in a second mode, combines an image of the selected object with the subject included in the saved image.

9. 9. The information processing apparatus according to claim 8, further comprising a management unit that manages switching between the first mode and the second mode based on an instruction from a user.

10. 9. The information processing apparatus according to claim 8, further comprising image selection means for selecting, from among the plurality of stored images, the stored image with which the image of the object is to be composited in the second mode, based on an instruction from a user.

11. the display processing means displays a user interface screen on which the plurality of stored images are arranged, 11. The information processing apparatus according to claim 10, wherein the image selection means selects a saved image designated by a user from among the plurality of saved images displayed on the user interface screen.

12. 12. The information processing apparatus according to claim 11, wherein a preview image in which an image of the object is composited with the subject included in the saved image is displayed on the user interface screen.

13. the object is an object representing a product, 13. The information processing apparatus according to claim 1, wherein the subject is a specific part of a user's body.

14. a storing step of storing the photographed image as a stored image when a subject included in the photographed image satisfies a predetermined determination condition; a selection step of selecting at least one object from among a plurality of objects; a synthesis step of synthesizing an image of the selected object with the subject included in the stored image; a display processing step of displaying a composite image obtained by the composition step; 13. An information processing method comprising:

15. Computer, a storage means for storing the photographed image as a stored image when a subject included in the photographed image satisfies a predetermined determination condition; a selection means for selecting at least one object from among a plurality of objects; a synthesis means for synthesizing an image of the selected object with the subject included in the stored image; a display processing means for displaying a composite image synthesized by the synthesis means; A program that causes the device to function as an information processing device having the above-mentioned configuration.

Images