Image processing device, method for controlling the same, and program

Abstract

To provide an image processing device capable of receiving a user input in a more suitable mode even under a situation where operation is limited to one hand, a method for controlling the same, and a program.SOLUTION: A terminal device 2 includes a screen display unit 205, an input receiving unit 206, and an imaging unit 207. An image processing device 1 includes a holding unit 401, a display control unit 402, a selection processing unit 403, a target portion acquisition unit 404, and a switching unit 405. The switching unit 405 switches, in accordance with a type of a virtual object to be displayed in a display area, a method of a user input via the terminal device 2 regarding selection of the virtual object to be displayed in the display area. The display control unit 402 superimposes an image of the selected virtual object on a captured image and displays it in the display area.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, a control method for an image processing device, and a program. [Background technology]

[0002] In recent years, various services using Augmented Reality (AR) technology have been proposed, and one example of such a service is a virtual try-on service. In the virtual try-on service, images of clothing and accessories are superimposed on an image of the user captured by an imaging device, allowing the user to experience trying on the clothing. The virtual try-on service can reduce the effort and time required for actually trying on clothes, and is therefore effective when checking the impression and size of a product from among many options, and smartphone applications for implementing the service are also provided. In addition, in a situation where a service using AR technology is realized by a portable terminal device such as a smartphone, the terminal device may be operated with one hand. In such a situation where the terminal device is operated with one hand, depending on the UI for receiving input from the user, input operation may be difficult. Patent Documents 1 and 2 disclose techniques for assisting input operation by a user, assuming a situation where the operation is performed with one hand. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6259169 [Patent Document 2] JP 2009-169820 A Summary of the Invention [Problem to be solved by the invention]

[0004] On the other hand, in the conventional technology, in order to assist the user in inputting operations when performing operations with one hand, the screen layout may be changed to accommodate the operation with the one hand. However, in a situation where a service using the above-mentioned AR technology is provided, changing the screen layout may not be desirable.

[0005] In view of the above problems, an object of the present invention is to make it possible to accept user input in a more suitable manner even in a situation where operation is restricted to one hand. [Means for solving the problem]

[0006] The image processing device according to the present invention is characterized in having a switching means for switching a user input method for selecting a virtual object to be displayed in a display area depending on the type of virtual object to be displayed in the display area, and a display control means for superimposing an image of the selected virtual object on a captured image and displaying it in the display area. Effect of the Invention

[0007] According to the present invention, it is possible to accept user input in a more suitable manner even in a situation where operation is restricted to one hand. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of an image processing apparatus. [Diagram 2] FIG. 2 is a schematic external view of a terminal device. [Diagram 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a terminal device. [Figure 4] FIG. 2 is a diagram illustrating an example of a functional configuration of an image processing device and a terminal device. [Diagram 5] 11 is a flowchart showing an example of processing by an image processing unit and a terminal device. [Figure 6] FIG. 13 is a diagram showing an example of an operation screen. [Figure 7]13 is a flowchart showing an example of processing by the image processing unit. [Figure 8] FIG. 13 is a diagram showing an example of an operation screen. [Figure 9] FIG. 13 is a diagram showing an example of an operation screen. [Figure 10] 13 is a flowchart showing an example of processing by the image processing unit. [Figure 11] FIG. 2 is a diagram illustrating an example of a functional configuration of an image processing device and a terminal device. [Figure 12] 11 is a flowchart showing an example of processing by an image processing unit and a terminal device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0010] <First embodiment> As a first embodiment of the present disclosure, an example of a technique for switching a user input method to a more suitable mode depending on the situation at the time when an image in which a virtual object is superimposed on a subject recognized from a captured image using AR technology is displayed will be described. Note that, for convenience, in this embodiment, various explanations will be given focusing on a case where a so-called try-on viewing service is provided in which a superimposed image in which an image of a user's body part is captured as a subject is displayed by superimposing an image of an accessory (corresponding to a virtual object) based on AR technology.

[0011] (Hardware configuration of image processing device) An example of a hardware configuration of an image processing device 1 according to this embodiment will be described with reference to Fig. 1. The image processing device 1 includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, and a RAM (Random Access Memory) 103. The image processing device 1 also includes an HDD (Hard Disk Drive) 104 and a NIC (Network Interface Card) 105. The components of the image processing device 1 described above are connected to each other via a system bus 106 so as to be able to transmit and receive information to and from each other.

[0012] The CPU 101 uses the RAM 103 as a working memory to execute an OS (Operating System) and various programs stored in the ROM 102, HDD 104, etc. The CPU 101 also controls the operation of each component via a system bus 106. The CPU 101 uses various recording media mounted on the HDD 104 as storage locations for various data. The NIC 105 is connected to a network such as the Internet, and transmits and receives information to and from an external device. The CPU 101 may present a UI (User Interface) provided by a program to a user via the external device by transmitting the UI to the external device via the NIC 105. The CPU 101 may also receive input such as an instruction received from a user by the UI presented via the external device from the external device via the NIC 105. The functional configuration described later with reference to Figures 4 and 11, and the processing described later with reference to Figures 5, 7, 10, and 12 are realized by program code stored in ROM 102, HDD 104, etc. being expanded into RAM 103 and executed by CPU 101.

[0013] (Terminal device hardware configuration) An example of the hardware configuration of the terminal device 2 used for trying on clothes will be described with reference to FIGS. Fig. 2 is a schematic external view of the terminal device 2. Specifically, Fig. 2(a) shows an example of a schematic configuration of the front side of the terminal device 2, that is, the side facing the user when the user operates the terminal device 2. Fig. 2(b) shows an example of a schematic configuration of the back side of the terminal device 2. The housing 201 is a schematic representation of the housing of the terminal device 2. The terminal device 2 has a display 202 and an imaging device 203. The display 202 is, for example, a liquid crystal or organic EL panel, and has a touch panel function. The imaging device 203 is, for example, an imaging unit composed of a CMOS (Complementary Metal Oxide Semiconductor) image sensor and optical members such as a lens. In this embodiment, the housing 201 of the terminal device 2 is formed in a plate shape, and the imaging device 203 is provided on the surface (rear surface) opposite to the surface (front surface) on which the display 202 of the housing 201 is arranged, so that various explanations will be given focusing on the case of a so-called out-camera type. On the other hand, the configuration shown in FIG. 2 is merely an example, and does not necessarily limit the configuration of the terminal device 2 according to this embodiment. As a specific example, a so-called in-camera type may be applied in which an imaging device is provided on the surface on the side on which the display is provided.

[0014] Fig. 3 is a diagram showing an example of a hardware configuration of the terminal device 2. The terminal device 2 includes a CPU 301, a ROM 302, a RAM 303, a HDD 304, and a NIC 305. As described above with reference to Fig. 2, the terminal device 2 also includes a housing 201, a display 202, and an imaging device 203. The above-mentioned components of the terminal device 2 are connected to each other via a system bus 306 so as to be able to transmit and receive information to and from each other. The CPU 301 uses the RAM 303 as a working memory and executes the OS and various programs stored in the ROM 302, HDD 304, etc. The CPU 301 also controls the operation of each component via a system bus 306. The CPU 101 uses various recording media mounted on the HDD 304 and recording media on a network such as the Internet as storage locations for various data. The NIC 305 is connected to a network such as the Internet, and transmits and receives information to and from external devices. The CPU 301 displays a UI based on the execution result of a program, a UI acquired via a network, and the like on the display 202, and acquires input information such as instructions received from a user via the touch panel of the display 202.

[0015] (Functional configuration of image processing device and terminal device) An example of the functional configuration of the image processing device 1 and the terminal device 2 according to this embodiment will be described with reference to Fig. 4. The terminal device 2 includes a screen display unit 205, an input receiving unit 206, and an imaging unit 207. The image processing device 1 includes a storage unit 401, a display control unit 402, a selection processing unit 403, a target part acquisition unit 404, and a switching unit 405.

[0016] The screen display unit 205 presents various information to a user by displaying the information as images on the display 202 of the terminal device 2. As a specific example, the screen display unit 205 may display images on the display 202 under the control of the display control unit 402 of the image processing device 1.

[0017] The input acceptance unit 206 acquires an input from a user via a touch panel provided on the display 202 of the terminal device 2. For example, the input acceptance unit 206 acquires the type of input method, such as a tap operation or a swipe operation on the screen of the display 202, and information indicating the content of the input operation (for example, a position on the screen and an operation direction, etc.). As a specific example, when a tap operation is performed on the screen, the input acceptance unit 206 may acquire information indicating the position of the screen that is the target of the tap operation as information indicating the content of the input operation. As another example, when a swipe operation is performed on the screen, the input acceptance unit 206 may acquire information indicating the position of the screen that is the target of the swipe operation and information indicating the swipe direction as information indicating the content of the input operation. Of course, the above is merely an example, and the type of operation to be accepted is not particularly limited, and the information indicating the content of the operation to be acquired may be appropriately changed according to the type of the target operation.

[0018] The imaging unit 207 acquires an image (hereinafter also referred to as a captured image) corresponding to the imaging result by the imaging device 203 included in the terminal device 2. As a specific example, the imaging unit 207 may read two-dimensional RGB captured images captured at a predetermined interval (for example, 1 / 30 second intervals) and record the captured images in the RAM 303. The captured images may be a plurality of still images captured continuously, or may be a moving image. The imaging unit 207 extracts images one by one if they are still images, or one frame by frame if they are moving images, and handles the images as captured images.

[0019] The storage unit 401 stores information on a virtual object to be superimposed on a captured image. As a specific example, the storage unit 401 may store information on a product in a fitting viewing service that uses AR technology to superimpose an image of the product, such as clothing, on a body part recognized as a subject from a captured image, thereby virtually simulating trying on the product. In this case, the product information may include information such as the product name, a CG model used for the virtual fitting, the part of the body on which the product is to be tried on, and the product category (type) such as a bag or shoes. The above information stored in the storage unit 401 corresponds to an example of setting information that associates a virtual object with a body part on which the virtual object is to be superimposed.

[0020] The display control unit 402 displays various information in a display area (e.g., the display 202) of the terminal device 2, thereby presenting the information to the user. For example, the display control unit 402 may display, on the display 202 of the terminal device 2, a selection screen for receiving instructions from the user regarding product selection when providing a try-on viewing service, or an image in which an elegant image is superimposed on an image captured by the imaging device 203. The display control unit 402 may also execute processing related to changing a virtual object (e.g., an image of a product) to be superimposed on a captured image using AR technology, ending a service using AR technology (e.g., a try-on viewing service), and the like.

[0021] The selection processing unit 403 selects a virtual object to be superimposed on the captured image based on an instruction input by a user via the terminal device 2. For example, when providing a fitting viewing service, the selection processing unit 403 may select an image of a product to be superimposed on a body part recognized as a subject from the captured image.

[0022] The target part acquisition unit 404 acquires information on the target on which the virtual object is to be superimposed. For example, when providing a fitting viewing service, the target part acquisition unit 404 may acquire information on the fitting part associated with the image of the product selected by the selection processing unit 403 from the product information stored in the storage unit 401. This identifies the target on which the virtual object is to be superimposed, for example, the fitting part on which the image of the product is to be superimposed.

[0023] The switching unit 405 switches the input method for receiving a predetermined instruction (user input) from the user, based on the information acquired by the target body part acquisition unit 404. For example, when the information on the try-on body part acquired by the target body part acquisition unit 404 indicates a hand or an arm, the switching unit 405 may recognize that an input operation via the terminal device 2 is performed with one hand. In this case, the switching unit 405 may switch the input method for receiving a predetermined instruction from the user (e.g., an instruction related to selecting an image of a product) to an input method that is easier for the user to input in a situation where the input operation is performed with one hand. As a specific example, when the default input method is an input method for selecting a product image by a tap operation, the switching unit 405 may switch the input method to an input method for selecting a product image by a swipe operation when the input operation is performed with one hand. In this way, the switching unit 405 switches the user input method for selecting a virtual object to be displayed in a predetermined display area according to the type of virtual object to be displayed in the display area (e.g., a part to be superimposed). In the above example, the default input method corresponds to an example of a first input method, and the input method switched to when an input operation is performed with one hand corresponds to an example of a second input method.

[0024] An example of the functional configuration of the image processing device 1 and the terminal device 2 according to this embodiment has been described above with reference to Fig. 4. However, the above-mentioned configuration is merely an example, and as long as it is possible to realize the functions of the respective components of the image processing device 1 and the terminal device 2, the configuration for achieving this is not necessarily limited to the configuration shown in Fig. 4. As a specific example, at least some of the components of the series of components of the image processing device 1 may be realized as components on the terminal device 2 side.

[0025] (Image processing device and terminal device processing) An example of the processing of the image processing device 1 and the terminal device 2 according to this embodiment will be described with reference to Fig. 5. Note that the series of processing shown in Fig. 5 is started by the image processing device 1 in response to an instruction from a user input via a touch panel provided on the display 202 of the terminal device 2, for example.

[0026] In S401, the display control unit 402 creates a screen on which a list of products is presented based on product information stored in the storage unit 401, and transmits the screen to the screen display unit 205 of the terminal device 2. The screen display unit 205 displays the screen transmitted from the display control unit 402 as a selection screen on the display 202, thereby presenting the selection screen to the user. For example, Fig. 6 is a diagram showing an example of a selection screen presented to a user via the display 202 of the terminal device 2. In the selection screen shown in Fig. 6, icons of images of products to be selected are presented as a list for each type of product. Note that the selection screen shown in Fig. 6 is merely an example, and does not necessarily limit the selection screen presented to a user in the process of S401. As a specific example, instead of an icon corresponding to each product, information indicating the product name, a thumbnail image of the product, or the like may be presented.

[0027] In S402, the selection processing unit 403 selects products (virtual objects) to be virtually tried on using the try-on browsing service in response to a user instruction input via a touch panel provided on the display 202 of the terminal device 2. As a specific example, an instruction from the user to select products to be tried on is accepted by a touch operation (e.g., a tap operation) on an icon associated with each product on the selection screen illustrated in FIG.

[0028] In S403, the target body part acquisition unit 404 acquires information on the body part to be tried on of the product selected by the selection processing unit 403 by referring to the information on the product stored in the storage unit 401. The body part to be tried on of the product indicates the body part on which the target product is to be worn, such as the hand, arm, leg, ear, head, etc. As a specific example, if the product to be tried on is a hat, the body part to be tried on of the product is the head. Similarly, if the product to be tried on is a ring, the body part to be tried on of the product is the hand. Moreover, if the product to be tried on is shoes, the body part to be tried on of the product is the foot.

[0029] In S404, the switching unit 405 determines whether the try-on area indicated by the information acquired by the target area acquisition unit 404 in S403 is a hand or an arm (in other words, whether the try-on area is a part that may be restricted to one-handed operation when using the try-on viewing service). If the switching unit 405 determines in S404 that the part to be tried on is the hand or arm, the processing proceeds to S405. In S405, the switching unit 405 switches the user input method applied in the virtual trying on of clothes using the try-on browsing service to an input method using a swipe operation. The processing of S405 will be described in detail later with reference to FIG. If the switching unit 405 determines in S404 that the part to be tried on is not a hand or an arm, the processing proceeds to S406. In S406, the switching unit 405 switches the user input method applied in the virtual trying on of clothes using the try-on browsing service to an input method using icon operations. The processing of S406 will be described in detail later with reference to FIG.

[0030] (Virtual try-on processing by swiping) With reference to FIG. 7, an example of processing related to realizing a virtual try-on using a try-on browsing service in the case where a swipe operation is applied as an input method, which is shown as the processing of S405 in FIG. 4, will be described.

[0031] In S4051, the display control unit 402 causes the display 202 of the terminal device 2 to display an initial screen (operation screen) related to use of the try-on browsing service. For example, FIG. 8 is a diagram showing an example of an operation screen related to the use of the try-on viewing service. The operation screen shown in FIG. 8 includes a display area 801 and an input area 802. The display area 801 is an area where an image captured by the imaging unit 207 of the terminal device 2 is displayed. The input area 802 is an input interface for receiving an instruction from a user related to the selection of a product (virtual object) to be virtually tried on using the try-on viewing service. In the example shown in FIG. 8, icons indicating a series of candidates for products to be virtually tried on are displayed side by side in the input area 802. Also, in the example shown in FIG. 8, an icon corresponding to the selected product candidate is arranged so as to be located at the center of the input area 802, and icons corresponding to the other candidates are arranged on the left and right of it.

[0032] In S4052, the display control unit 402 displays a UI for prompting the user to perform a swipe operation on the operation screen displayed on the display 202 of the terminal device 2 in S4051. For example, Fig. 9 is a diagram showing an example of a UI for prompting the user to perform a swipe operation, which is additionally displayed on the operation screen. Display information 803 shows an example of a UI additionally displayed to prompt the user to perform a swipe operation to the left or right.

[0033] Thereafter, the display control unit 402 repeatedly executes a series of processes from S4054 to S4057 indicated by the loop end S4053 until a termination condition based on the determination in S4056 is satisfied. Each of the processes from S4054 to S4057 will be described in detail below.

[0034] In S4054, the display control unit 402 generates an image (hereinafter also referred to as a superimposed image) in which an image of a product (virtual object) is superimposed on an image captured by the imaging unit 207 of the terminal device 2, and displays the superimposed image in the display area 801 on the operation screen.

[0035] Specifically, the display control unit 402 determines whether or not the fitting part is captured as a subject in the captured image, and if the fitting part is captured, calculates the size, position, and orientation of the fitting part in the captured image. Known techniques such as object recognition using deep learning can be applied to recognize the fitting part in the captured image and calculate the size, position, and orientation of the fitting part. Next, the display control unit 402 acquires a CG model of the target product stored in the storage unit 401, rotates the CG model according to the recognized posture of the trying-on part, and performs a transformation process of enlarging or reducing the CG model according to the size of the trying-on part. The display control unit 402 also generates a compositing image to be superimposed on the captured image by arranging the CG model according to the position of the trying-on part in the captured image. Then, the display control unit 402 superimposes the generated compositing image on the captured image to generate a superimposed image in which the image of the product is superimposed on the trying-on part captured as a subject (in other words, a superimposed image simulating trying on the target product). If the display control unit 402 determines that the try-on body part is not captured as a subject in the captured image, it treats the captured image as a superimposed image (that is, an image to be displayed in the display area 801). Then, the display control unit 402 displays the created superimposed image in the display area 801 of the operation screen.

[0036] In S4055, the display control unit 402 acquires information indicating the content of an operation on the operation screen that is accepted from the user by the input accepting unit 206 of the terminal device 2. The information includes, for example, information indicating the type of accepted operation (for example, a swipe operation, etc.) and the position (position on the screen) at which the operation is accepted. In addition, when the type of operation is an operation in which the meaning of the operation (for example, the instruction content from the user) differs depending on the operation direction, such as a swipe operation, the information includes information indicating the operation direction (for example, the swipe direction).

[0037] In S4056, the display control unit 402 determines whether the input from the user received in S4055 is an instruction to end use of the try-on viewing service. As a specific example, the display control unit 402 may determine that an instruction to end use of the try-on viewing service has been given when a button related to the reception of an instruction to return to the previous screen or a button related to the reception of an instruction to end the try-on viewing service is pressed. If the display control unit 402 determines in S4056 that the input from the user is an instruction to end use of the try-on browsing service, it ends the series of processes shown in FIG. 7 (i.e., the process of S405 shown in FIG. 4). On the other hand, if the display control unit 402 determines in S4056 that the input from the user is not an instruction to end use of the try-on viewing service, the display control unit 402 advances the process to S4057.

[0038] In S4057, the display control unit 402 changes the product to be virtually tried on (in other words, the virtual object) in response to the input from the user accepted in S4055. As a specific example, when an operation of swiping the input area 802 to the right is accepted, the display control unit 402 selects, as a new product, the product associated with the icon located to the right of the icon corresponding to the product currently selected as the product to be tried on. As another example, when an operation of swiping the input area 802 to the left is accepted, the display control unit 402 selects, as a new product, the product associated with the icon located to the left of the icon corresponding to the product currently selected as the product to be tried on. In this manner, the display control unit 402 switches the image of the product according to the user input successively received by the applied user input method (swipe operation), and displays a superimposed image in which the image of the product is superimposed on the trying-on area in the display area 801 of the operation screen.

[0039] (Virtual fitting process using icon operations) An example of a process for realizing a virtual try-on using a try-on browsing service when icon operation (tap operation on an icon) is applied as an input method, as shown in the process of S406 in Fig. 4, will be described with reference to Fig. 10. Note that the processes of S4051, S4054, S4055, and S4056 are substantially similar to the processes of S4051, S4054, S4055, and S4056 described with reference to Fig. 7, and therefore detailed description thereof will be omitted.

[0040] The display control unit 402 repeatedly executes a series of processes from S4054 to S4056 and S4062 indicated by a loop end S4061 until a termination condition based on the determination in S4056 is satisfied. In S4062, the display control unit 402 changes the product to be virtually tried on (in other words, the virtual object) in response to the input from the user accepted in S4055. Specifically, the display control unit 402 selects, as the product to be changed, the product associated with the icon arranged at the position on the operation screen where the tap operation from the user is accepted. In this manner, the display control unit 402 switches the image of the product according to the user input successively received by the applied user input method (icon layer operation), and displays a superimposed image in which the image of the product is superimposed on the trying-on area in the display area 801 of the operation screen.

[0041] (Action and effect) As described above, the image processing device 1 according to the present embodiment switches the method of user input for selecting a virtual object to be displayed in the display area 801, depending on the type of virtual object to be displayed in the display area 801 (for example, a subject associated as a superimposition target). Specifically, when the try-on body part associated with the product (virtual object) selected by the user is the hand or arm, the image processing device 1 switches the method of user input for changing the product to be tried on thereafter to a method using a swipe operation. By applying such control, even in a situation where operation is limited to one hand, the user can select the product to be tried on with a simple operation without changing the layout of the operation screen.

[0042] <Second embodiment> A second embodiment of the present disclosure will be described below. In the above-mentioned first embodiment, an example of a case where a user input method for selecting a virtual object is switched depending on a virtual object (for example, a product to be virtually tried on) selected based on a user input is described. In contrast, in this embodiment, an example of a case where a user input method for selecting a virtual object is switched depending on a subject recognized from a captured image is described. Note that, for convenience, in this embodiment as well, various explanations will be given focusing on a case where a try-on viewing service is provided in which a superimposed image in which an image of clothing or the like is superimposed based on AR technology on a captured image in which a part of the user's body is captured as a subject is displayed. Specifically, in this embodiment, a recognition process is performed on the captured image to recognize the body part captured as the subject, i.e., the body part to be virtually tried on. Then, based on the recognition result of the body part to be virtually tried on, it is determined whether or not the situation is one in which operation is to be performed with one hand, and the method of user input is switched according to the result of the determination. Hereinafter, this embodiment will be described in more detail, focusing particularly on the parts that are different from the first embodiment, and detailed descriptions of parts that are substantially similar to the first embodiment will be omitted.

[0043] (Functional configuration of image processing device and terminal device) An example of the functional configuration of the image processing device 1 and the terminal device 2 according to this embodiment will be described with reference to Fig. 11. The terminal device 2 includes a screen display unit 205, an input reception unit 206, and an imaging unit 207. The image processing device 1 includes a storage unit 401, a display control unit 402, an acquisition unit 1102, and a switching unit 1103. The screen display unit 205, the input reception unit 206, the imaging unit 207, the storage unit 401, and the display control unit 402 are substantially similar to the screen display unit 205, the input reception unit 206, the imaging unit 207, the storage unit 401, and the display control unit 402 in the example shown in Fig. 4.

[0044] The acquisition unit 1102 acquires an image captured by the imaging unit 207 of the terminal device 2 . The switching unit 1103 performs a recognition process on the captured image acquired by the acquisition unit 1102, and switches the input method for receiving a predetermined instruction (user input) from the user according to the subject identified by the recognition process. Specifically, when the switching unit 1103 identifies the subject as a hand or an arm based on the recognition result of the subject from the captured image, it may recognize that an input operation via the terminal device 2 is performed with one hand. In this case, the switching unit 1103 may switch the input method for receiving a predetermined instruction from the user (e.g., an instruction related to selecting an image of a product) to an input method that is easier for the user to input under a situation where an input operation is performed with one hand. As a specific example, the switching unit 1103 may switch to an input method for selecting an image of a product by a swipe operation as in the first embodiment. In this way, the switching unit 1103 switches the method of user input related to the selection of a virtual object to be displayed in a predetermined display area according to the type of the virtual object to be displayed in the display area (e.g., according to a virtual object associated with a recognized object).

[0045] (Image processing device and terminal device processing) An example of the processing of the image processing device 1 and the terminal device 2 according to this embodiment will be described with reference to Fig. 12. Note that the processing of S401, S402, S404, S405, and S406 is substantially similar to the processing of S401, S402, S404, S405, and S406 in the example shown in Fig. 5.

[0046] In S1201 , the acquisition unit 1102 acquires an image captured by the imaging unit 207 of the terminal device 2 . In S1202, the switching unit 1103 recognizes the subject captured in the captured image by performing a recognition process on the captured image acquired in S1201. Note that the recognition process of the subject in the captured image can be performed using a known technique such as object recognition using deep learning. In this way, the switching unit 1103 recognizes which part of the body the subject in the captured image is, and sets that part as the part to be tried on. From this point on, similarly to the first embodiment described above, the method of user input applied during virtual trying on of clothes using the try-on viewing service is switched depending on the result of recognition of the parts to be tried on. (Action and effect) As described above, the image processing device 1 according to this embodiment switches the method of user input for selecting a virtual object to be displayed in the display area 801 according to the type of the virtual object to be displayed in the display area 801 (for example, a subject associated as a superimposed subject). Specifically, the image processing device 1 recognizes a body part captured as a subject by performing a recognition process on a captured image, and switches the method of user input based on the recognition result of the body part. This makes it possible to determine whether or not a situation is to be operated with one hand even in a situation where, for example, information on a part to be tried on of a product (virtual object) to be virtually tried on is not held. Therefore, as in the first embodiment, even in a situation where operation is limited to one hand, the user can select a product to be tried on by a simple operation without changing the layout of the operation screen.

[0047] <Modification> A modified example of an embodiment of the present disclosure will be described below. In each of the above-described embodiments, an example has been described in which the terminal device 2, which is different from the image processing device 1, is responsible for accepting input from the user and presenting information to the user. On the other hand, the image processing device 1 and the terminal device 2 may be integrally configured. In other words, the image processing device 1 may be provided with a component for accepting input from the user and presenting information to the user.

[0048] In the above-described embodiment, the product information stored in the storage unit 401 includes information such as the product name, the CG model used for virtual trying on, the part of the body where the product is to be tried on, the product category such as bags and shoes, etc., but other information may also be included. As a specific example, the product information may include more detailed information about the target product such as the size and material of the product.

[0049] In the above-described embodiment, as explained with reference to FIG. 2, an example of a case where an image corresponding to the imaging result by the outer camera of the terminal device 2 is used is described, but an image corresponding to the imaging result by the inner camera may also be used.

[0050] In the above embodiment, an example of a case where a tap operation on a product icon is applied as a user input method for selecting a product on the operation screen displayed as the initial screen has been described, but other methods may be applied. As a specific example, a method of selecting a target product by manually inputting a product name or a method of selecting a product by a swipe operation may be applied.

[0051] In the above embodiment, an example of a case where the try-on part is a hand or an arm is applied as a criterion for determining when switching the direction of the user input has been described, but the try-on part may be a part included in the hand or the arm as a criterion for the same determination. As a specific example, the recognition result of the fingers or nails may be used as a criterion for determining that the try-on part is a hand. As another example, the recognition result of the wrist may be used as a criterion for determining that the try-on part is an arm.

[0052] The arrangement of the icons of the products to be virtually tried on and displayed in the input area 802, as described with reference to Fig. 8, is merely an example, and the arrangement of the icons is not limited to this. That is, the position of the icon associated with the selected product candidate does not necessarily have to be at the center of the input area 802. Furthermore, the direction in which the icons associated with the other candidates are arranged is not limited to left and right. It goes without saying that the direction of the swipe operation for selecting a product may be changed depending on the arrangement direction of the icons.

[0053] The method of recognizing a subject in a captured image (e.g., a body part to be virtually tried on) is not limited to a method based on the features of the subject in the image, and other methods may be applied. As a specific example, a user may wear a marker and the body part to be virtually tried on (in other words, a position to superimpose a virtual object) may be recognized by detecting the marker.

[0054] In the above embodiment, an example of a case where a swipe operation method is applied as a user input method applied when the fitting part is an arm or a hand has been described, but other types of operation methods may be applied. As a specific example, a tap operation method on the operation screen may be applied as a user input method applied when the fitting part is an arm or a hand. In this case, a UI that prompts the user to perform a tap operation may be additionally displayed on the operation screen.

[0055] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0056] The disclosure of the present embodiment also includes the following configurations, methods, and programs. (Configuration 1) An image processing device characterized by having a switching means for switching a user input method for selecting a virtual object to be displayed in a display area depending on the type of virtual object to be displayed in the display area, and a display control means for superimposing an image of the selected virtual object on a captured image and displaying it in the display area. (Configuration 2) The image processing device according to configuration 1, wherein the switching means switches a method of user input related to subsequent selection of a virtual object in accordance with a virtual object selected based on a user input. (Configuration 3) The image processing device according to configuration 1, wherein the switching means switches a method of user input for selecting the virtual object according to a subject recognized from the captured image. (Configuration 4) The image processing device according to configuration 3, wherein the switching means switches a method of user input for selecting the virtual object according to a part of the body recognized as the subject. (Configuration 5) The image processing device according to configuration 1, further comprising: a means for identifying a body part associated with a virtual object to be displayed in the display area based on setting information associating the virtual object with a body part to be superimposed on the virtual object; and the switching means for switching a method of user input for selecting the virtual object according to the body part identified by the identifying means. (Configuration 6) An image processing device that superimposes an image of a product as the virtual object on a body part recognized from the captured image, wherein the display control means displays a selection screen in the display area to accept selection of the product image to be displayed from a user based on setting information that corresponds the product image to the body part to be superimposed on the product image. The image processing device described in any one of configurations 1 to 5. (Configuration 7) The image processing device described in Configuration 6, characterized in that the display control means switches the image of the product in response to user input sequentially received by a user input method for selecting the applied virtual object, and displays the image of the product in the display area by superimposing it on a body part recognized as a subject from the captured image. (Configuration 8) The image processing device according to configuration 6 or 7, characterized in that when the switching means recognizes that an operation is being performed with one hand, it switches the method of user input for selecting the virtual object from a first input method set as a default input method to a second input method associated with the operation with one hand. (Configuration 9) The image processing device described in Configuration 8, wherein the switching means recognizes that the operation is being performed with one hand when the image of the product is superimposed on a hand or arm recognized as a subject from the captured image. (Configuration 10) The image processing device described in configuration 8 or 9, characterized in that the first input method is an input method for selecting, by a tap operation, an image of a product to be superimposed on a body part recognized as a subject from the captured image from a list of product images displayed in the display area. (Configuration 11) The image processing device described in any one of configurations 8 to 10, characterized in that the second input method is an input method for selecting a product image to be superimposed on a body part recognized as the subject from the captured image by swiping a list of product images displayed in the display area, and the display control means, when switched to the second input method, displays display information in the display area prompting the user to perform the swipe operation. (Configuration 12) The image processing device described in any one of configurations 8 to 10, characterized in that the second input method is an input method for selecting, by a tapping operation, an image of a product to be superimposed on a body part recognized as the subject from the captured image from a list of product images displayed in the display area, and the display control means, when switched to the second input method, displays display information in the display area prompting the user to perform the tapping operation. (Method 1) A control method for an image processing device, comprising: a switching step for switching a method of user input for selecting a virtual object to be displayed in a display area depending on the type of virtual object to be displayed in the display area; and a display control step for superimposing an image of the selected virtual object on a captured image and displaying it in the display area. (Program 1) A program for causing a computer to function as an image processing device, characterized by having a switching means for switching a user input method for selecting a virtual object to be displayed in a display area depending on the type of virtual object to be displayed in the display area, and a display control means for superimposing an image of the selected virtual object on a captured image and displaying it in the display area. [Explanation of symbols]

[0057] 1 Image processing device 402 Display control unit 405 Switching section

Claims

1. a switching means for switching a user input method for selecting a virtual object to be displayed in a display area according to a type of the virtual object to be displayed in the display area; a display control means for superimposing an image of the selected virtual object on a captured image and displaying the image in the display area; 13. An image processing device comprising:

2. The image processing apparatus according to claim 1 , wherein the switching means switches a method of user input relating to the selection of a virtual object thereafter, depending on the virtual object selected based on a user input.

3. The image processing device according to claim 1 , wherein the switching means switches a method of user input for selecting the virtual object according to a subject recognized from the captured image.

4. The image processing device according to claim 3 , wherein the switching means switches a method of user input for selecting the virtual object according to a part of the body recognized as the subject.

5. a specifying means for specifying a body part associated with a virtual object to be displayed in the display area based on setting information that associates the virtual object with a body part to be superimposed on the virtual object, The switching means switches a method of user input for selecting the virtual object in accordance with the part of the body identified by the identification means.

2. The image processing device according to claim 1 .

6. An image processing device that superimposes an image of a product as the virtual object on a body part recognized from the captured image, The display control means displays a selection screen in the display area for receiving a selection of the product image to be displayed from a user based on setting information that associates the product image with a body part on which the product image is to be superimposed.

2. The image processing device according to claim 1 .

7. 7. The image processing device according to claim 6, wherein the display control means switches the image of the product in response to user input sequentially received by a user input method for selecting the virtual object being applied, and displays the image of the product in the display area by superimposing it on a body part recognized as a subject from the captured image.

8. 7. The image processing device according to claim 6, wherein the switching means, when it recognizes that an operation is to be performed with one hand, switches a method of user input for selecting the virtual object from a first input method set as a default input method to a second input method associated with the operation with one hand.

9. The image processing device according to claim 8, characterized in that the switching means recognizes that the operation is being performed with one hand when the image of the product is superimposed on a hand or arm recognized as a subject from the captured image.

10. 9. The image processing device according to claim 8, wherein the first input method is an input method for selecting, by a tap operation, an image of a product to be superimposed on a body part recognized as a subject from the captured image from a list of images of the product displayed in the display area.

11. the second input method is an input method for selecting an image of a product to be superimposed on a body part recognized as a subject from the captured image by swiping a list of images of the product displayed in the display area, The display control means displays, when the input method is switched to the second input method, display information that prompts a user to perform the swipe operation in the display area. The image processing device according to claim 8 .

12. the second input method is an input method of selecting, by a tap operation, an image of a product to be superimposed on a body part recognized as a subject from the captured image, from a list of images of products displayed in the display area; The display control means displays, when the input method is switched to the second input method, display information that prompts a user to perform the tap operation in the display area. The image processing device according to claim 8 .

13. A control method for an image processing device, comprising: a switching step of switching a method of user input for selecting a virtual object to be displayed in a display area according to a type of the virtual object to be displayed in the display area; a display control step of superimposing an image of the selected virtual object on the captured image and displaying the image in the display area; 2. A method for controlling an image processing device, comprising:

14. Computer, a switching means for switching a user input method for selecting a virtual object to be displayed in a display area according to a type of the virtual object to be displayed in the display area; a display control means for superimposing an image of the selected virtual object on a captured image and displaying the image in the display area; A program for causing an image processing device to function, comprising:

Images