Virtual makeup trial method, device, and computer equipment

Abstract

The present application discloses a virtual makeup trial method, device, and computer equipment, and belongs to the field of video display technology. The virtual makeup trial method of the present application includes: obtaining a first image captured by a first camera of a terminal device, and determining the scene in which the user is currently located based on the first image; obtaining interactive information associated with the scene from a server, and displaying the interactive information on the screen of the terminal device in an AR manner; receiving a makeup trial instruction triggered by a user, calling the second camera of the terminal device to capture a face image according to the makeup trial instruction, and displaying the face image and a beauty template associated with the scene on the screen after the face image is captured; obtaining the beauty model selected by the user in the beauty template; using the beauty model to apply makeup to the face image part corresponding to the beauty template, and displaying the face image after makeup processing on the screen. The present application can improve the user experience.

Description

Technical Field

[0001] The present application relates to the field of video display technology, and in particular to a virtual makeup trial method, device, computer equipment, and computer-readable storage medium. Background Art

[0002] As living standards continue to improve, consumers' ability to purchase cosmetics is also increasing. When purchasing cosmetics, in order to buy suitable and desired cosmetics, consumers need to try out multiple cosmetics of the same type and different models. However, offline makeup trials take up a lot of users' time. In order to solve the problem that offline makeup trials take up a lot of users' time, virtual makeup technology has begun to develop.

[0003] Virtual makeup technology refers to the user trying on makeup online through a virtual makeup app without having to go offline. However, when trying on makeup through virtual makeup technology, there is a lack of interactivity between different users, resulting in a poor user experience. Summary of the invention

[0004] In view of this, a virtual makeup trial method, apparatus, computer device and computer-readable storage medium are now provided to solve the problem of lack of interactivity between users when applying makeup using existing virtual makeup technology, resulting in poor user experience.

[0005] The present application provides a virtual makeup trial method, comprising:

[0006] Acquire a first image captured by a first camera of the terminal device, and determine a scene in which the user is currently located according to the first image;

[0007] Obtaining interactive information associated with the scene from a server, and displaying the interactive information on a screen of the terminal device in an AR manner, wherein the interactive information includes at least one interactive content;

[0008] receiving a makeup trial instruction triggered by a user, calling a second camera of the terminal device to capture a face image according to the makeup trial instruction, and displaying the face image and a makeup template associated with the scene on the screen after the face image is captured;

[0009] Obtaining the makeup model selected by the user in the makeup template;

[0010] The makeup model is used to apply makeup to the facial image portion corresponding to the makeup template, and the facial image after the makeup is displayed on the screen.

[0011] Optionally, acquiring a first image captured by a first camera of a terminal device, and determining a scene in which the user is currently located according to the first image includes:

[0012] Acquire a first image captured by a first camera of the terminal device;

[0013] Determining whether there is a second image matching the first image in a preset image library;

[0014] If so, the scene associated with the second image is used as the scene the user is currently in.

[0015] Optionally, acquiring a first image captured by a first camera of a terminal device, and determining a scene in which the user is currently located according to the first image includes:

[0016] Acquire a first image captured by a first camera of the terminal device, where the first image carries first location information;

[0017] Determining whether there is second location information matching the first location information among the preset multiple second location information;

[0018] If so, the scene associated with the matching second location information is used as the scene where the user is currently located.

[0019] Optionally, the acquiring the interactive information associated with the scene from the server includes:

[0020] Uploading identification information of the scene to a server, so that the server searches for interactive information associated with the scene according to the identification information;

[0021] The server obtains the interaction information associated with the scene.

[0022] Optionally, after the step of displaying the face image and the beauty makeup template associated with the scene on the screen after the face image is acquired, the method further includes:

[0023] Acquire a third image captured by the first camera of the terminal device at the current moment;

[0024] The third image is analyzed, and when a beauty product is included in the third image, the beauty template is switched to a beauty template corresponding to the beauty product.

[0025] Optionally, after acquiring the face image, displaying the face image and the beauty makeup template associated with the scene on the screen includes:

[0026] After the face image is captured, the face image and the beauty template associated with the scene are displayed on the screen in a floating layer or split screen manner.

[0027] Optionally, the method further comprises:

[0028] When it is detected that the user is not in the scene, the picture displayed on the screen is closed.

[0029] The present application also provides a virtual makeup trial device, comprising:

[0030] A determination module, used to obtain a first image captured by a first camera of a terminal device, and determine a scene in which a user is currently located based on the first image;

[0031] A display module, configured to obtain interactive information associated with the scene from a server, and display the interactive information on a screen of the terminal device in an AR manner, wherein the interactive information includes at least one interactive content;

[0032] a receiving module, configured to receive a makeup trial instruction triggered by a user, call the second camera of the terminal device to capture a face image according to the makeup trial instruction, and display the face image and a makeup template associated with the scene on the screen after the face image is captured;

[0033] An acquisition module, used to acquire a makeup model selected by a user in the makeup template;

[0034] The processing module is used to apply makeup to the facial image portion corresponding to the makeup template using the makeup model, and display the facial image after the makeup process on the screen.

[0035] The present application also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0036] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0037] Beneficial effects of the above technical solution:

[0038] In this technical solution, by combining the AR makeup trial function with the interactive content associated with the scene the user is currently in, the user can try on makeup while watching the interactive content associated with the scene, that is, the user can interact with other users during the makeup trial process, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A framework diagram of an embodiment of the system framework diagram of the virtual makeup trial described in the present application;

[0040] Figure 2A flowchart of an embodiment of the virtual makeup trial method described in the present application;

[0041] Figure 3 A detailed flowchart of the steps of obtaining a first image captured by a first camera of a terminal device and determining a scene in which a user is currently located based on the first image for the present application;

[0042] Figure 4 A detailed flowchart of the steps of obtaining a first image captured by a first camera of a terminal device and determining a scene in which a user is currently located based on the first image for the present application;

[0043] Figure 5 A detailed flowchart of the steps of obtaining interactive information associated with the scene from a server for the present application;

[0044] Figure 6 A schematic diagram showing the beauty template for this application;

[0045] Figure 7 is a flow chart of another embodiment of the virtual makeup trial method described in the present application;

[0046] Figure 8 A module diagram of an embodiment of the virtual makeup trial device described in the present application;

[0047] Fig. 9 A schematic diagram of the hardware structure of a computer device for executing a virtual makeup trial method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The advantages of the present application are further described below in conjunction with the accompanying drawings and specific embodiments.

[0049] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0050] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0051] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0052] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0053] Please refer to Figure 1 , Figure 1 is a schematic diagram showing an application environment of a virtual makeup trial method according to an embodiment of the present application. In an exemplary embodiment, a terminal device 2 can obtain data from a server 4 through a network 6. The terminal device 2 can be a mobile phone, a tablet personal computer, or an electronic device with a data transmission function. The network 6 can be the Internet.

[0054] Figure 2 The flowchart of the virtual makeup trial method according to the first embodiment of the present application is schematically shown. The method is applied in a terminal device to realize the user's offline makeup trial or online makeup trial. It can be understood that the flowchart in the embodiment of the method is not used to limit the order of executing the steps. The following is an exemplary description with the terminal device as the execution subject.

[0055] like Figure 2 As shown, the virtual makeup trial method includes steps S20 to S23, wherein:

[0056] Step S20: acquiring a first image captured by a first camera of the terminal device, and determining a scene in which the user is currently located according to the first image;

[0057] Specifically, the first image is an image captured by the first camera of the terminal device after the user turns on the AR (Augmented Reality) mode. It is understandable that if the AR mode is not currently in use, before the terminal device acquires the first image, the user needs to trigger the AR (Augmented Reality) mode on command by clicking a physical button on the terminal device, or by clicking an AR mode on control set on the terminal device to trigger the AR mode on command, or by sliding on the screen of the terminal device to trigger the AR mode on command, for example, the user triggers the AR mode on command by pulling down.

[0058] When the user triggers the AR mode on command, if the user triggers the AR mode on command for the first time, before calling the first camera of the terminal device to capture the first image, the user needs to first authorize the terminal device to call the camera, and only after receiving the user-triggered consent authorization operation instruction, can the terminal device's camera be called to capture the first image. It is understandable that when the user has authorized the terminal device to call the camera's operation permission in advance, then when the user triggers the AR mode on command for the first time, there is no need to authorize the terminal device to call the camera's operation permission again.

[0059] After the terminal device acquires the first image, the currently acquired first image can be identified to determine the scene the user is currently in. The scene the user is currently in can be the counter the user is currently in, the store the user is currently in, or an online activity held by a store the user is currently participating in, or an offline activity held by a store the user is currently participating in, etc.

[0060] It should be noted that the first camera is preferably a rear camera of a terminal device, and the terminal device is an electronic device with a camera, such as a mobile phone or a tablet computer.

[0061] In an exemplary embodiment, referring to Figure 3 , the step of acquiring a first image captured by a first camera of a terminal device and determining the current scene of the user according to the first image includes:

[0062] Step S30: Acquire a first image captured by a first camera of the terminal device.

[0063] Step S31, determining whether there is a second image matching the first image in a preset image library.

[0064] Specifically, images or 3D scene graphs corresponding to each offline real scene are stored in advance in the image library, or images corresponding to the online scene, wherein the images corresponding to the online scene can be marketing web page pictures with identification target scenes. In this embodiment, the image corresponding to each scene can be one or more images, wherein the image library can be stored in the local storage of the terminal device or in the server. When the image library is stored in the server, it is necessary to first download the image library from the server to the local storage. Of course, when the image library is stored in the server, it is also possible not to download the image library from the server to the local storage, but by uploading the first image to the server when identifying the acquired first image, and then judging whether there is a second image matching the first image in the preset image library through the server, and then the server sends the judgment result to the terminal device, so that the terminal device judges whether there is a second image matching the first image in the preset image library according to the judgment result.

[0065] In one embodiment, when there is an image in the preset image library whose similarity with the first image is greater than a preset threshold, it can be considered that there is a second image matching the first image in the image library. It should be noted that when there are multiple images in the image library whose similarity values ​​are greater than the preset threshold, the image with the largest similarity value can be used as the second image.

[0066] Step S32: If so, the scene associated with the second image is used as the scene the user is currently in.

[0067] Specifically, since the images stored in the image library are pre-associated with scenes, when it is determined that there is a second image matching the first image in the preset image library, the scene associated with the second image can be used as the scene where the user is currently located.

[0068] In this embodiment, an image library corresponding to the scene is pre-established so that when the first image is acquired, the scene in which the user is currently located can be accurately identified according to the image library.

[0069] In another exemplary embodiment, referring to Figure 4 , the step of acquiring a first image captured by a first camera of a terminal device and determining the current scene of the user according to the first image includes:

[0070] Step S40: Acquire a first image captured by a first camera of the terminal device, where the first image carries first location information.

[0071] Specifically, when the first camera of the terminal device collects the first image, the first location information is obtained through the positioning module in the terminal device, so that the collected first image can carry the first location information. In this embodiment, the first location information is the longitude and latitude information collected by the positioning module.

[0072] Step S41, determining whether there is second location information matching the first location information among the preset multiple second location information.

[0073] Step S42: If so, the scene associated with the matched second location information is used as the scene where the user is currently located.

[0074] Specifically, the second location information corresponding to each scene is pre-stored. When the first location information is obtained, the first location information can be matched with each preset second location information. When the first location information matches one of the second location information, the scene associated with the matched second location information can be used as the scene where the user is currently located.

[0075] Exemplarily, when the first location information is within the preset location range of the second location information, it can be considered that the first location information matches the second location information. The preset location range can be set according to actual conditions. For example, if the preset location range is within 10 meters, then when the first location information is within 10 meters of the second location information, it can be determined that the first location information matches the second location information.

[0076] It should be noted that the second location information mentioned above is also longitude and latitude information.

[0077] In this embodiment, since the second location information is pre-associated with the scene, when it is determined that there is second location information matching the first location information, the scene associated with the second location information can be used as the scene where the user is currently located.

[0078] This embodiment pre-establishes an association relationship between the scene and the second location information, so that when the second location information matching the first location information is obtained, the scene where the user is currently located can be accurately identified according to the second location information.

[0079] Step S21, obtaining interactive information associated with the scene from a server, and displaying the interactive information on the screen of the terminal device in an AR manner, wherein the interactive information includes at least one interactive content.

[0080] Specifically, the server stores interactive information associated with each scene. In this embodiment, after determining the scene the user is currently in, the terminal device triggers an interactive information acquisition instruction to obtain interactive information associated with the scene from the server. The interactive information may be videos, comments, etc. posted by multiple users in various scenes that are related to the scene they are currently in, wherein each video or comment posted by a user is an interactive content. In a specific scene, the interactive information may be videos or comments posted by the UP host and other users in the scene they are currently in, videos or comments with local check-in information, videos or comments with titles and content fields related to the scene they are currently in, etc. In one embodiment, the interactive information may also be fixed content that the user has prepared in advance for related content in the promotional scene, such as a promotional video prepared by the user for a certain brand of lipstick sold in a certain store. Since the interactive information is posted by the user in the current scene, or is produced for the purpose of promoting related content in the scene, each interactive content will have an association with the corresponding scene. That is, when a user posts an interactive content in scene A, when uploading the interactive content to the server, the server will associate the interactive content with scene A based on the association between the interactive content and scene A.

[0081] In this embodiment, when the interactive information is obtained, the obtained interactive information can be displayed in an AR manner. In one embodiment, when the interactive information is displayed, if there are many interactive contents obtained, all the obtained interactive contents can be sorted by the latest, most popular or most relevant method, and then the preset interactive contents in the front order are selected for display.

[0082] In one embodiment, when the interactive content displayed on the screen includes a video, the user can also click on the video to jump to the play page to view the full content. When the interactive content displayed on the screen includes comments, the user can also click on the comments to view the comments or related dynamics.

[0083] In an exemplary embodiment, referring to Figure 5 , the acquiring of interactive information associated with the scene from the server includes:

[0084] Step S50: uploading the identification information of the scene to the server, so that the server searches for interactive information associated with the scene according to the identification information.

[0085] The identification information is information used to represent the scene. For example, when the scene is a store, the identification information may be the name of the store; when the scene is a counter, the identification information may also be the name of the counter; when the scene is an online activity of a store, the identification information may also be the name of the store.

[0086] In this embodiment, each interactive content stored in the server is associated with a scene, that is, each interactive content in the server has identification information of the corresponding scene. In this way, when the server receives the identification information of the scene uploaded by the terminal device, it can accurately find the interactive information associated with the scene based on the identification information.

[0087] Step S51, obtaining the interactive information associated with the scene found and returned by the server.

[0088] Specifically, after finding the interactive information associated with the scene, the server can return the found interactive information to the terminal device, so that the terminal device can obtain the interactive information associated with the scene.

[0089] This embodiment uploads the tag information of the scene to the server, so that the server can find the interactive information associated with the scene according to the identification information, thereby improving the accuracy of the acquired interactive information.

[0090] Step S22, receiving a makeup trial instruction triggered by a user, calling the second camera of the terminal device to capture a facial image according to the makeup trial instruction, and displaying the facial image and a makeup template associated with the scene on the screen after the facial image is captured.

[0091] Specifically, when the user is watching interactive content, if the user wants to try on makeup online, the user will perform a makeup trial instruction trigger operation. Once the user triggers the makeup trial instruction, the terminal device will receive the makeup trial instruction. In this embodiment, the user can trigger the makeup trial instruction by clicking on the makeup trial control displayed on the terminal screen. The user can also trigger the makeup trial instruction by double-clicking, long pressing, sliding in a specific shape or direction, or sliding a joint across the display interface.

[0092] In this embodiment, when the terminal device receives the makeup trial instruction, it can call the second camera to collect the face image. After collecting the face image, the makeup trial function will be turned on, that is, the face image and the beauty template associated with the scene will be displayed on the screen. Among them, the beauty template is used to display the models of beauty products that can be tried by the user. The beauty template is a combination of beauty products and models set by the brand party corresponding to each scene in advance, and is stored in the database of the makeup trial function in a combined manner. Beauty products can be eye makeup, lipstick, blush, earrings, etc.

[0093] In one embodiment, after a face image is captured, the face image and a beauty template associated with the scene can be displayed on the screen in a floating layer or split screen manner. In a specific scene, the displayed face image and beauty template are as follows: Figure 6 shown.

[0094] In this embodiment, each scene may be associated with a beauty template of a beauty product, or may be associated with beauty templates of multiple beauty products, for example, a beauty template of lipstick and a beauty template of eye makeup. In this embodiment, if a scene is associated with beauty templates of multiple products, the beauty templates of multiple products may be displayed in a list format when displaying the beauty templates, so that the user can choose which beauty product to try on.

[0095] It should be noted that the second camera is preferably a front camera of the terminal device.

[0096] In an exemplary embodiment, when the terminal device displays the facial image and the beauty template associated with the scene on the screen, it can also provide guiding links such as sharing, coupons, and purchases. When the user clicks "Share", the terminal device will capture the user's makeup trial screen and bring in the dynamic publishing function. Clicking "Buy" can link to the e-commerce app or product page, and clicking on the coupon can receive a discount code or red envelope.

[0097] In an exemplary embodiment, referring to Figure 7 , after the step of displaying the face image and the beauty makeup template associated with the scene on the screen after the face image is collected, it also includes:

[0098] Step S70, obtaining a third image captured by the first camera of the terminal device at the current moment.

[0099] Step S71, analyzing the third image, and when the third image contains a beauty product, switching the beauty template to a beauty template corresponding to the beauty product.

[0100] Specifically, when the current scene is associated with multiple beauty products, the third image currently captured by the first camera can be analyzed to determine whether the third image currently captured contains beauty products. For example, if the third product currently captured contains eye makeup products, the currently displayed beauty template can be automatically switched to the beauty template corresponding to the eye makeup product. It is understandable that if the currently displayed beauty template is an eye makeup template, there is no need to switch.

[0101] This embodiment can simplify user operations and improve user experience by automatically switching makeup templates in different makeup scenarios without the need for manual switching by the user.

[0102] Step S23, obtaining the makeup model selected by the user in the makeup template.

[0103] Specifically, the beauty template provides a variety of beauty models for users to choose from. For example, if the beauty template is a lipstick beauty template, all models of the lipstick can be provided in the template for users to choose from, such as bright red, brick red, pink, purple red, peach red, orange, red orange, tomato color and other models.

[0104] Step S24, using the beauty makeup model to apply makeup to the facial image portion corresponding to the beauty makeup template, and displaying the facial image after the makeup process on the screen.

[0105] Specifically, before applying makeup, it is necessary to first identify the trial makeup part corresponding to the trial makeup template from the face image. For example, if the trial makeup template is a lipstick trial makeup template, it is necessary to first identify the lip area in the face image from the face image; when the trial makeup template is an eye makeup template, it is necessary to first identify the eyebrow part in the face image from the face image. In this embodiment, when identifying the trial makeup part in the face image, the existing recognition technology can be used, which will not be repeated in this embodiment.

[0106] In one embodiment, when applying makeup, the acquired image samples corresponding to the makeup model can be set at the corresponding facial image parts; then the edges of the image samples corresponding to the makeup model can be blurred so that the edges of the image samples corresponding to the makeup model can transition naturally with the user's facial image.

[0107] This embodiment combines the AR makeup trial function with the interactive content associated with the scene the user is currently in, so that the user can try on makeup while watching the interactive content associated with the scene. That is, the user can interact with other users during the makeup trial process, thereby improving the user experience.

[0108] In an exemplary embodiment, the virtual makeup trial method further includes:

[0109] When it is detected that the user is not in the scene, the picture displayed on the screen is closed.

[0110] Specifically, it is detected in real time or at a scheduled time whether the user is still in the scene. When the user is not in the scene, the picture displayed on the screen is automatically closed, that is, the makeup trial function and the interactive content display function are automatically closed.

[0111] In this embodiment, by automatically closing the picture displayed on the screen when it is identified that the user is not in the scene, the user does not need to close it manually, thereby improving the user experience.

[0112] In order to facilitate the understanding of the present application, the application of the present application will be described in detail below in conjunction with specific embodiments.

[0113] See also Figure 6 When the user arrives at store A, he opens the client makeup trial APP or web makeup trial client installed on the terminal device such as mobile phone and tablet computer, and then turns on the AR mode. The terminal device calls the rear camera to collect the image of the user's current scene, and then performs image recognition on the collected image to identify the user's current scene, that is, to identify that the user is currently in store A. After identifying that the user is in store A, the terminal device obtains interactive information related to store A from the server, and after obtaining the interactive information, uses AR to display the obtained interactive information on the screen of the terminal device so that the user can watch the interactive information. At any time when the user is watching the interactive information, the user can trigger the makeup trial instruction, such as Figure 6 As shown, the user can trigger the makeup trial instruction by clicking the Try Color Now button displayed on the screen. After the user triggers the makeup trial instruction, the terminal device will call the front camera to scan the face image, and after scanning the face image, obtain the makeup trial template associated with the user's current scene, and then display the makeup trial template and the face image on the screen in a floating layer. After that, the user can try on the makeup.

[0114] Figure 8 The block diagram of a virtual makeup trial device according to an embodiment of the present application is schematically shown. The virtual makeup trial device can be divided into one or more program modules, one or more program modules are stored in a storage medium, and are executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions, which is more suitable for describing the execution process of the video processing system in the storage medium than the program itself. The following description will specifically introduce the functions of each program module of this embodiment.

[0115] like Figure 8 As shown, the virtual makeup trial device 800 may include a position determination module 801, a display module 802, a receiving module 803, an acquisition module 804 and a processing module 805, wherein:

[0116] The determination module 801 is used to obtain a first image captured by a first camera of a terminal device, and determine the scene in which the user is currently located according to the first image;

[0117] Specifically, the first image is an image captured by the first camera of the terminal device after the user turns on the AR (Augmented Reality) mode. It is understandable that if the AR mode is not currently in use, before the terminal device acquires the first image, the user needs to trigger the AR (Augmented Reality) mode on command by clicking a physical button on the terminal device, or by clicking an AR mode on control set on the terminal device to trigger the AR mode on command, or by sliding on the screen of the terminal device to trigger the AR mode on command, for example, the user triggers the AR mode on command by pulling down.

[0118] When the user triggers the AR mode on command, if the user triggers the AR mode on command for the first time, before calling the first camera of the terminal device to capture the first image, the user needs to first authorize the terminal device to call the camera, and only after receiving the user-triggered consent authorization operation instruction, can the terminal device's camera be called to capture the first image. It is understandable that when the user has authorized the terminal device to call the camera's operation permission in advance, then when the user triggers the AR mode on command for the first time, there is no need to authorize the terminal device to call the camera's operation permission again.

[0119] After the terminal device acquires the first image, the currently acquired first image can be identified to determine the scene the user is currently in. The scene the user is currently in can be the counter the user is currently in, the store the user is currently in, or an online activity held by a store the user is currently participating in, or an offline activity held by a store the user is currently participating in, etc.

[0120] It should be noted that the first camera is preferably a rear camera of a terminal device, and the terminal device is an electronic device with a camera, such as a mobile phone or a tablet computer.

[0121] In an exemplary implementation, the determination module 801 is further configured to obtain a first image captured by a first camera of the terminal device.

[0122] The determination module 801 is further configured to determine whether there is a second image matching the first image in a preset image library.

[0123] Specifically, images or 3D scene graphs corresponding to each offline real scene are stored in advance in the image library, or images corresponding to the online scene, wherein the images corresponding to the online scene can be marketing web page pictures with identification target scenes. In this embodiment, the image corresponding to each scene can be one or more images, wherein the image library can be stored in the local storage of the terminal device or in the server. When the image library is stored in the server, it is necessary to first download the image library from the server to the local storage. Of course, when the image library is stored in the server, it is also possible not to download the image library from the server to the local storage, but by uploading the first image to the server when identifying the acquired first image, and then judging whether there is a second image matching the first image in the preset image library through the server, and then the server sends the judgment result to the terminal device, so that the terminal device judges whether there is a second image matching the first image in the preset image library according to the judgment result.

[0124] In one embodiment, when there is an image in the preset image library whose similarity with the first image is greater than a preset threshold, it can be considered that there is a second image matching the first image in the image library. It should be noted that when there are multiple images in the image library whose similarity values ​​are greater than the preset threshold, the image with the largest similarity value can be used as the second image.

[0125] The determination module 801 is further configured to, if there is a second image matching the first image, use the scene associated with the second image as the scene where the user is currently located.

[0126] Specifically, since the images stored in the image library are pre-associated with scenes, when it is determined that there is a second image matching the first image in the preset image library, the scene associated with the second image can be used as the scene where the user is currently located.

[0127] In this embodiment, an image library corresponding to the scene is pre-established so that when the first image is acquired, the scene in which the user is currently located can be accurately identified according to the image library.

[0128] In another exemplary implementation, the determination module 801 is further configured to obtain a first image captured by a first camera of the terminal device, where the first image carries first location information.

[0129] Specifically, when the first camera of the terminal device collects the first image, the first location information is obtained through the positioning module in the terminal device, so that the collected first image can carry the first location information. In this embodiment, the first location information is the longitude and latitude information collected by the positioning module.

[0130] The determination module 801 is further used to determine whether there is second location information matching the first location information among the preset multiple second location information.

[0131] The determination module 801 is further configured to, if there is second location information matching the first location information, use the scene associated with the matching second location information as the scene where the user is currently located.

[0132] Specifically, the second location information corresponding to each scene is pre-stored. When the first location information is obtained, the first location information can be matched with each preset second location information. When the first location information matches one of the second location information, the scene associated with the matched second location information can be used as the scene where the user is currently located.

[0133] Exemplarily, when the first location information is within the preset location range of the second location information, it can be considered that the first location information matches the second location information. The preset location range can be set according to actual conditions. For example, if the preset location range is within 10 meters, then when the first location information is within 10 meters of the second location information, it can be determined that the first location information matches the second location information.

[0134] It should be noted that the second location information mentioned above is also longitude and latitude information.

[0135] In this embodiment, since the second location information is pre-associated with the scene, when it is determined that there is second location information matching the first location information, the scene associated with the second location information can be used as the scene where the user is currently located.

[0136] This embodiment pre-establishes an association relationship between the scene and the second location information, so that when the second location information matching the first location information is obtained, the scene where the user is currently located can be accurately identified according to the second location information.

[0137] The display module 802 is used to obtain interactive information associated with the scene from the server, and display the interactive information on the screen of the terminal device in an AR manner, wherein the interactive information includes at least one interactive content.

[0138] Specifically, the server stores interactive information associated with each scene. In this embodiment, after determining the scene the user is currently in, the terminal device triggers an interactive information acquisition instruction to obtain interactive information associated with the scene from the server. The interactive information may be videos, comments, etc. posted by multiple users in various scenes that are related to the scene they are currently in, wherein each video or comment posted by a user is an interactive content. In a specific scene, the interactive information may be videos or comments posted by the UP host and other users in the scene they are currently in, videos or comments with local check-in information, videos or comments with titles and content fields related to the scene they are currently in, etc. In one embodiment, the interactive information may also be fixed content that the user has prepared in advance for related content in the promotional scene, such as a promotional video prepared by the user for a certain brand of lipstick sold in a certain store. Since the interactive information is posted by the user in the current scene, or is produced for the purpose of promoting related content in the scene, each interactive content will have an association with the corresponding scene. That is, when a user posts an interactive content in scene A, when uploading the interactive content to the server, the server will associate the interactive content with scene A based on the association between the interactive content and scene A.

[0139] In this embodiment, when the interactive information is obtained, the obtained interactive information can be displayed in an AR manner. In one embodiment, when the interactive information is displayed, if there are many interactive contents obtained, all the obtained interactive contents can be sorted by the latest, most popular or most relevant method, and then the preset interactive contents in the front order are selected for display.

[0140] In one embodiment, when the interactive content displayed on the screen includes a video, the user can also click on the video to jump to the play page to view the full content. When the interactive content displayed on the screen includes comments, the user can also click on the comments to view the comments or related dynamics.

[0141] In an exemplary embodiment, the display module 802 is further configured to upload identification information of the scene to a server, so that the server searches for interactive information associated with the scene according to the identification information.

[0142] The identification information is information used to represent the scene. For example, when the scene is a store, the identification information may be the name of the store; when the scene is a counter, the identification information may also be the name of the counter; when the scene is an online activity of a store, the identification information may also be the name of the store.

[0143] In this embodiment, each interactive content stored in the server is associated with a scene, that is, each interactive content in the server has identification information of the corresponding scene. In this way, when the server receives the identification information of the scene uploaded by the terminal device, it can accurately find the interactive information associated with the scene based on the identification information.

[0144] The display module 802 is further used to obtain the interactive information associated with the scene found and returned by the server.

[0145] Specifically, after finding the interactive information associated with the scene, the server can return the found interactive information to the terminal device, so that the terminal device can obtain the interactive information associated with the scene.

[0146] This embodiment uploads the tag information of the scene to the server, so that the server can find the interactive information associated with the scene according to the identification information, thereby improving the accuracy of the acquired interactive information.

[0147] The receiving module 803 is used to receive a makeup trial instruction triggered by a user, call the second camera of the terminal device to capture a facial image according to the makeup trial instruction, and display the facial image and the beauty template associated with the scene on the screen after the facial image is captured.

[0148] Specifically, when the user is watching interactive content, if the user wants to try on makeup online, the user will perform a makeup trial instruction trigger operation. Once the user triggers the makeup trial instruction, the terminal device will receive the makeup trial instruction. In this embodiment, the user can trigger the makeup trial instruction by clicking on the makeup trial control displayed on the terminal screen. The user can also trigger the makeup trial instruction by double-clicking, long pressing, sliding in a specific shape or direction, or sliding a joint across the display interface.

[0149] In this embodiment, when the terminal device receives the makeup trial instruction, it can call the second camera to collect the face image. After collecting the face image, the makeup trial function will be turned on, that is, the face image and the beauty template associated with the scene will be displayed on the screen. Among them, the beauty template is used to display the models of beauty products that can be tried by the user. The beauty template is a combination of beauty products and models set by the brand party corresponding to each scene in advance, and is stored in the database of the makeup trial function in a combined manner. Beauty products can be eye makeup, lipstick, blush, earrings, etc.

[0150] In one embodiment, after a face image is captured, the face image and a beauty template associated with the scene can be displayed on the screen in a floating layer or split screen manner. In a specific scene, the displayed face image and beauty template are as follows: Figure 6 shown.

[0151] In this embodiment, each scene may be associated with a beauty template of a beauty product, or may be associated with beauty templates of multiple beauty products, for example, a beauty template of lipstick and a beauty template of eye makeup. In this embodiment, if a scene is associated with beauty templates of multiple products, the beauty templates of multiple products may be displayed in a list format when displaying the beauty templates, so that the user can choose which beauty product to try on.

[0152] It should be noted that the second camera is preferably a front camera of the terminal device.

[0153] In an exemplary embodiment, when the terminal device displays the facial image and the beauty template associated with the scene on the screen, it can also provide guiding links such as sharing, coupons, and purchases. When the user clicks "Share", the terminal device will capture the user's makeup trial screen and bring in the dynamic publishing function. Clicking "Buy" can link to the e-commerce app or product page, and clicking on the coupon can receive a discount code or red envelope.

[0154] In an exemplary embodiment, the receiving module 803 is further configured to obtain a third image captured by the first camera of the terminal device at a current moment.

[0155] The receiving module 803 is further configured to analyze the third image, and when the third image contains a beauty product, switch the beauty template to a beauty template corresponding to the beauty product.

[0156] Specifically, when the current scene is associated with multiple beauty products, the third image currently captured by the first camera can be analyzed to determine whether the third image currently captured contains beauty products. For example, if the third product currently captured contains eye makeup products, the currently displayed beauty template can be automatically switched to the beauty template corresponding to the eye makeup product. It is understandable that if the currently displayed beauty template is an eye makeup template, there is no need to switch.

[0157] This embodiment can simplify user operations and improve user experience by automatically switching makeup templates in different makeup scenarios without the need for manual switching by the user.

[0158] The acquisition module 804 is used to acquire the makeup model selected by the user in the makeup template.

[0159] Specifically, the beauty template provides a variety of beauty models for users to choose from. For example, if the beauty template is a lipstick beauty template, all models of the lipstick can be provided in the template for users to choose from, such as bright red, brick red, pink, purple red, peach red, orange, red orange, tomato color and other models.

[0160] The processing module 805 is further used to apply makeup to the facial image portion corresponding to the makeup template using the makeup model, and display the facial image after the makeup process on the screen.

[0161] Specifically, before applying makeup, it is necessary to first identify the trial makeup part corresponding to the trial makeup template from the face image. For example, if the trial makeup template is a lipstick trial makeup template, it is necessary to first identify the lip area in the face image from the face image; when the trial makeup template is an eye makeup template, it is necessary to first identify the eyebrow part in the face image from the face image. In this embodiment, when identifying the trial makeup part in the face image, the existing recognition technology can be used, which will not be repeated in this embodiment.

[0162] In one embodiment, when applying makeup, the acquired image samples corresponding to the makeup model can be set at the corresponding facial image parts; then the edges of the image samples corresponding to the makeup model can be blurred so that the edges of the image samples corresponding to the makeup model can transition naturally with the user's facial image.

[0163] This embodiment combines the AR makeup trial function with the interactive content associated with the scene the user is currently in, so that the user can try on makeup while watching the interactive content associated with the scene. That is, the user can interact with other users during the makeup trial process, thereby improving the user experience.

[0164] In an exemplary embodiment, the virtual makeup trial device 800 further includes: a detection module.

[0165] The detection module is used to close the picture displayed on the screen when it is detected that the user is not in the scene.

[0166] Specifically, it is detected in real time or at a scheduled time whether the user is still in the scene. When the user is not in the scene, the picture displayed on the screen is automatically closed, that is, the makeup trial function and the interactive content display function are automatically closed.

[0167] In this embodiment, by automatically closing the picture displayed on the screen when it is identified that the user is not in the scene, the user does not need to close it manually, thereby improving the user experience.

[0168] Fig. 9The schematic diagram of the hardware architecture of a computer device suitable for implementing the virtual makeup trial method according to the fourth embodiment of the present application is shown schematically. In this embodiment, the computer device 9 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. For example, it can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a rack server, a blade server, a tower server or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Fig. 9 As shown, the computer device 9 includes at least but is not limited to: a memory 910, a processor 920, and a network interface 930 which can communicate with each other through a system bus. Among them:

[0169] The memory 910 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 910 can be an internal storage module of the computer device 9, such as a hard disk or memory of the computer device 9. In other embodiments, the memory 910 can also be an external storage device of the computer device 9, such as a plug-in hard disk equipped on the computer device 9, a smart memory card (SmartMedia Card, referred to as SMC), a secure digital (Secure Digital, referred to as SD) card, a flash card, etc. Of course, the memory 910 can also include both the internal storage module of the computer device 9 and its external storage device. In this embodiment, the memory 910 is generally used to store the operating system and various application software installed on the computer device 9, such as the program code of the virtual makeup trial method, etc. In addition, the memory 910 can also be used to temporarily store various types of data that have been output or are to be output.

[0170] In some embodiments, the processor 920 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 920 is generally used to control the overall operation of the computer device 9, such as performing control and processing related to data interaction or communication with the computer device 9. In this embodiment, the processor 920 is used to run the program code stored in the memory 910 or process data.

[0171] The network interface 930 may include a wireless network interface or a wired network interface, and the network interface 930 is generally used to establish a communication link between the computer device 9 and other computer devices. For example, the network interface 930 is used to connect the computer device 9 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 9 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0172] It should be pointed out that Fig. 9 Only a computer device having components 910 to 930 is shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0173] In this embodiment, the virtual makeup trial method stored in the memory 910 can also be divided into one or more program modules and executed by one or more processors (processor 920 in this embodiment) to complete the present application.

[0174] This embodiment further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the virtual makeup trial method in the embodiment are implemented.

[0175] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, referred to as SMC), a secure digital (Secure Digital, referred to as SD) card, a flash card, etc. equipped on the computer device. Of course, the computer-readable storage medium may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the virtual makeup trial method in the embodiment, etc. In addition, the computer-readable storage medium may also be used to temporarily store various types of data that have been output or are to be output.

[0176] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0177] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A virtual makeup trial method, characterized in that: include: Acquire a first image captured by a first camera of the terminal device, and determine the scene in which the user is currently located according to the first image; wherein the first image is an image captured by calling the first camera of the terminal device after the user turns on the AR mode; Obtaining interactive information associated with the scene from a server, and displaying the interactive information on a screen of the terminal device in an AR manner, wherein the interactive information includes at least one interactive content; receiving a makeup trial instruction triggered by a user, calling a second camera of the terminal device to capture a face image according to the makeup trial instruction, and displaying the face image and a makeup template associated with the scene on the screen after the face image is captured; Obtaining the makeup model selected by the user in the makeup template; The makeup model is used to apply makeup to the facial image portion corresponding to the makeup template, and the facial image after the makeup is displayed on the screen.

2. The virtual makeup trial method according to claim 1, characterized in that: The acquiring a first image captured by a first camera of the terminal device and determining the current scene of the user according to the first image includes: Acquire a first image captured by a first camera of the terminal device; Determining whether there is a second image matching the first image in a preset image library; If so, the scene associated with the second image is used as the scene the user is currently in.

3. The virtual makeup trial method according to claim 1, characterized in that: The acquiring a first image captured by a first camera of the terminal device and determining the current scene of the user according to the first image includes: Acquire a first image captured by a first camera of the terminal device, where the first image carries first location information; Determining whether there is second location information matching the first location information among the preset multiple second location information; If so, the scene associated with the matching second location information is used as the scene where the user is currently located.

4. The virtual makeup trial method according to claim 1, characterized in that: The acquiring the interactive information associated with the scene from the server includes: Uploading identification information of the scene to a server, so that the server searches for interactive information associated with the scene according to the identification information; The server obtains the interaction information associated with the scene.

5. The virtual makeup trial method according to claim 1, characterized in that: After the step of displaying the face image and the beauty makeup template associated with the scene on the screen after the face image is collected, the method further includes: Acquire a third image captured by the first camera of the terminal device at the current moment; The third image is analyzed, and when a beauty product is included in the third image, the beauty template is switched to a beauty template corresponding to the beauty product.

6. The virtual makeup trial method according to claim 1, characterized in that: The method of displaying the face image and the beauty makeup template associated with the scene on the screen after the face image is collected includes: After the face image is captured, the face image and the beauty template associated with the scene are displayed on the screen in a floating layer or split screen manner.

7. The virtual makeup trial method according to any one of claims 1 to 6, characterized in that: The method further comprises: When it is detected that the user is not in the scene, the picture displayed on the screen is closed.

8. A virtual makeup trial device, characterized in that: include: A determination module, configured to obtain a first image captured by a first camera of a terminal device, and determine a scene in which a user is currently located according to the first image; wherein the first image is an image captured by calling the first camera of the terminal device after the user turns on the AR mode; A display module, configured to obtain interactive information associated with the scene from a server, and display the interactive information on a screen of the terminal device in an AR manner, wherein the interactive information includes at least one interactive content; a receiving module, configured to receive a makeup trial instruction triggered by a user, call the second camera of the terminal device to capture a face image according to the makeup trial instruction, and display the face image and a makeup template associated with the scene on the screen after the face image is captured; An acquisition module, used to acquire a makeup model selected by a user in the makeup template; The processing module is used to apply makeup to the facial image portion corresponding to the makeup template using the makeup model, and display the facial image after the makeup process on the screen.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the virtual makeup trial method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the virtual makeup trial method according to any one of claims 1 to 7 are implemented.

Images