Method and device for providing photographing service including artificial intelligence-based expression utilization advertisement function

Abstract

The present invention relates to a method performed by a processor of a photographing service provision device, and the method comprises the steps of: receiving at least one instant print image according to an advertisement event participation request from a photographing device or a user device; determining whether an advertisement expression is included in the instant print image by inputting the instant print image to an expression recognition model configured to output whether the advertisement expression is included by using an image as an input; and when the advertisement expression is included in the instant print image, providing a reward to the photographing device or the user device.

Description

Method and device for providing a photo shooting service including an artificial intelligence-based facial expression-based advertising function

[0001] The present invention relates to a method and device for providing a photo shooting service including an artificial intelligence-based facial expression-utilizing advertising function.

[0002] Unmanned photo studios offering self-portrait services are establishing themselves as new experiential spaces for young people to express themselves and appeal to them. Taking advantage of their booth-style design, unmanned photo studios are being installed in locations with high traffic, such as densely populated cultural facilities, event venues, exhibition halls, and concert halls.

[0003] Accordingly, advertising targeting users visiting unmanned photo studios is being introduced. Examples include ads that insert idols, athletes, and influencers into the background or frame of a photo, ads that display advertisements on the display screen while the photo is being developed, and ads that display advertisements on websites visited to save photos.

[0004] However, advertising using celebrities has the disadvantage of a narrow target audience, and because the advertisements provided on display screens or web pages are one-sided advertisements provided to users, there are limitations in achieving dramatic advertising effects in a short period of time.

[0005] The background technology of the invention has been prepared to facilitate a better understanding of the present invention. It should not be construed as an admission that the matters described in the background technology of the invention constitute prior art.

[0006] Accordingly, a new advertising delivery method is required to maximize advertising effectiveness through unmanned photo studios that provide self-photography services.

[0007] As a result, the inventors of the present invention sought to develop a method for increasing the advertising effect through the provision of user-participatory advertising services by inducing a specific facial expression in the user while the user is making a facial expression for taking a photo and providing an advertisement matching the facial expression.

[0008] The tasks of the present invention are not limited to the tasks mentioned above, and other tasks not mentioned will be clearly understood by those skilled in the art from the description below.

[0009] In order to solve the above-described problem, a method for providing a photography service including an artificial intelligence-based facial expression utilization function according to an embodiment of the present invention is provided. The method is configured to include the steps of receiving at least one instant output image in response to a request for participation in an advertising event from a photographing device or a user device, inputting the instant output image into a facial expression recognition model configured to output whether an advertisement facial expression is included by taking an image as an input and determining whether an advertisement facial expression is included in the instant output image, and providing a reward to the photographing device or the user device when an advertisement facial expression is included in the instant output image.

[0010] According to a feature of the present invention, the facial expression recognition model may be a model configured to output a probability that the user's facial expression included in the instant output image corresponds to a predefined advertisement facial expression.

[0011] According to a feature of the present invention, the facial expression recognition model may be a model configured to output a probability that the user's emotion included in the image corresponds to a predefined emotional expression for each advertisement.

[0012] According to a feature of the present invention, the method may further include a step of determining an advertisement expression or emotional expression to be recognized by the facial expression recognition model by using an advertisement learning model that is trained to detect a user expression or emotional expression according to the type of advertisement event by inputting advertisement content.

[0013] According to a feature of the present invention, the method further includes a step of providing a guideline for an advertising event using the photographing device, wherein the guideline may include at least one item among a face exposure range for recognition of the advertising expression, a face rotation direction, whether or not a facial feature point is exposed, and a change in size or direction based on the coordinates of the facial feature point.

[0014] According to a feature of the present invention, the step of receiving the image is a step of receiving an image that can be output in real time from the photographing device, and the step of determining whether the advertisement expression includes an expression may further include a step of providing the photographing device with a probability indicating whether the advertisement expression is included in the image that can be output in real time.

[0015] According to a feature of the present invention, the step of receiving the image may further include the step of identifying an advertisement target included in the advertisement event participation request from the photographing device and the step of providing a background or object image that can be inserted into an instant output image using the attributes of the identified advertisement target.

[0016] According to a feature of the present invention, the step of receiving the image may be a step of receiving a plurality of instant output images composed of a plurality of frames, and the step of providing the reward may be a step of providing the reward when an advertisement expression matching the advertisement event participation request is included in at least one instant output image among the plurality of instant output images.

[0017] In order to solve the problem as described above, a photo shooting service providing device according to another embodiment of the present invention is provided. The device includes a communication interface, a memory, and a processor operably connected to the communication interface and the memory, and the processor is configured to receive at least one instant output image in response to an advertising event participation request from a photographing device or a user device, input the instant output image into an expression recognition model configured to output whether an advertisement expression is included by taking an image as an input, and determine whether an advertisement expression is included in the instant output image, and provide a reward to the photographing device or the user device if an advertisement expression is included in the instant output image.

[0018] In order to solve the above-described problem, a photographing device providing an artificial intelligence-based facial expression utilization function according to another embodiment of the present invention is provided. The device includes a communication interface, an I / O interface, a memory, and a processor operably connected to the communication interface, the I / O interface, and the memory, wherein the processor is configured to obtain at least one instant output image including a user's request for participation in an advertising event through the I / O interface, input the instant output image into an expression recognition model configured to output whether an advertising expression is included by taking an image as an input, and determine whether an advertising expression is included in the instant output image, and if an advertising expression is included in the image, provide a reward to a user matching the instant output image through the communication interface.

[0019] Specific details of other embodiments are included in the detailed description and drawings.

[0020] The present invention can encourage active user participation by inducing a specific facial expression associated with an advertising target in a self-photography service (hereinafter referred to as a "photography device") at an unmanned photo studio (hereinafter referred to as a "photography device") and providing a corresponding reward to the user upon detection of the facial expression. In particular, advertising that utilizes user participation can enable more active exposure to users.

[0021] Furthermore, the present invention can improve the recognition accuracy of facial expression recognition models that detect user expressions by providing a camera device with guidelines for advertising targets and specific facial expressions associated with them. In particular, by improving recognition accuracy, rewards can be properly provided to users who participate in advertising events, thereby enhancing user satisfaction with participation.

[0022] In addition, in the process of learning advertising content using a recognition model, the present invention learns about facial expressions expressing emotions in addition to the user's facial expression composition based on the position or shape of the eyes, nose, and mouth of the face, and then detects emotions from emotional expressions expressing specific emotions in a photographing device, thereby enabling the user to express emotions that match the advertising intent of the advertising content as facial expressions, thereby allowing the user to imprint the advertising target.

[0023] The effects according to the present invention are not limited to those exemplified above, and more diverse effects are included within the present invention.

[0024] FIG. 1 is a block diagram showing the configuration of a photo shooting service system including an artificial intelligence-based facial expression utilization advertising function according to one embodiment of the present invention.

[0025] FIG. 2 is a block diagram showing the configuration of a photo shooting service providing device according to one embodiment of the present invention.

[0026] FIG. 3 is a schematic flowchart of a photo shooting service providing method performed in a photo shooting service providing device according to one embodiment of the present invention.

[0027] FIG. 4 is a schematic diagram illustrating a facial expression recognition model according to one embodiment of the present invention.

[0028] FIG. 5 is a block diagram showing the configuration of a photographing device including an artificial intelligence-based facial expression utilization advertising function according to one embodiment of the present invention.

[0029] FIG. 6 is a schematic flowchart of a method for providing a photo shooting service performed in a photo shooting service system according to one embodiment of the present invention.

[0030] FIG. 7a and FIG. 7b are exemplary diagrams showing an interface screen of a photographing device that provides a photo shooting service including an expression utilization advertising function according to one embodiment of the present invention.

[0031] FIG. 8 is a schematic flowchart of a method for providing a photo shooting service performed in a photo shooting service system according to another embodiment of the present invention.

[0032] FIGS. 9a, 9b, 9c, and 9d are exemplary diagrams showing interface screens of a user device that provides a photo shooting service including an expression utilization advertising function according to one embodiment of the present invention.

[0033] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms. These embodiments are provided solely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. In connection with the description of the drawings, similar reference numerals may be used for similar components.

[0034] In this document, the expressions "has," "may have," "includes," or "may include" indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), but do not exclude the presence of additional features.

[0035] In this document, the expressions "A or B," "at least one of A and / or B," or "one or more of A or / and B" can include all possible combinations of the listed items. For example, "A or B," "at least one of A and B," or "at least one of A or B" can all refer to cases where (1) at least one A is included, (2) at least one B is included, or (3) at least one A and at least one B are included.

[0036] The terms "first," "second," "first," or "second," as used herein, may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, without limiting the components. For example, a first user device and a second user device may represent different user devices, regardless of order or importance. For example, without departing from the scope of the rights set forth in this document, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0037] When it is said that a component (e.g., a first component) is "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., a second component), it should be understood that the component is directly coupled to the other component, or can be connected via another component (e.g., a third component). Conversely, when it is said that a component (e.g., a first component) is "directly coupled to" or "directly connected to" another component (e.g., a second component), it should be understood that no other component (e.g., a third component) exists between the first component and the other component.

[0038] The expression "configured to" as used herein can be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily mean something that is "specifically designed to" in hardware. Instead, in some contexts, the expression "a device configured to" can mean that the device, together with other devices or components, is "capable of." For example, the phrase "a processor configured (or set) to perform A, B, and C" may mean a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0039] The terms used in this document are used only to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include the plural expression unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by those of ordinary skill in the art described in this document. Terms defined in general dictionaries among the terms used in this document may be interpreted as having the same or similar meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in this document. In some cases, even if a term is defined in this document, it cannot be interpreted to exclude the embodiments of this document.

[0040] The individual features of the various embodiments of the present invention can be partially or wholly combined or combined with each other, and as can be fully understood by those skilled in the art, various technical connections and operations are possible, and each embodiment can be implemented independently of each other or can be implemented together in a related relationship.

[0041] For clarity in the interpretation of this specification, the terms used in this specification are defined below.

[0042] As used herein, the term "instant output image" may be an image captured by a camera device and immediately output. The instant output image may include at least one user with a facial expression. Depending on the shooting method, the instant output image may be composed of a single image of the same user or may be composed of multiple images. In addition, the instant output image may be mapped with identification data of the camera device that captured the image, date data regarding the shooting time, and identification data of the user included in the image.

[0043] As used herein, the term "advertising learning model" may be a model trained to detect user facial expressions or emotional expressions using advertising content as input. Specifically, the advertising content may be an image or video containing a product or event being promoted. The advertising learning model may detect emotional expressions in the image or video, defining facial expressions or images or videos that other users may imitate. Emotional expressions may be expressions expressing joy, surprise, sadness, anger, interest, boredom, fear, pain, etc., and the advertising learning model may detect user facial expressions or emotional expressions appropriate to the type of advertising event.

[0044] The term "expression recognition model" used herein may be a model trained to input an image and output whether or not an advertisement expression is included. Specifically, the image may be an instant output image acquired through a photographing device, and the expression recognition model may output a probability that the user's expression included in the instant output image corresponds to a predefined advertisement expression. In addition, the expression recognition model may output a probability that the user's expression included in the instant output image corresponds to a predefined advertisement-specific emotional expression. Meanwhile, the user's expression may be determined based on feature points included in the user's face, and for example, the feature points may include the eyes, nose, mouth, eyebrows, ears, etc. of the face.

[0045] In various embodiments, the advertising learning model and the facial expression recognition model may be composed of at least one model or two or more ensemble models of VGG net, R, DenseNet based on CNN (Convolutional Neural Network), FCN (Fully Convolutional Network) with encoder-decoder structure, SegNet, DeconvNet, DeepLAB V3+, DNN (deep neural network) such as U-net, SqueezeNet, Alexnet, ResNet18, MobileNet-v2, MobileNet-v3, GoogLeNet, Resnet50, Resnet101, Inception-v3, Efficientnet.

[0046] Hereinafter, the present invention will be described in detail by describing a preferred embodiment of the present invention with reference to the attached drawings.

[0047] FIG. 1 is a block diagram showing the configuration of a photo shooting service system including an artificial intelligence-based facial expression utilization advertising function according to one embodiment of the present invention.

[0048] Referring to FIG. 1, a photo shooting service system (1000) including an artificial intelligence-based facial expression utilization advertising function (hereinafter referred to as a “photo shooting service system”) may be a photo shooting service system that analyzes a user’s facial expression in an instantly captured photo (hereinafter referred to as an “instant output image”) through an artificial neural network model and provides a reward to the user based on whether the analyzed facial expression matches a predefined advertising facial expression. To this end, the photo shooting service system (1000) of the present invention may include a photo shooting service providing device (100), a photographing device (200), and a user device (300).

[0049] The photo shooting service providing device (100) is a server device of a service provider that analyzes a user's facial expression in an instant output image and determines whether to provide a reward based on the analysis result, and may include a general-purpose computer, a laptop, a data server, etc. Specifically, the photo shooting service providing device (100) may receive an instant output image including a user from a photographing device (200) or a user device (300). The photo shooting service providing device (100) may determine whether participation in an advertising event is possible based on at least one of identification data of the photographing device mapped to the instant output image, date data regarding the time of shooting, and identification data of the user included in the image. Here, whether participation in an advertisement is possible can be understood as a result of determining whether an advertisement is currently in progress through the photo shooting service system (1000), whether the provided instant output image was shot by another photographing device, or whether it is an instant output image shot a long time ago.

[0050] The photo shooting service providing device (100) can determine whether an advertisement expression is included in a valid instant output image as a result of the judgment. For example, the photo shooting service providing device (100) can input the instant output image into an expression recognition model and obtain a probability value indicating whether the user's expression present in the instant output image matches any one of a plurality of pre-specified advertisement expressions. The photo shooting service providing device (100) can determine whether the user's expression matches the advertisement expression based on the probability value. As another example, the photo shooting service providing device (100) can input the instant output image into an expression recognition model and obtain a probability that the user's expression present in the instant output image corresponds to a pre-specified advertisement-specific emotional expression.

[0051] The photo shooting service providing device (100) may provide a reward to the photo shooting device (200) or the user device (300) if an advertisement expression (an advertisement expression representing a predefined advertisement, an emotional expression for a predefined advertisement) is included in the instant output image. For example, the reward may be points usable in the photo shooting service. The photo shooting service providing device (100) may provide a coupon number or two-dimensional barcode that can provide a reward to the photo shooting device (200), or may provide a reward to a user account through an application or web page installed or executed on the user device (300).

[0052] In various embodiments, the reward provided by the photography service providing device (100) may further include a secret frame, an event raffle ticket, etc. Specifically, the photography service providing device (100) may transmit a reward selection request to the photography device (200) or the user device (300) to determine which reward to provide to the user among points, a secret frame, and an event raffle ticket. If the secret frame is selected as the reward, the photography service providing device (100) may provide an additional photography service to the photography device (200) or provide a coupon for using a paid photo frame to the user device (300).

[0053] In various embodiments, the photography service providing device (100) may receive an advertisement event participation request from any one of the photography devices (200) before receiving an instant output image. Here, the advertisement event participation request may be a participation request for any one of the advertisements or events, and the photography service providing device (100) may provide guidelines for the advertisement event to the photography device (200) according to the advertisement event participation request. Specifically, the photography service providing device (100) may receive an image that can be instantly output from the photography device (200) in real time. The photography service providing device (100) may provide guidelines for a face exposure range for recognition of an advertisement expression, a face rotation direction, whether facial features are exposed, and a change in size or direction based on the coordinates of the facial features to the photography device (200). For example, through the guidelines provided by the photo shooting service providing device (100), the photo shooting device (200) can visually and audibly output guidelines such as 'Please smile a little wider', 'Please rotate your face to the left', 'Please move back a little more for face recognition', and 'Please move your eyebrows up' on the image being captured. In addition, these guidelines can be provided to all multiple users in the image being captured by the photo shooting device (200). In addition, the photo shooting service providing device (100) can provide a probability value indicating whether an advertisement expression is included in an image that can be instantly printed by the photo shooting device (200). For example, the photo shooting service providing device (100) can provide the photo shooting device (200) with a probability value indicating whether the expression of the user included in the instantly printable image is similar to any one advertisement expression selected by the user.

[0054] In various embodiments, the photo shooting service providing device (100) may identify an advertisement target included in an advertisement event participation request from one of the photo shooting devices (200) before receiving an instant output image. The photo shooting service providing device (100) may use the attributes of the identified advertisement target to provide a background or object image that can be inserted into the instant output image to the photo shooting device (200). For example, if the identified advertisement target is a specific brand of coffee, the photo shooting device (200) may recommend a character image of the brand as an insertable graphic object, or may recommend a representative color of the brand as a frame color.

[0055] In various embodiments, the photo shooting service providing device (100) may receive a plurality of instant output images composed of a plurality of frames. The photo shooting service providing device (100) may segment each of the plurality of instant output images, input the segmented plurality of instant output images into an expression recognition model, and determine whether or not an advertisement expression is included in each image. The photo shooting service providing device (100) may provide a reward if an advertisement expression matching an advertisement event participation request is included in at least one of the plurality of instant output images.

[0056] In various embodiments, the photo service providing device (100) may designate a specific advertisement through an advertisement event participation request. For example, the photo service providing device (100) may receive an instant output image along with a specific advertisement event participation request from the photographing device (200) or the user device (300). In such a case, the facial expression recognition model may input the instant output image and output only whether the advertisement is included within the instant output image.

[0057] In various embodiments, the photography service providing device (100) may include various DB servers for providing a photography service including an advertising function. Specifically, the photography service providing device (100) may include a device DB server that stores identification data of the photography device (200), the operating status of each photography device (200) (e.g., normal, idle, error), a CMS (Content management system) DB server that stores various data (e.g., service UI, instant photography image frame, advertising event, payment data, etc.) related to the photography service provided through the photography device (200), a storage server that stores instant photography images captured by the photography device (200), a mobile DB server that stores user accounts of users using the photography service, etc.

[0058] The photographing device (200) may be a photographing device installed in an unmanned photo studio providing a self-photography service. The photographing device (200) may capture an instant output image including a user, and may provide the captured instant output image to the photo shooting service providing device (100) in response to the user's request for participation in an advertising event. For example, the request for participation in an advertising event provided by the photographing device (200) to the photo shooting service providing device (100) may include identification data of the photographing device (200), an instant output image, user identification data, and optionally, advertising event selection data.

[0059] In various embodiments, the photographing device (200) may provide an instant output image including the user to the user device (300) upon the user's request for provision of an instant output image. For example, after the instant output image is captured, the photographing device (200) may request user identification data (e.g., user phone number, ID, etc.) to transmit the instant output image, or output data for accessing a photo shooting service web page or application program that can download the instant output image.

[0060] In various embodiments, the photographing device (200) can perform the facial expression utilization advertising function of the photo shooting service providing device (100), and in such case, the photographing device (200) can determine whether to include an advertising facial expression in the user's instant output image without providing the instant output image to the photo shooting service providing device (100).

[0061] The user device (300) is a device possessed by a service user who uses the photo shooting service, and may include a smartphone, tablet PC, PC, laptop, etc. The user device (300) may install or execute a web or mobile application or program provided by the photo shooting service providing device (100) to output a user interface screen for requesting the photo shooting service. Here, the photo shooting service may include a service for downloading an instant output image captured by the photo shooting device (200), a service for participating in an advertising event through an instant output image, etc.

[0062] In various embodiments, the user device (300) may transmit at least one instant output image to the photo shooting service providing device (100) in response to a request for participation in an advertising event. For example, the user device (300) may execute a web service page or application to select one of a plurality of currently ongoing advertising events and upload an image captured by the photographing device (200). In addition, the user device (300) may receive a reward from the photo shooting service providing device (100) based on whether an advertising expression is included in the instant output image. For example, the user device (300) may receive a reward selection request from the photo shooting service providing device (100) to select one of points, a secret frame, and an event lottery ticket, and may receive one type of reward through the user account based on the user's selection.

[0063] The photo shooting service system (1000) according to one embodiment of the present invention has been described so far. According to the present invention, the photo shooting service system (1000) can reduce users' resistance to direct or participatory advertising promotion while increasing the advertising impression effect on the advertising target by utilizing facial expressions as an advertising medium.

[0064] Hereinafter, a photo shooting service providing device (100) that provides a photo shooting service will be described with reference to FIG. 2.

[0065] FIG. 2 is a block diagram showing the configuration of a photo shooting service providing device according to one embodiment of the present invention.

[0066] Referring to FIG. 2, the photo shooting service providing device (100) may include a communication interface (110), a memory (120), an I / O interface (130), and a processor (140), and each component may communicate with each other through one or more communication buses or signal lines.

[0067] The communication interface (110) can be connected to the photographing device (200) and the user device (300) via a wired / wireless communication network to exchange data. For example, the communication interface (110) can receive an instant output image from the photographing device (200) and the user device (300), and can transmit a reward to the photographing device (200) and the user device (300) according to whether the instant output image includes an advertisement expression. As another example, the communication interface (110) can receive an image that can be instantly output in real time from the photographing device (200), and can transmit a probability value indicating whether an advertisement expression is included in the image that can be instantly output to the photographing device (200).

[0068] Meanwhile, the communication interface (110) that enables transmission and reception of such data includes a wired communication port (111) and a wireless circuit (112), wherein the wired communication port (111) may include one or more wired interfaces, for example, Ethernet, Universal Serial Bus (USB), FireWire, etc. In addition, the wireless circuit (112) may transmit and receive data with an external device via an RF signal or an optical signal. In addition, the wireless communication may use at least one of a plurality of communication standards, protocols, and technologies, for example, GSM, EDGE, CDMA, TDMA, Bluetooth, Wi-Fi, VoIP, Wi-MAX, or any other suitable communication protocol.

[0069] The memory (120) can store various data used in the photo shooting service providing device (100). For example, the memory (120) can store identification data of the photographing device (200) and the user device (300), and user interface configuration data of a service provided through the photographing device (200) and the user device (300). As another example, the memory (120) can store an expression recognition model configured to output whether an advertisement expression is included by taking an image as input and its learning data, and an advertisement learning model configured to detect a predefined advertisement expression or a predefined advertisement-specific emotional expression by taking advertisement content as input and its learning data.

[0070] In various embodiments, the memory (120) may include a volatile or non-volatile storage medium capable of storing various data, commands, and information. For example, the memory (120) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., an SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, network storage, cloud, and blockchain database.

[0071] In various embodiments, the memory (120) may store configurations of at least one of an operating system (121), a communication module (122), a user interface module (123), and one or more applications (124).

[0072] An operating system (121) (e.g., embedded operating systems such as LINUX, UNIX, MAC OS, WINDOWS, VxWorks, etc.) may include various software components and drivers to control and manage general system operations (e.g., memory management, storage device control, power management, etc.) and may support communication between various hardware, firmware, and software components.

[0073] The communication module (123) can support communication with other devices through the communication interface (110). The communication module (120) can include various software components for processing data received by the wired communication port (111) or wireless circuit (112) of the communication interface (110).

[0074] The user interface module (123) can receive a user's request or input from a keyboard, touch screen, keyboard, mouse, microphone, etc. through an I / O interface (130) and provide a user interface on the display.

[0075] The application (124) may include a program or module configured to be executed by one or more processors (140). Here, the application for advertising service using instant output images may be implemented on a server farm.

[0076] The I / O interface (130) can connect at least one of input / output devices (not shown) of the photo shooting service providing device (100), such as a display, a keyboard, a touch screen, and a microphone, to the user interface module (123). The I / O interface (130) can receive user input (e.g., voice input, keyboard input, touch input, etc.) together with the user interface module (123) and process commands according to the received input.

[0077] The processor (140) is connected to a communication interface (110), a memory (120), and an I / O interface (130) to control the overall operation of the photo shooting service providing device (100), and can execute various commands for performing a photo shooting service including an artificial intelligence-based facial expression utilization function through an application or program stored in the memory (120).

[0078] The processor (140) may correspond to a computing device such as a CPU (Central Processing Unit) or an AP (Application Processor). Furthermore, the processor (140) may be implemented in the form of an integrated chip (IC), such as a SoC (System on Chip) in which various computing devices are integrated. Alternatively, the processor (140) may include a module for calculating an artificial neural network model, such as an NPU (Neural Processing Unit).

[0079] Hereinafter, with reference to FIGS. 3 and 4, a method for providing a photo shooting service by a processor (140) of a photo shooting service providing device (100) that incorporates an advertising event will be described.

[0080] FIG. 3 is a schematic flowchart of a photo shooting service providing method performed in a photo shooting service providing device according to one embodiment of the present invention.

[0081] Referring to FIG. 3, the processor (140) may receive at least one instant output image from the photographing device (200) or the user device (300) (S110). Specifically, the processor (140) may determine whether participation in an advertising event is possible based on at least one of identification data of the photographing device mapped to the instant output image, date data regarding the time of shooting, and user identification data included in the image. Here, whether participation in an advertising event is possible may be understood as a result of determining whether an advertisement is currently in progress through the photo shooting service system (1000), whether the provided instant output image was taken by another photographing device, or whether it is an instant output image taken a long time ago. As a result of the determination, the processor (140) may determine whether an advertising expression is included in a valid instant output image.

[0082] After step S110, the processor (140) inputs an instant output image into an expression recognition model configured to output whether an advertisement expression is included by taking an image as input and determine whether an advertisement expression is included in the instant output image (S120).

[0083] In relation to this, FIG. 4 is a schematic diagram for explaining a facial expression recognition model according to one embodiment of the present invention.

[0084] Referring to (a) of FIG. 4, the processor (140) inputs an instant output image (12) into an expression recognition model (11-1) configured to output whether an image contains an advertisement expression by taking the image as input, and obtains a probability value (13) indicating whether the user's expression matches any one of a plurality of pre-specified advertisement expressions.

[0085] Referring to (b) of FIG. 4, the processor (140) inputs an instant output image (14) into an expression recognition model (11-2) configured to output whether an emotional expression defined in an advertisement is included by taking an image as input, and obtains a probability (15) that the user's expression existing in the instant output image corresponds to an emotional expression of a predefined advertisement.

[0086] The processor (140) can determine which advertisement expression matches the user's facial expression based on the probability value output from the facial expression recognition model (11-1)(11-2). In addition, the processor (140) can also receive advertisement event selection data / information regarding which advertisement event the user wishes to participate in at the time of receiving the instant output image.

[0087] After step S120, if an advertisement expression is included in the instant output image, the processor (140) may provide a reward to the photographing device (200) or the user device (300) (S130). For example, the reward may be points usable in a photo shooting service. The processor (140)3 may provide a coupon number or two-dimensional barcode that can provide a reward to the photographing device (200), or may provide the reward to the user account through an application or web page installed or executed on the user device (300).

[0088] In various embodiments, the reward provided by the processor (140) may further include a secret frame, an event raffle ticket, etc. Specifically, the processor (140) may transmit a reward selection request to the camera device (200) or the user device (300) to determine which reward to provide to the user among points, a secret frame, and an event raffle ticket. If the secret frame is selected as the reward, the processor (140) may provide an additional photography service to the camera device (200) or provide a coupon for using a paid photo frame to the user device (300).

[0089] In various embodiments, the processor (140) may receive an advertisement event participation request from any one of the photographing devices (200) before receiving an instant output image. Here, the advertisement event participation request may be a participation request for any one of the advertisements or events, and the processor (140) may provide guidelines for the advertisement event to the photographing device (200) according to the advertisement event participation request. Specifically, the processor (140) may receive an image that can be instantly output in real time from the photographing device (200). The processor (140) may provide guidelines for a face exposure range for recognition of an advertisement expression, a face rotation direction, whether facial features are exposed, and a change in size or direction based on the coordinates of the facial features to the photographing device (200). For example, through the guidelines provided by the processor (140), the photographing device (200) can visually and audibly output guidelines such as 'Please smile a little wider', 'Please rotate your face to the left', 'Please move back a little more for face recognition', and 'Please move your eyebrows up' on the image being photographed. In addition, these guidelines can be provided to all multiple users in the image being photographed by the photographing device (200). In addition, the processor (140) can provide a probability value indicating whether an advertisement expression is included in an image that can be instantly printed by the photographing device (200). For example, the processor (140) can provide the photographing device (200) with a probability value indicating whether the expression of the user included in the instantly printable image is similar to any one advertisement expression selected by the user.

[0090] In various embodiments, the processor (140) may identify an advertising target included in an advertising event participation request from one of the photographing devices (200) prior to receiving an instant output image. The processor (140) may use the attributes of the identified advertising target to provide the photographing device (200) with a background or object image that can be inserted into the instant output image. For example, if the identified advertising target is a specific brand of coffee, the photographing device (200) may recommend a character image of the brand as an insertable graphic object, or a representative color of the brand may be recommended as a frame color.

[0091] In various embodiments, the processor (140) may receive a plurality of instant output images composed of a plurality of frames. The processor (140) may segment each of the plurality of instant output images, input the segmented plurality of instant output images into an expression recognition model, and determine whether each image includes an advertisement expression. The processor (140) may provide a reward if at least one of the plurality of instant output images includes an advertisement expression matching an advertisement event participation request.

[0092] In various embodiments, the processor (140) may designate a specific advertisement through an advertisement event participation request. For example, the processor (140) may receive an instant output image along with a specific advertisement event participation request from the camera device (200) or the user device (300). In this case, the facial expression recognition model may input the instant output image and output only whether the advertisement is included within the instant output image.

[0093] So far, a photo shooting service providing device (100) according to one embodiment of the present invention has been described, and below, with reference to FIG. 5, a photo shooting device (200) and a user device (300) in which such a photo shooting service is actually implemented will be described.

[0094] FIG. 5 is a block diagram showing the configuration of a photographing device including an artificial intelligence-based facial expression utilization advertising function according to one embodiment of the present invention.

[0095] Referring to FIG. 5, the photographing device (200) may include a communication interface (210), a memory (220), an I / O interface (230), and a processor (240), and each component may communicate with each other through one or more communication buses or signal lines.

[0096] The communication interface (210) can be connected to the photography service providing device (100) and the user device (300) via a wired / wireless communication network to exchange data. For example, the communication interface (210) can transmit an instant output image to the photography service providing device (100) and receive data indicating whether the instant output image includes an advertising expression or a reward depending on whether the instant output image includes an advertising expression from the photography service providing device (100). As another example, the communication interface (210) can receive user identification data from the user device (300) and transmit an instant output image or a reward depending on whether the instant output image includes an advertising expression to the user device (300).

[0097] Meanwhile, the communication interface (210) that enables transmission and reception of such data includes a wired communication port (211) and a wireless circuit (212), wherein the wired communication port (211) may include one or more wired interfaces, for example, Ethernet, Universal Serial Bus (USB), FireWire, etc. In addition, the wireless circuit (212) may transmit and receive data with an external device via an RF signal or an optical signal. In addition, the wireless communication may use at least one of a plurality of communication standards, protocols, and technologies, for example, GSM, EDGE, CDMA, TDMA, Bluetooth, Wi-Fi, VoIP, Wi-MAX, or any other suitable communication protocol.

[0098] The memory (220) can store various data used in the photographing device (200). For example, the memory (220) can store identification data and user interface configuration data for performing artificial intelligence-based facial expression-based advertising functions. As another example, the memory (220) can store an expression recognition model configured to input an image and output whether or not it contains an advertising expression.

[0099] In various embodiments, the memory (220) may include a volatile or non-volatile storage medium capable of storing various data, commands, and information. For example, the memory (220) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., an SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, network storage, cloud, and a blockchain database.

[0100] In various embodiments, the memory (220) may store at least one configuration of an operating system (221), a communication module (222), a user interface module (223), one or more applications (224), a payment module (225), a camera module (226), and a photo printing module (227).

[0101] An operating system (221) (e.g., embedded operating systems such as LINUX, UNIX, MAC OS, WINDOWS, VxWorks, etc.) may include various software components and drivers to control and manage general system operations (e.g., memory management, storage device control, power management, etc.) and may support communication between various hardware, firmware, and software components.

[0102] The communication module (223) can support communication with other devices through the communication interface (210). The communication module (220) can include various software components for processing data received by the wired communication port (211) or wireless circuit (212) of the communication interface (210).

[0103] The user interface module (223) can receive a user's request or input from a keyboard, touch screen, keyboard, mouse, microphone, etc. through an I / O interface (230) and provide a user interface on a display (231).

[0104] The application (224) may include a program or module configured to be executed by one or more processors (240). Here, the application for capturing an instant output image and providing an advertising service using the same may be implemented on a server farm.

[0105] The payment module (225) can perform a payment corresponding to a photo shooting service request based on a user request or input obtained through the I / O interface (230). The payment module (225) can perform contact or contactless payment. For example, the payment module (225) can perform payment using MST (Magnetic Secure Transmission), IC chip, NFC (Near Field Communication), RFID (Radio Frequency Identification), or EMV. In addition, the payment module (225) can perform payment corresponding to a photo shooting service request using cash or a 2D barcode.

[0106] The camera module (226) can capture instantaneous images and videos. The camera module (226) can capture images or videos including a user through a light sensor (232) placed on the front of the photographing device (200), and the image being captured can be displayed on the display (231).

[0107] The photo printing module (227) can generate at least one image from among a plurality of images captured by the camera module (226) according to a user's request or input obtained through the I / O interface (230), and can print the image through the printer (233). For example, the photo printing module (227) can generate an instant shooting image by combining a frame, background, and graphic object with an image or video captured by the camera module (226).

[0108] The I / O interface (230) can connect at least one of an input / output device (not shown) of the photographing device (200), such as a display (231), a keyboard, a touch screen, and a microphone, to the user interface module (223). The I / O interface (230) can receive user input (e.g., voice input, keyboard input, touch input, etc.) together with the user interface module (223) and process a command according to the received input.

[0109] In various embodiments, the I / O interface (230) may include a display (231), a light sensor (232), and a printer (233). The display (231) may display an image being captured by the camera module (226) and may detect user contact and movement or cessation of contact and movement using any one of a plurality of touch sensing technologies, such as capacitive, resistive, infrared, surface acoustic wave technology, or a proximity sensor array. In addition, the light sensor (232) may receive light from the surrounding environment projected through one or more lenses and convert the light into data representing the image. Accordingly, the light sensor (232) may capture a still image or video that includes the user. In addition, the printer (233) may print an instant capture image generated by the photo printing module (227) into an actual photograph.

[0110] The processor (240) is connected to a communication interface (210), a memory (220), and an I / O interface (230) to control the overall operation of the photographing device (200), and can execute various commands for performing a photo shooting service including an artificial intelligence-based facial expression utilization function through an application or program stored in the memory (220).

[0111] The processor (240) may correspond to a computing device such as a Central Processing Unit (CPU) or an Application Processor (AP). Furthermore, the processor (240) may be implemented in the form of an integrated chip (IC), such as a System on Chip (SoC) in which various computing devices are integrated. Alternatively, the processor (240) may include a module for calculating an artificial neural network model, such as a Neural Processing Unit (NPU).

[0112] In various embodiments, the processor (240) may obtain at least one image including a user's request for participation in an advertising event via the I / O interface (230). For example, the processor (240) may obtain a user's request for taking a photo and a request for participation in an event via the display (231). The processor (240) may display an image of the user obtained via the camera module (226) via the display (231), and may capture an image according to the user's request to obtain an instant output image including the user.

[0113] In various embodiments, the processor (240) may input an instant output image into a facial expression recognition model configured to output whether an advertisement expression is included in the instant output image by taking an image as input and determining whether the instant output image includes an advertisement expression. In addition, if the instant output image includes an advertisement expression, the processor (240) may provide a reward to a user matching the instant output image through the communication interface (210). For example, the processor (240) may provide an additional photo service using a secret frame as a reward through the display (231), or may provide points that can be used in a subsequent photo service to the user account.

[0114] So far, a photographing device (200) providing an AI-based facial expression utilization function according to one embodiment of the present invention has been described. According to the present invention, the photographing device (200) providing a self-photography service can induce a user to make a specific facial expression related to an advertisement target, and when the facial expression is detected, provide the user with a corresponding reward, thereby inducing active participation of the user.

[0115] Hereinafter, the overall operation method of the photo shooting service system (1000) will be described with reference to FIGS. 6 to 9.

[0116] FIG. 6 is a schematic flowchart of a method for providing a photo shooting service performed in a photo shooting service system according to one embodiment of the present invention.

[0117] Referring to FIG. 6, a method for providing a photo shooting service performed in a photo shooting service system according to one embodiment of the present invention is an embodiment of a method for providing a reward immediately after an instant output image is acquired through a photo shooting device (200). Specifically, the photo shooting service providing device (100) can provide an advertising event participation guideline to the photo shooting device (200) (S11). For example, when the photo shooting service providing device (100) receives a request to participate in a certain advertising event through the photo shooting device (200), the photo shooting service providing device (100) can provide an advertising event participation guideline.

[0118] In relation to this, FIGS. 7a and 7b are exemplary diagrams showing an interface screen of a photographing device that provides a photo shooting service including an expression utilization advertising function according to one embodiment of the present invention.

[0119] Referring to FIGS. 7A and 7B , the photographing device (200) can display a plurality of advertising events (21) currently in progress in the photo shooting service through the display (231). For example, the advertising events may include an event of imitating the facial expressions of a CF advertisement for a specific product, an event of making an emotional facial expression that fits a specific campaign, etc. When any one of the plurality of advertising events (21) is selected, the photographing device (200) can receive an object (22) that can be inserted into an instant output image from the photo shooting service providing device (100) and display it through the display (231). In addition, the photographing device (200) can receive a guide line (23) indicating a facial expression from the photo shooting service providing device (100) and display it through the display (231). In addition, the photographing device (200) can provide the user with guide text for changing the user's facial expression, position, or angle based on the real-time user image displayed on the display (231).

[0120] Referring back to FIG. 6, the photographing device (200) can capture an instant output image (S12) and transmit the instant output image together with a request for participation in an advertising event to the photographing service providing device (100) (S13). The photographing service providing device (100) can determine whether an advertising expression is included in the instant output image (S14). If an advertising expression is included, the photographing service providing device (100) can determine through which device the reward will be provided via the photographing device (200). The photographing service providing device (100) can provide various types of rewards to the photographing device (200) or the user device (300) according to the user's selection via the photographing device (200) (S15).

[0121] Next, FIG. 8 is a schematic flowchart of a method for providing a photo shooting service performed in a photo shooting service system according to another embodiment of the present invention.

[0122] Referring to FIG. 8, a method for providing a photo shooting service performed in a photo shooting service system according to another embodiment of the present invention is an embodiment of a method for providing a reward based on an instant image provided by a user device (300). Specifically, the photographing device (200) can capture an instant output image and provide it to the user device (300) (S21). The user device (300) can store the instant output image and transmit a request for participation in an advertising event including the instant output image to the photo shooting service providing device (100) (S22).

[0123] In relation to this, FIGS. 9a, 9b, 9c and 9d are exemplary diagrams showing an interface screen of a user device that provides a photo shooting service including an expression utilization advertising function according to one embodiment of the present invention.

[0124] Referring to FIG. 9A, the user device (300) can install and execute an application provided by the photo shooting service providing device (100). The interface screen executed on the user device (300) may include information (31) regarding the user account and an advertising event (32) in which the user can participate. When the user selects the advertising event (32), the user device (300) may display an interface screen including an area (33) for uploading an instant output image, as shown in FIG. 9B, and an instant output image (34) uploaded by the user may be displayed in one area. In addition, when the user selects the upload button (35), the user device (300) may transmit a request for participation in an advertising event including an instant output image to the photo shooting service providing device (100).

[0125] The photo shooting service providing device (100) can determine whether participation in an advertising event using an instant output image transmitted from the user device (300) is possible (S23). For example, the photo shooting service providing device (100) can determine whether the instant output image transmitted from the user device (300) was taken by another photographing device or is an instant output image taken a long time ago. As a result of the determination, the photo shooting service providing device (100) can determine whether an advertising expression is included in the valid instant output image (S24). If an advertising expression is included, the photo shooting service providing device (100) can confirm what type of reward will be provided through the user device (300). For example, the photo shooting service providing device (100) can provide points usable in the photo shooting service as a reward to the user device (300), as shown in FIG. 9d.

[0126] The present invention has been described above with respect to a photography service system according to one embodiment of the present invention. According to the present invention, by detecting emotions from emotional expressions expressing specific emotions in a photographic device, a user can express emotions consistent with the advertising intent of the advertising content through facial expressions, thereby allowing the user to identify the advertising target.

[0127] Although the embodiments of the present invention have been described in more detail with reference to the attached drawings, the present invention is not necessarily limited to these embodiments, and various modifications may be implemented without departing from the technical spirit of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain it, and the scope of the technical spirit of the present invention is not limited by these embodiments. Therefore, it should be understood that the embodiments described above are illustrative in all aspects and not restrictive. The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of the rights of the present invention.

Claims

1. A method performed by a processor of a photo shooting service providing device, A step of receiving at least one instant output image in response to a request for participation in an advertising event from a shooting device or a user device; A step of inputting the instant output image into an expression recognition model configured to output whether an advertisement expression is included or not by taking an image as input and determining whether an advertisement expression is included or not in the instant output image; and A method for providing a photo shooting service including an artificial intelligence-based facial expression utilization function, comprising: a step of providing a reward to the photographing device or the user device when an advertising facial expression is included in the instant output image; 2. In paragraph 1, The above facial expression recognition model is, A method for providing a photo shooting service including an artificial intelligence-based facial expression utilization function, wherein the model is configured to output a probability that the user's facial expression included in the instant output image corresponds to a predefined advertisement facial expression.

3. In paragraph 1, The above facial expression recognition model is, A method for providing a photo shooting service including an artificial intelligence-based facial expression utilization function, wherein the model is configured to output a probability that the user's emotion included in the image corresponds to a predefined emotional expression for each advertisement.

4. In paragraph 2 or 3, A method for providing a photo shooting service including an artificial intelligence-based facial expression utilization function, further comprising a step of determining an advertisement facial expression or emotional expression to be recognized by the facial expression recognition model using an advertisement learning model learned to detect a user facial expression or emotional expression by type of advertisement event by inputting advertisement content.

5. In paragraph 1, Further comprising a step of providing guidelines for an advertising event using the above shooting device, The above guidelines are: A method for providing a photo shooting service including an artificial intelligence-based facial expression utilization function, which includes at least one item among a face exposure range for recognition of the above-mentioned advertising facial expression, a face rotation direction, whether facial feature points are exposed, and a change in size or direction based on the coordinates of facial feature points.

6. In paragraph 5, The step of receiving the above image is: A step of receiving an image that can be output in real time from the above photographing device, The step of determining whether or not the above advertisement expression includes an expression is as follows: A method for providing a photo shooting service including an artificial intelligence-based facial expression utilization function, further comprising a step of providing a probability indicating whether an advertising facial expression is included in the instantly outputtable image using the above photographing device.

7. In paragraph 1, The step of receiving the above image is: A step of identifying an advertising target included in the advertising event participation request from the above shooting device, and A method for providing a photography service including an artificial intelligence-based facial expression utilization function, further comprising a step of providing a background or object image that can be inserted into an instant output image using the photographing device by using the attributes of the identified advertising target.

8. In paragraph 1, The step of receiving the above image is: A step of receiving multiple instantaneous output images composed of multiple frames, The steps for providing the above rewards are: A method for providing a photo shooting service including an artificial intelligence-based facial expression utilization function, wherein the step is to provide a reward when an advertisement expression matching the advertisement event participation request is included in at least one instant output image among the plurality of instant output images.

9. Communication interface; memory; and a processor operably connected to the communication interface and the memory; The above processor, A photo shooting service providing device including an artificial intelligence-based facial expression utilization function, configured to receive at least one instant output image in response to a request for participation in an advertising event from a shooting device or a user device, input the instant output image into an expression recognition model configured to output whether the image contains an advertising expression as input, determine whether an advertising expression is included in the instant output image, and provide a reward to the shooting device or the user device if an advertising expression is included in the instant output image.

10. In paragraph 9, The above facial expression recognition model is, A photo shooting service providing device including an artificial intelligence-based facial expression utilization function, wherein the model is configured to output a probability that the user's facial expression included in the instant output image corresponds to a predefined advertisement facial expression.

11. In paragraph 9, The above facial expression recognition model is, A device for providing a photo shooting service including an artificial intelligence-based facial expression utilization function, wherein the model is configured to output a probability that the user's emotion included in the image corresponds to a predefined emotional expression for each advertisement.

12. In paragraph 10 or 11, The above processor, A photo shooting service providing device including an artificial intelligence-based facial expression utilization function further configured to determine an advertisement facial expression or emotional expression to be recognized by the facial expression recognition model by using an advertisement learning model learned to detect a user facial expression or emotional expression by type of advertisement event by inputting advertisement content.

13. In paragraph 10, The above processor, Further configured to provide guidelines for advertising events with the above shooting device, The above guidelines are: A device for providing a photo shooting service including an artificial intelligence-based facial expression utilization function, which includes at least one item among a face exposure range for recognition of the above-mentioned advertising facial expression, a face rotation direction, whether facial feature points are exposed, and a change in size or direction based on the coordinates of facial feature points.

14. In paragraph 13, The above processor, A photo shooting service providing device including an artificial intelligence-based facial expression utilization function, which is further configured to receive an image that can be instantly printed in real time from the above-mentioned shooting device and to provide a probability indicating whether an advertising facial expression is included in the image that can be instantly printed by the above-mentioned shooting device.

15. In paragraph 10, The above processor, A photo shooting service providing device including an artificial intelligence-based facial expression utilization function, further configured to identify an advertisement target included in the advertisement event participation request from the photographing device and provide a background or object image that can be inserted into an instant output image using the attributes of the identified advertisement target.

16. In paragraph 10, The above processor, Receive multiple instantaneous output images composed of multiple frames, A photo shooting service providing device including an artificial intelligence-based facial expression utilization function configured to provide a reward when an advertising facial expression matching the advertising event participation request is included in at least one instant output image among the plurality of instant output images.

17. Communication interface; I / O interface; memory; and a processor operably connected to the communication interface, the I / O interface, and the memory; The above processor, A photographing device providing an artificial intelligence-based facial expression utilization function, which obtains at least one instant output image including a user's request for participation in an advertising event through the I / O interface, inputs the instant output image into an expression recognition model configured to output whether the image contains an advertising expression as input, determines whether an advertising expression is included in the instant output image, and provides a reward to a user matching the instant output image through the communication interface if the image contains an advertising expression.

Images