Behavior determination device, behavior determination method, program, and recording medium

Abstract

To provide an action assessment device with which it is possible to assess whether a user has taken a specific action.SOLUTION: In an action assessment device 10, a setting unit 11 sets recommended action information that includes primary acceptance label information. The recommended action information indicates an action recommended to a user and an indication to capture an image associated with the action. The primary acceptance label information indicates the object to be imaged that is associated with the action recommended to the user and includes a threshold for a certainty factor with respect to the label information. An acquisition unit 12 acquires the image imaged by the user. A detection unit 13 inputs the acquired image to a trained model and detects the label information of the imaging object included in the image and the certainty factory with respect to the label information. An assessment unit 14 assesses the image as being "accepted" when the label information matches the primary acceptance label information and the certainty factor with respect to the label information satisfies a threshold for the primary acceptance label information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a behavior determination device, a behavior determination method, a program, and a recording medium. [Background technology]

[0002] It is important to encourage specific actions of users in various fields such as revitalization of local economies and health management. For example, Patent Document 1 reports a technology for encouraging users to exercise. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2013-090758 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology of Patent Document 1 and the like has a problem in that it is not clear whether the user has actually taken a specific action.

[0005] Therefore, an object of the present invention is to provide a behavior determination device, a behavior determination method, a program, and a recording medium that are capable of determining whether a user has performed a specific behavior. [Means for solving the problem]

[0006] In order to achieve the above object, the behavior determination device of the present invention comprises: A setting unit, an acquisition unit, a detection unit, and a determination unit are included, The setting unit sets recommended action information including first-stage pass label information, the recommended action information is information indicating an action recommended for the user and an instruction to capture an image related to the action, The first pass label information is label information indicating an imaging target related to an action recommended for a user, The acquisition unit acquires an image captured by a user, The detection unit inputs the acquired image into a trained model to detect label information of an imaging object contained in the image, The determination unit determines whether the image is “passed” or not based on the label information and the label information of the first pass label information, "Passed" means that the device performed the action indicated by the recommended action information and captured an image related to the action.

[0007] The behavior determination method of the present invention includes the steps of: The method includes a setting step, an acquisition step, a detection step, and a determination step, The setting step sets recommended action information including first pass label information; the recommended action information is information indicating an action recommended for the user and an instruction to capture an image related to the action, The first pass label information is label information indicating an imaging target related to an action recommended for a user, The acquiring step acquires an image captured by a user, The detection step includes inputting the acquired image into a trained model to detect label information of an imaging object contained in the image, The judging step judges whether the image is “passed” or not based on the label information and the label information of the first pass label information, "Pass" means that the action indicated by the recommended action information is executed and an image related to the action is captured. Effect of the Invention

[0008] According to the present invention, it is possible to determine whether a user has performed a particular action. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram illustrating a configuration of an example of a behavior determining device according to the first embodiment. [Diagram 2]FIG. 2 is a block diagram illustrating an example of a hardware configuration of the behavior determining device according to the first embodiment. [Diagram 3] FIG. 3 is a flowchart illustrating an example of processing in the behavior determining device of the first embodiment. [Figure 4A] FIG. 4A is a schematic diagram showing an example of adding secondary pass label information. [Figure 4B] FIG. 4B is a schematic diagram showing an example of adding secondary pass label information. [Figure 4C] FIG. 4C is a schematic diagram showing an example of adding secondary pass label information. [Figure 5A] FIG. 5A is a schematic diagram showing an example in which secondary pass label information is not added. [Figure 5B] FIG. 5B is a schematic diagram showing an example in which secondary pass label information is not added. [Figure 5C] FIG. 5C is a schematic diagram showing an example in which secondary pass label information is not added. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] In the behavior determination device of the present invention, for example, The first pass label information includes a confidence threshold for the label information, The certainty is an index indicating a likelihood that the image is the subject, The detection unit further detects the confidence level for the label information, The determination unit may further determine the image as "pass" when the certainty factor for the label information satisfies a threshold value of the primary pass label information.

[0011] The behavior determination device of the present invention includes, for example, Further, the recording unit and the extracting unit are included, the recording unit accumulates and records, as execution history data, label information of an object to be captured that is included in an image captured by the user and a result of the determination by the determination unit, in association with each other; The extraction unit extracts, from the execution history data, label information of the image for which the result of the judgment is “pass” other than the label information that matches the first pass label information; the setting unit adds the label information that satisfies a preset condition from the extracted label information to the recommended action information as secondary pass label information, and sets the label information; The determination unit determines the label information based on the label information and the label information of the first pass label information, and It may be an aspect in which the image is judged as "pass" when at least one of the judgments based on the label information and the label information of the secondary pass label information is judged as "pass".

[0012] In the behavior determination device of the present invention, for example, The extraction section may extract the label information when the number of the images determined to be "passed" in the execution history data is equal to or greater than a preset number.

[0013] In the behavior determination device of the present invention, for example, In the execution history data, label information other than the label information that matches the first pass label information is duplicated, The setting unit may calculate one of the average, median, or mode of the certainty levels from the certainty levels for each of the overlapping label information, and add the calculated numerical value to the imaging instruction information as the certainty level for the secondary pass label.

[0014] In the behavior determination device of the present invention, for example, The condition preset in the setting unit may be at least one of the following (1) and (2). (1) The execution history data contains overlapping label information other than the label information that matches the initially accepted label information, and at least one of the average, median, and mode calculated from the certainty of each of the overlapping label information is equal to or greater than a preset value. (2) In the execution history data, label information other than the label information that matches the initially approved label information is duplicated, and the number of occurrences of the duplicated label information in the execution history data is equal to or greater than a preset number.

[0015] The behavior determination device of the present invention includes, for example, Further, the system includes a recommended behavior information presenting unit and a token granting unit, the recommended action information presentation unit presents the recommended action information to the user in association with token information; The token information is information regarding a token granted to a user upon completion of the action, The token granting unit may grant, to a user, token information linked to the recommended behavior information, when the determination unit determines that the user is “passed.”

[0016] In the behavior determination method of the present invention, for example, The first pass label information includes a confidence threshold for the label information, The certainty is an index indicating a likelihood that the image is the subject, The detection step further includes detecting the confidence level for the label information, The judging step may further include judging the image as "pass" when the certainty factor for the label information satisfies a threshold value of the primary pass label information.

[0017] The behavior determination method of the present invention includes, for example, Further, the method includes a recording step and an extracting step, The recording step includes accumulating and recording, as execution history data, label information of an object to be captured that is included in an image captured by the user and a result of the determination step in association with each other; The extraction step extracts, from the execution history data, label information of the image for which the result of the judgment is “pass” other than the label information that matches the first pass label information; the setting step includes adding, to the recommended action information, label information that satisfies a preset condition from among the extracted label information as secondary pass label information; The determination step includes the step of: determining based on the label information and the label information of the first pass label information; and It may be an aspect in which the image is judged as "pass" when at least one of the judgments based on the label information and the label information of the secondary pass label information is judged as "pass".

[0018] In the behavior determination method of the present invention, for example, The extraction step may be configured to extract the label information when the number of images determined to be "passed" in the execution history data is equal to or greater than a preset number.

[0019] In the behavior determination method of the present invention, for example, In the execution history data, label information other than the label information that matches the first pass label information is duplicated, The setting process may include calculating one of the average, median, or mode of the certainty degrees from the respective certainty degrees for the overlapping label information, and adding the calculated numerical value to the imaging instruction information as the certainty degree for the secondary pass label.

[0020] In the behavior determination method of the present invention, for example, The condition preset in the setting step may be at least one of the following (1) and (2). (1) The execution history data contains overlapping label information other than the label information that matches the initially accepted label information, and at least one of the average, median, and mode calculated from the certainty of each of the overlapping label information is equal to or greater than a preset value. (2) In the execution history data, label information other than the label information that matches the initially approved label information is duplicated, and the number of occurrences of the duplicated label information in the execution history data is equal to or greater than a preset number.

[0021] The behavior determination method of the present invention includes, for example, Further, the system includes a recommended action information presenting step and a token granting step, The recommended action information presenting step includes presenting the recommended action information to a user in association with token information; The token information is information regarding a token granted to a user upon completion of the action, The token granting step may be configured to grant, to the user, token information linked to the recommended behavior information, when the result of the determination step is “passed.”

[0022] The program of the present invention is a program for causing a computer to execute each step of the method of the present invention as a procedure.

[0023] The recording medium of the present invention is a computer-readable recording medium having the program of the present invention recorded thereon.

[0024] The application field of this invention is any field in which specific actions are recommended to users, in other words, fields in which the actions that users should take toward goals are managed and the actions that users have already taken are confirmed, such as tourism, revitalization of local economies, health management, employee management, and education.

[0025] Next, an embodiment of the present invention will be described with reference to the drawings. The present invention is not limited to the following embodiment. In each of the drawings, the same parts are given the same reference numerals. In addition, the explanations of each embodiment can be mutually incorporated unless otherwise specified, and the configurations of each embodiment can be combined unless otherwise specified.

[0026] [Embodiment 1] Fig. 1 is a block diagram showing an example of the configuration of a behavior determination device 10 according to the present embodiment. As shown in Fig. 1, the device 10 includes a setting unit 11, an acquisition unit 12, a detection unit 13, and a determination unit 14. The device 10 may further include, as optional components, a recording unit 15, an extraction unit 16, a recommended behavior information presentation unit 17, and a token granting unit 18.

[0027] The device 10 may be, for example, a single device including each of the above-mentioned units, or a device to which each of the above-mentioned units can be connected via a communication line network. The device 10 may be connected to an external device described later via the communication line network. The communication line network is not particularly limited, and a known network may be used, for example, wired or wireless. Examples of the communication line network include an Internet line, a World Wide Web (WWW), a telephone line, a Local Area Network (LAN), a Storage Area Network (SAN), a Delay Tolerant Networking (DTN), a Low Power Wide Area (LPWA), and a Local 5G (L5G). Examples of wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), a Local 5G, and a LPWA. Examples of the wireless communication may include a form in which each device directly communicates with each other (Ad Hoc communication), an infrastructure (infrastructure communication), and an indirect communication via an access point. The device 10 may be incorporated into a server as a system, for example. The device 10 may be, for example, a personal computer (PC, for example, desktop type or notebook type) on which the program of the present invention is installed, a smartphone, a tablet terminal, etc. Furthermore, the device 10 may be in the form of cloud computing or edge computing, for example, in which at least one of the units is on a server and the other units are on a terminal.

[0028] 2 illustrates a block diagram of the hardware configuration of the device 10. The device 10 includes, for example, a central processing unit (CPU, GPU, etc.) 101, a memory 102, a bus 103, a storage device 104, an input device 105, a display device 106, and a communication device 107. The components of the device 10 are connected to each other via the bus 103 and their respective interfaces (I / F).

[0029] The central processing unit 101 is responsible for the overall control of the device 10. In the device 10, the central processing unit 101 executes, for example, the program of the present invention and other programs, and also reads and writes various information. Specifically, for example, the central processing unit 101 functions as a setting unit 11, an acquisition unit 12, a detection unit 13, and a determination unit 14. The central processing unit 101 also functions as, for example, a recording unit 15, an extraction unit 16, a recommended behavior information presentation unit 17, and a token granting unit 18, etc.

[0030] The bus 103 can also be connected to, for example, an external device. Examples of the external device include an external storage device (external database, etc.), a printer, an external input device, an external display device, an external imaging device, etc. The present device 10 can be connected to an external network (the above-mentioned communication line network) by, for example, a communication device 107 connected to the bus 103, and can also be connected to other devices via the external network.

[0031] An example of the memory 102 is a main memory (primary storage device). When the central processing unit 101 performs processing, the memory 102 reads various operation programs, such as the program of the present invention, stored in the storage device 104 described below, and the central processing unit 101 receives data from the memory 102 and executes the program. The main memory is, for example, a RAM (random access memory). The memory 102 may also be, for example, a ROM (read only memory).

[0032] The storage device 104 is also called an auxiliary storage device, for example, in contrast to the main memory (primary storage device). As described above, the storage device 104 stores an operation program including the program of the present invention. The storage device 104 may be, for example, a combination of a recording medium and a drive for reading and writing from and to the recording medium. The recording medium is not particularly limited, and may be, for example, an internal or external type, such as a hard disk (HD), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, or memory card. The storage device 104 may be, for example, a hard disk drive (HDD) in which a recording medium and a drive are integrated, or a solid state drive (SSD).

[0033] In the present device 10, the memory 102 and the storage device 104 can store various information such as log information, information acquired from an external database (not shown) or an external device, information generated by the present device 10, and information used when the present device 10 executes processing. At least a part of the information may be stored, for example, in an external server other than the memory 102 and the storage device 104, or may be stored in a distributed manner in multiple terminals using block chain technology or the like. The memory 102 and the storage device 104 may also store various information recorded by the recording unit 15, for example.

[0034] The device 10 may further include, for example, an input device 105 and a display device 106. The input device 105 is, for example, a touch panel, a keyboard, a mouse, etc. The display device 106 is, for example, an LED display, a liquid crystal display, etc.

[0035] Next, an example of the behavior determination method of this embodiment will be described with reference to the flowchart of Fig. 3. The behavior determination method of this embodiment is implemented as follows, for example, using the behavior determination device 10 of Fig. 1. Note that the behavior determination method of this embodiment is not limited to use of the behavior determination device 10 of Fig. 1. Also, the steps in parentheses in Fig. 3 are optional steps.

[0036] First, the setting unit 11 sets recommended action information including the first pass label information (S11a, setting step). The recommended action information is information indicating an action recommended to the user. The action is not particularly limited, and may be, for example, actions related to shopping, exercise, sightseeing, watching, learning, etc. The first pass label information is label information indicating an imaging object related to an action recommended to the user, and may include, for example, a threshold value of a certainty for the label information. The label information may be, for example, a character string such as a word, or identification information such as a number, an alphabet, or a combination of any two or more of them, but is information specific to the imaging object. The imaging object is not particularly limited. The certainty is an index indicating the likelihood that the imaging object is the imaging object. There may be a plurality of label information of the first pass label information, and in this case, the certainty threshold may be set for each label information. The setting unit 11 performs the setting according to, for example, information inputted via the input device 105. The recommended action information may be transmitted to the outside, for example, via the communication line network.

[0037] Next, the acquisition unit 12 acquires an image captured by the user (S12, acquisition step). The user is, for example, a user to whom the recommended action information is presented. That is, the image is presumed to be an image related to the action captured according to the recommended action information. The user acquires the recommended action information and captures the image, for example, using the user's terminal (smartphone, etc.). The acquisition unit 12 acquires the image from the user's terminal (smartphone, etc.) via, for example, the communication line network. The image may be, for example, a video composed of a plurality of frames (images). The acquisition unit 12 may acquire, for example, location information of the user (user's terminal), identification information of the terminal that captured the image, identification information of the terminal that transmitted the image to the device 10, etc., in association with the image. The acquisition unit 12 may also acquire, for example, the recommended action information acquired (performed) by the user in association with the image.

[0038] Next, the detection unit 13 inputs the acquired image into a trained model to detect label information of the imaging object contained in the image (S13, detection step). The detection unit 13 may, for example, input the acquired image into a trained model to detect label information of the imaging object contained in the image and the certainty factor for the label information. The label information detected by the detection unit 13 is also referred to as a detected label, for example. The trained model may be a model trained to detect label information of the imaging object contained in the image from an input image, or may be a model trained to detect label information of the imaging object contained in the image from an input image and the certainty factor for the label information. The trained model may be stored in the memory 102 and the storage device 104 in advance, or may be acquired from an external device via a communication line network. In addition, the detection unit 13 may use a trained model held by the external device, for example, without acquiring the trained model from an external device. A plurality of pieces of label information may be detected, and the certainty factor may be detected for each piece of label information. As the trained model, for example, a trained model provided by the Vision API of Google Cloud Platform can be used.

[0039] The trained model includes, for example, an input layer for inputting an image, an output layer for outputting label information of the imaging target (and the confidence level for the label information), and at least one intermediate layer provided between the input layer and the output layer. The trained model may be a program module that is a part of artificial intelligence software. An example of the multi-layered network is a neural network. An example of the neural network is a convolution neural network (CNN), but is not limited to CNN, and the trained model may be a neural network other than CNN, a support vector machine (SVM), a Bayesian network, a regression tree, or another learning algorithm.

[0040] Then, the judgment unit 14 judges whether the image is "passed" or not based on the label information and the label information of the first pass label information (S14a, judgment step), and ends the process (END). The judgment unit 14 may further judge the image as "passed" if the confidence level for the detected label information satisfies the threshold value of the first pass label information, for example. Note that "passed" means that the user has taken the action indicated by the previous recommended action information and captured an image related to the action, but the term is not limited to this and may be "OK", "mission successful", etc. Specifically, the judgment unit 14 may judge the image as "passed" if the detected label information matches the label information of the first pass label information, for example. Note that the match is not limited to a perfect match, and may be, for example, a partial match, or may be determined as a "match" depending on the rate of match (hereinafter, the same applies to "match"). The rate is not particularly limited and can be set arbitrarily. When there are multiple pieces of label information in the first pass label information, the judgment unit 14 may judge the image as a "match" if at least one of the multiple pieces of label information is satisfied, or may judge the image as a "match" if all of the multiple pieces of label information are satisfied. The first pass label information is, in other words, information for judging the image as a "pass".

[0041] On the other hand, the judgment unit 14 may judge the image to be "failed" when, for example, the detected label information does not match the label information of the first pass label information, or the confidence level for the label information does not satisfy the threshold value of the first pass label information, etc. Note that "failed" means, for example, that the user performed the action indicated by the recommended action information but was unable to capture an image related to the action, or was unable to perform the action indicated by the recommended action information, but the term is not limited thereto and may be "NG", "mission failed", etc.

[0042] The setting unit 11 may further set recommended behavior information including prohibited label information. The prohibited label information is, for example, label information indicating an image target that should not be included in an image, and may include, for example, a threshold value of confidence for the label information. The determination unit 14 may determine the image as "failed" based on, for example, the detected label information and the label information of the prohibited label information (for example, when the detected label information matches the label information of the prohibited label information, etc.). The determination unit 14 may further determine the image as "failed" when, for example, the confidence for the label information satisfies the threshold value of the prohibited label information. That is, the prohibited label information is, in other words, information for determining the image as "failed". Note that the determination unit 14 does not necessarily determine the image as "passed" when, for example, the detected label information does not match the label information of the prohibited label information or the confidence for the label information does not satisfy the threshold value of the prohibited label information.

[0043] Furthermore, the primary pass label information may be information including, for example, the position information of the imaging target. Then, the judgment unit 14 may, for example, compare the user's position information linked to the image with the position information of the imaging target in the primary pass label information, and judge the image as "failed" if they do not match. Note that, for example, if the user's position information linked to the image matches the position information of the imaging target, which is part of the imaging instruction information, the judgment unit 14 does not necessarily judge the image as "passed."

[0044] Furthermore, the determination unit 14 may, for example, compare the identification information of the terminal that captured the image with the identification information of the terminal that transmitted the image to the device 10, and if they do not match, determine that the image is "failed." Note that, for example, if the identification information of the terminal that captured the image matches the identification information of the terminal that transmitted the image to the device 10, the determination unit 14 does not necessarily determine that the image is "passed."

[0045] As described above, the present device 10 may include the recording unit 15 and the extraction unit 16. For example, after the step (S14a), the recording unit 15 associates and accumulates the label information of the imaging target included in the image captured by the user with the result of the judgment by the judgment unit 14 as execution history data (S15, recording step). In addition, the recording unit 15 may further associate and accumulate the confidence level for the label information as the execution history data, for example. The execution history data may be stored in the memory 102 and the storage device 104, or may be stored in an external database. In addition, the present device 10 may further include a database for storing the execution history data, and store the execution history data in the database.

[0046] After executing the step (S15), for example, the extraction unit 16 extracts, from the execution history data, label information (hereinafter also referred to as label candidates) other than the label information that matches the first pass label information and is for the images for which the judgment result is "pass" (S16, extraction step). In other words, the label candidates are label information that are not set as the first pass label information. For example, the extraction unit 16 may extract the label candidates (step (S16)) when the number of the images judged to be "pass" in the execution history data is equal to or greater than a preset number. The number is not particularly limited and can be set arbitrarily.

[0047] After executing the step (S16), for example, the setting unit 11 adds the label candidate that satisfies a preset condition from among the extracted label candidates to the recommended behavior information as label information of the secondary passed label information and sets it (S11b, setting step). That is, it can be said that the secondary passed label information is, for example, label information indicating an imaging target different from the imaging target indicated by the label information of the primary passed label information. The secondary passed label information may include, for example, an arbitrarily set threshold value of the confidence level. For example, a preset numerical value (for example, 50, etc.) may be assigned as the confidence level of the secondary passed label information. For example, the label candidates may overlap in the execution history data. When the label candidates overlap, the confidence level for the secondary passed label may be, for example, any one of an average value, a median value, or a mode value calculated from the confidence levels for the overlapping label candidates.

[0048] The above conditions are not particularly limited and can be set arbitrarily. For example, the above conditions include at least one of the following (1) and (2). (1) The label candidates overlap in the execution history data, and at least one of the mean, median, and mode calculated from the certainty of each of the overlapping label candidates is equal to or greater than a predetermined value (e.g., 50). (2) The label candidates are duplicated in the execution history data, and the number of occurrences of the duplicated label candidates in the execution history data is a preset number (for example, 40 or more) or more.

[0049] The condition may also include, for example, an upper limit on the number of secondary passed label information to be added. For example, if the upper limit on the number of secondary passed label information to be added is "1", the label information with the highest confidence (which may be an average value calculated from the confidences of the overlapping label candidates) among the multiple label candidates that satisfy the condition is added as the label information of the secondary passed label information.

[0050] After executing the step (S11b), for example, if the judgment unit 14 judges the image as "pass" in at least one of the judgments (1) to (4) below, the image is judged as "pass" (S14b, judgment step), and the process ends (END). (1) Determination based on the detected label information and the label information of the first pass label information (for example, when the detected label information matches the label information of the first pass label information) (2) In addition to the above (1), the certainty factor for the detected label information satisfies a certainty factor threshold for the first pass label information. (3) Determination based on the detected label information and the label information of the second pass label information (for example, when the detected label information and the label information of the second pass label information match). (4) In addition to the above (3), the certainty factor for the detected label information satisfies a certainty factor threshold for the secondary pass label information. In other words, even if the detected label information is judged to be "failed" (e.g., does not match) in a judgment based on the label information of the first passed label information, it should be judged to be "passed" (e.g., matches) in a judgment based on the label information of the second passed label information.

[0051] As described above, the device 10 may include a recommended behavior information presenting unit 17 and a token granting unit 18. For example, after the step (S11a) and before the step (S12), the recommended behavior information presenting unit 17 presents the recommended behavior information to the user by linking it with token information (S17, recommended behavior information presenting step). The token information is information about a token granted to the user upon completion of the behavior. The token information may be, for example, points, coins, tickets, etc. having monetary value. "Points, coins, tickets, etc. having monetary value" means, for example, that points, etc. can be converted into actual money, can be used as money (for example, 1 point = 1 yen, etc.), can be exchanged for products, etc. The token information having monetary value may be, for example, available only in a limited area. This can promote the revitalization of economic activity in the area. In addition, by making the tickets, etc., into tickets to watch sports teams, etc. in the area, it can be linked to an increase in supporters of the sports team as well as the area. In addition, the recommended action information presented to the user may be information excluding the primary pass label information and the secondary pass label information, for example, as long as the recommended action information and the primary pass label information can be associated in subsequent processing (e.g., the step (S14a), etc.).

[0052] Then, for example, after the step (S14a), if the determination unit 14 determines that the user has passed, the token granting unit 18 grants the user token information linked to the recommended behavior information (S18, token granting step), and ends the process (END). Specifically, the token information may be electronically granted to the user's terminal as, for example, an electronic coupon or electronic money.

[0053] The device 10 may, for example, tally up the tokens given to each user and display the users in a ranking format.

[0054] According to this embodiment, for example, by setting recommended behavior information including primary pass label information, it is possible to determine whether a user has taken a specific behavior using an image captured by the user. Also, according to this embodiment, for example, secondary pass label information can be added, and the image can be more accurately determined to be "pass" or "fail."

[0055] In recent years, the revitalization of regional economies has become an important issue due to the problem of concentration in one place in a large city. Specifically, it is important to lay the foundation for many people to participate in purchasing activities widely in various fields such as shopping, eating and drinking, and tourism. However, according to the present embodiment, for example, by linking the recommended behavior information with the token information and presenting it to the user, the user can be guided to various places, behavior such as purchasing can be promoted, and services (tokens) can be returned to the user. Therefore, it is possible to elicit active behavior from the user and lead to regional revitalization. The range of the region is not particularly limited, and may be, for example, a town unit, a ward unit, a city unit, or a prefecture unit. Therefore, when the device 10 includes the recommended behavior information presenting unit 17 and the token granting unit 18, the device 10 can also be called a behavior promotion device. Similarly, the behavior determination method can also be called a behavior promotion method.

[0056] [Embodiment 2] An example of a process for adding the secondary passed label information will be described. In this embodiment, the primary passed label information and the secondary passed label information each include a threshold value of the confidence level for the label information, but as described above, this is not limited to this.

[0057] 4A to 4C are schematic diagrams showing an example of adding the secondary pass label information. As a precondition of Figs. 4A to 4C, the action recommended to the user is "Play soccer and get moving!" and the imaging instruction is to capture an image of "a scene of soccer being played". Then, as shown in the figure, the setting unit 11 sets "soccer" and "ball" as label information of the primary pass label information, sets "80%" as the confidence level for each of "soccer" and "ball", and sets that it is sufficient to satisfy at least one of these (matching condition: OR). At this time, the primary pass label information is in the state of data in the "initial state" as shown in Fig. 4A.

[0058] 4A, it is assumed that the execution start condition for the step (S16) shown in Fig. 3 is set to be that the number of images determined to be "passed" in the execution history data is 100 or more. It is also assumed that the following two conditions are set for the label candidate to be added as the secondary passed label information. (1) The average value (average confidence) calculated from the confidences of each of the label candidates is 50% or more. (2) The number of occurrences of the label candidate is 50% or more for all label information other than the label information of the first pass label information among the images judged as "pass."

[0059] Next, assume that many users perform the recommended action information (also called a mission) and transmit images to the device 10. Then, the process from step (S12) to step (S16) shown in Fig. 3 is repeatedly executed, and the recording unit 15 accumulates and stores execution history data linking the label information of the object included in the image captured by the user, the confidence level for the label information, and the result of the judgment by the judgment unit. An example of the execution history data is shown in the row of item "2" in Fig. 4B.

[0060] When the number of images judged as "passed" by the judgment unit 14 exceeds 100, the step (S16) shown in FIG. 3 is executed. Specifically, as shown in the row of item "3" in FIG. 4B, the extraction unit 16 extracts, from the execution history data, label information (label candidates of provisional secondary pass label information) other than the label information that matches the primary pass label information, which is the image for which the judgment result is "passed". Then, as shown in the row of item "4" in FIG. 4C, the setting unit 11 adds and sets the label candidates that satisfy the above conditions (1) and (2) from the extracted label candidates as label information of secondary pass label information to the recommended behavior information. In this example, "football" and "sportswear" are added to the recommended behavior information as label information of the secondary pass label information. The confidence threshold of the secondary pass label information is also set in association with the label information. In this example, the setting unit 11 sets, for example, the added secondary pass label information as "matching condition: OR".

[0061] After that, assume that a new mission is performed by the user, and the label information and the certainty detected by the detection unit 13 are "label information: soccer, certainty: 75%" and "label information: football, certainty: 80%". In this case, even if the detected "label information: soccer, certainty: 75%" does not satisfy the certainty threshold "label information: soccer, threshold: 80%" of the first pass label information, the judgment unit 14 judges the mission as "passed" because the detected "label information: football, certainty: 80%" satisfies the label information and the threshold "label information: football, certainty: 75%" of the second pass label information. In this way, by adding the second pass label information, it is not necessary to periodically review the judgment condition (matching condition with the first pass label information), and the judgment condition can be dynamically corrected.

[0062] 5A to 5C are schematic diagrams showing an example in which the secondary pass label information is not added. The preconditions for Figs. 5A to 5C are the same as those for Figs. 4A to 4C, except that the above condition (2) is changed as follows. (2) The number of occurrences of the label candidate is 80% or more for all label information other than the label information of the first pass label information among the images judged to be “pass”.

[0063] FIG. 5A is similar to FIG. 4A except that the above condition (2) is changed as described above, and FIG. 5B is similar to FIG. 4B.

[0064] Then, as shown in the row of item “4” in FIG. 5C, since there is no label candidate among the extracted label candidates that satisfies the above conditions (1) and (2), the setting unit 11 does not add the secondary passed label information.

[0065] After that, assume that a new mission is performed by the user, and the label information and the certainty detected by the detection unit 13 are "label information: soccer, certainty: 75%" and "label information: football, certainty: 80%". In this case, the determination unit 14 determines that the detected "label information: soccer, certainty: 75%" does not satisfy the certainty threshold "label information: soccer, threshold: 80%" of the first pass label information, and the second pass label information corresponding to the detected "label information: football, certainty: 80%" has not been added, so that the image is determined to be "failed". Therefore, the image is not regarded as an image indicating that the action has been performed.

[0066] [Embodiment 3] The program of the present embodiment is a program for causing a computer to execute each step of the method of the present invention as a procedure. In the present invention, the "procedure" may be read as "processing." The program of the present embodiment may be recorded, for example, in a computer-readable recording medium. The recording medium is not particularly limited, and examples thereof include a read-only memory (ROM), a hard disk (HD), and an optical disk.

[0067] Although the present invention has been described above with reference to the embodiment, the present invention is not limited to the above embodiment. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0068] <Additional Notes> Some or all of the above embodiments may be described as follows, but are not limited to the following supplementary notes. (Appendix 1) A setting unit, an acquisition unit, a detection unit, and a determination unit are included, The setting unit sets recommended action information including first-stage pass label information, the recommended action information is information indicating an action recommended for the user and an instruction to capture an image related to the action, The first pass label information is label information indicating an imaging target related to an action recommended for a user, The acquisition unit acquires an image captured by a user, The detection unit inputs the acquired image into a trained model to detect label information of an imaging object contained in the image, The determination unit determines whether the image is “passed” or not based on the label information and the label information of the first pass label information, "Pass" means that the behavior indicated by the recommended behavior information has been performed and an image related to the behavior has been captured by the behavior determination device. (Appendix 2) The first pass label information includes a confidence threshold for the label information, The certainty is an index indicating a likelihood that the image is the subject, The detection unit further detects the confidence level for the label information, 2. The behavior determination device according to claim 1, wherein the determination unit further determines the image as "pass" if the certainty factor for the label information satisfies a threshold value of the primary pass label information. (Appendix 3) Further, the recording unit and the extracting unit are included, the recording unit accumulates and records, as execution history data, label information of an object to be captured that is included in an image captured by the user and a result of the determination by the determination unit, in association with each other; The extraction unit extracts, from the execution history data, label information of the image for which the result of the judgment is “pass” other than the label information that matches the first pass label information; the setting unit adds the label information that satisfies a preset condition from the extracted label information to the recommended action information as secondary pass label information, and sets the label information; The determination unit determines the label information based on the label information and the label information of the first pass label information, and 3. The behavior determination device according to claim 1 or 2, wherein the image is determined to be "passed" if at least one of the determinations based on the label information and the label information of the secondary pass label information is determined to be "passed." (Appendix 4) 4. The behavior determination device according to claim 3, wherein the extraction unit extracts the label information when a number of the images determined to be "passed" in the execution history data is equal to or greater than a preset number. (Appendix 5) In the execution history data, label information other than the label information that matches the first pass label information is duplicated, The behavior determination device according to claim 3 or 4, wherein the setting unit calculates one of an average value, a median value, or a mode value of the certainty degrees from the respective certainty degrees for the overlapping label information, and adds the calculated numerical value to the imaging instruction information as the certainty degree for the secondary pass label. (Appendix 6) 6. The behavior determination device according to claim 3, wherein the condition preset in the setting unit is at least one of the following (1) and (2). (1) The execution history data contains overlapping label information other than the label information that matches the initially accepted label information, and at least one of the average, median, and mode calculated from the certainty of each of the overlapping label information is equal to or greater than a preset value. (2) In the execution history data, label information other than the label information that matches the initially approved label information is duplicated, and the number of occurrences of the duplicated label information in the execution history data is equal to or greater than a preset number. (Appendix 7) Further, the system includes a recommended behavior information presenting unit and a token granting unit, the recommended action information presentation unit presents the recommended action information to the user in association with token information; The token information is information regarding a token granted to a user upon completion of the action, 7. The behavior determination device according to claim 1, wherein the token granting unit grants token information linked to the recommended behavior information to the user when the determination unit determines that the user is "passed." (Appendix 8) The method includes a setting step, an acquisition step, a detection step, and a determination step, The setting step sets recommended action information including first pass label information; the recommended action information is information indicating an action recommended for the user and an instruction to capture an image related to the action, The first pass label information is label information indicating an imaging target related to an action recommended for a user, The acquiring step acquires an image captured by a user, The detection step includes inputting the acquired image into a trained model to detect label information of an imaging object contained in the image, The judging step judges whether the image is “passed” or not based on the label information and the label information of the first pass label information, In this behavior determination method, "passed" means that the behavior indicated by the recommended behavior information is executed and an image related to the behavior is captured. (Appendix 9) The first pass label information includes a confidence threshold for the label information, The certainty is an index indicating a likelihood that the image is the subject, The detection step further includes detecting the confidence level for the label information, The behavior determination method according to claim 8, wherein the determination step further determines the image as "pass" if the certainty factor for the label information satisfies a threshold value of the primary pass label information. (Appendix 10) Further, the method includes a recording step and an extracting step, The recording step includes accumulating and recording, as execution history data, label information of an object to be captured that is included in an image captured by the user and a result of the determination step in association with each other; The extraction step extracts, from the execution history data, label information of the image for which the result of the judgment is “pass” other than the label information that matches the first pass label information; the setting step includes adding, to the recommended action information, label information that satisfies a preset condition from among the extracted label information as secondary pass label information; The determination step includes the step of: determining based on the label information and the label information of the first pass label information; and The behavior judgment method according to claim 8 or 9, wherein the image is judged as "passed" if at least one of the judgments based on the label information and the label information of the secondary passed label information is judged as "passed". (Appendix 11) The behavior determination method according to claim 10, wherein the extraction step extracts the label information when the number of the images determined to be "passed" in the execution history data is equal to or greater than a preset number. (Appendix 12) In the execution history data, label information other than the label information that matches the first pass label information is duplicated, The behavior determination method according to claim 10 or 11, wherein the setting step calculates one of an average value, a median value, or a mode value of the certainty degrees from the respective certainty degrees for the overlapping label information, and adds the calculated numerical value to the imaging instruction information as the certainty degree for the secondary pass label. (Appendix 13) 13. The behavior determination method according to any one of appendices 10 to 12, wherein the condition preset in the setting step is at least one of the following (1) and (2). (1) The execution history data contains overlapping label information other than the label information that matches the initially accepted label information, and at least one of the average, median, and mode calculated from the certainty of each of the overlapping label information is equal to or greater than a preset value. (2) In the execution history data, label information other than the label information that matches the initially approved label information is duplicated, and the number of occurrences of the duplicated label information in the execution history data is equal to or greater than a preset number. (Appendix 14) Further, the system includes a recommended action information presenting step and a token granting step, The recommended action information presenting step includes presenting the recommended action information to a user in association with token information; The token information is information regarding a token granted to a user upon completion of the action, The behavior determination method according to any one of appendices 8 to 13, wherein the token granting step grants token information linked to the recommended behavior information to the user when the determination step determines that the user is “passed.” (Appendix 15) A program for causing a computer to execute procedures including a setting procedure, an acquisition procedure, a detection procedure, and a determination procedure: The setting step sets recommended action information including first pass label information; the recommended action information is information indicating an action recommended for the user and an instruction to capture an image related to the action, The first pass label information is label information indicating an imaging target related to an action recommended for a user, The acquiring step acquires an image captured by a user; The detection step includes inputting the acquired image into a trained model and detecting label information of an object to be imaged that is included in the image; The judgment step judges whether the image is “passed” or not based on the label information and the label information of the first pass label information, "Pass" means that the action indicated by the recommended action information was performed and an image related to the action was captured. (Appendix 16) The first pass label information includes a confidence threshold for the label information, The certainty is an index indicating a likelihood that the image is the subject, The detection step further includes detecting the confidence level for the label information; 16. The program according to claim 15, wherein the judgment step further comprises judging the image as "pass" if the certainty factor for the label information satisfies a threshold value of the primary pass label information. (Appendix 17) Further, the method includes a recording step and an extracting step, The recording step includes accumulating and recording, as execution history data, label information of an object to be captured that is included in an image captured by the user and a result of a judgment made by the judgment step, in association with each other; The extraction step includes extracting, from the execution history data, label information of the image for which the result of the judgment is “pass” and other than the label information that matches the first pass label information; the setting step adding and setting, from the extracted label information, label information that satisfies a preset condition as secondary pass label information to the recommended action information; The determination step includes the step of: determining based on the label information and the label information of the first pass label information; and The program according to claim 15 or 16, which judges the image as "pass" if at least one of the judgments based on the label information and the label information of the secondary pass label information is judged as "pass". (Appendix 18) 18. The program according to claim 17, wherein the extraction step extracts the label information when the number of the images judged to be "passed" in the execution history data is equal to or greater than a preset number. (Appendix 19) In the execution history data, label information other than the label information that matches the first pass label information is duplicated, The program described in Appendix 17 or 18, wherein the setting procedure calculates one of an average value, a median value, or a mode value of the certainty degrees from the respective certainty degrees for the overlapping label information, and adds the calculated numerical value to the imaging instruction information as the certainty degree for the secondary pass label. (Appendix 20) 20. The program according to any one of appendixes 17 to 19, wherein the preset condition in the setting procedure is at least one of the following (1) and (2). (1) The execution history data contains overlapping label information other than the label information that matches the initially accepted label information, and at least one of the average, median, and mode calculated from the certainty of each of the overlapping label information is equal to or greater than a preset value. (2) In the execution history data, label information other than the label information that matches the initially approved label information is duplicated, and the number of occurrences of the duplicated label information in the execution history data is equal to or greater than a preset number. (Appendix 21) Further, the method includes a procedure for presenting recommended action information and a procedure for granting tokens, The step of presenting recommended action information includes presenting the recommended action information to a user in association with token information; The token information is information regarding a token granted to a user upon completion of the action, 21. The program according to any one of appendices 15 to 20, wherein the token granting step grants token information linked to the recommended behavior information to the user if the judgment step judges the user to be “passed.” (Appendix 22) A computer-readable recording medium having a program according to any one of appendices 15 to 21 recorded thereon. [Industrial Applicability]

[0069] According to the present invention, it is possible to determine whether a user has taken a specific action by image analysis, and therefore the present invention is useful, for example, in the field of recommending a specific action to a user. [Explanation of symbols]

[0070] 10 Behavior determination device 11 Setting section 12 Acquisition Department 13 Detection section 14 Judgment section 15 Recording section 16 Extraction part 17 Recommended Action Information Presentation Section 18 Token Granting Department 101 Central Processing Unit 102 Memory 103 Bus 104 Storage device 105 Input Device 106 Display device 107 Communication Devices

Claims

1. A setting unit, an acquisition unit, a detection unit, and a determination unit are included, The setting unit sets recommended action information including first-stage pass label information, the recommended action information is information indicating an action recommended for the user and an instruction to capture an image related to the action, The first pass label information is label information indicating an imaging target related to an action recommended for a user, The acquisition unit acquires an image captured by a user, The detection unit inputs the acquired image into a trained model to detect label information of an imaging object contained in the image, The determination unit determines whether the image is “passed” or not based on the label information and the label information of the first pass label information, "Pass" means that the action indicated by the recommended action information has been performed and an image related to the action has been captured; Further, the recording unit and the extracting unit are included, the recording unit accumulates and records, as execution history data, label information of an object to be captured that is included in an image captured by the user and a result of the determination by the determination unit, in association with each other; The extraction unit extracts, from the execution history data, label information of the image for which the result of the judgment is “pass” other than the label information that matches the first pass label information; the setting unit adds the label information that satisfies a preset condition from the extracted label information to the recommended action information as secondary pass label information, and sets the label information; The determination unit determines the label information based on the label information and the label information of the first pass label information, and determining that the image is “passed” when at least one of the determinations based on the label information and the label information of the secondary pass label information is “passed”; Behavior determination device.

2. The behavior determination device of claim 1, wherein the extraction unit extracts the label information when the number of images determined to be "passed" in the execution history data is equal to or greater than a predetermined number.

3. The initially accepted label information includes a confidence threshold for the label information, The certainty is an index indicating a likelihood that the image is the subject, The detection unit further detects the confidence level for the label information, The behavior determining device according to claim 1 , wherein the determining unit further determines the image as “pass” when the certainty factor for the label information satisfies a threshold value of the primary pass label information.

4. In the execution history data, label information other than the label information that matches the first pass label information is duplicated, 4. The behavior determination device according to claim 3, wherein the setting unit calculates one of an average value, a median value, or a mode value of the certainty degrees from the respective certainty degrees for the overlapping label information, and adds the calculated numerical value to the imaging instruction information as the certainty degree for the secondary pass label.

5. Further, the system includes a recommended behavior information presenting unit and a token granting unit, the recommended action information presentation unit presents the recommended action information to the user in association with token information; The token information is information regarding a token granted to a user upon completion of the action, The behavior determination device according to claim 1 , wherein the token granting unit grants, to the user, token information linked to the recommended behavior information, when the user is determined to be “passed” by the determining unit.

6. The method includes a setting step, an acquisition step, a detection step, and a determination step, The setting step sets recommended action information including first pass label information; the recommended action information is information indicating an action recommended for the user and an instruction to capture an image related to the action, The first pass label information is label information indicating an imaging target related to an action recommended for a user, The acquiring step acquires an image captured by a user, The detection step includes inputting the acquired image into a trained model to detect label information of an imaging object contained in the image, The judging step judges whether the image is “passed” or not based on the label information and the label information of the first pass label information, "Pass" means that the action indicated by the recommended action information has been performed and an image related to the action has been captured; Further, the method includes a recording step and an extracting step, The recording step includes accumulating and recording, as execution history data, label information of an object to be captured that is included in an image captured by the user and a result of the determination step in association with each other; The extraction step extracts, from the execution history data, label information of the image for which the result of the judgment is “pass” other than the label information that matches the first pass label information; the setting step includes adding, to the recommended action information, label information that satisfies a preset condition from among the extracted label information as secondary pass label information; The determination step includes the step of: determining based on the label information and the label information of the first pass label information; and A behavior judgment method in which each step is executed by a computer, and when at least one of the judgments based on the label information and the label information of the secondary pass label information is judged to be "pass", the image is judged to be "pass".

7. The first pass label information includes a confidence threshold for the label information, The certainty is an index indicating a likelihood that the image is the subject, The detection step further includes detecting the confidence level for the label information, 7. The behavior determination method according to claim 6, wherein the determination step further comprises determining the image as "pass" when the confidence factor for the label information satisfies a threshold value of the primary pass label information.

8. A program for causing a computer to execute procedures including a setting procedure, an acquisition procedure, a detection procedure, a determination procedure, a recording procedure, and an extraction procedure: The setting step sets recommended action information including first pass label information; the recommended action information is information indicating an action recommended for the user and an instruction to capture an image related to the action, The first pass label information is label information indicating an imaging target related to an action recommended for a user, The acquiring step acquires an image captured by a user; The detection step includes inputting the acquired image into a trained model and detecting label information of an object to be imaged that is included in the image; The judgment step judges whether the image is “passed” or not based on the label information and the label information of the first pass label information, "Pass" means that the action indicated by the recommended action information has been performed and an image related to the action has been captured; The recording step includes accumulating and recording, as execution history data, label information of an object to be captured that is included in an image captured by the user and a result of a judgment made by the judgment step, in association with each other; The extraction step includes extracting, from the execution history data, label information of the image for which the result of the judgment is “pass” and other than the label information that matches the first pass label information; the setting step adding and setting, from the extracted label information, label information that satisfies a preset condition as secondary pass label information to the recommended action information; The determination step includes the step of: determining based on the label information and the label information of the first pass label information; and When at least one of the judgments based on the label information and the label information of the secondary pass label information is judged as "pass", the image is judged as "pass".

Images