Information processing device, information processing method, and information processing program

Abstract

To obtain prompts that allow for efficient analysis of the user's actions within the video. [Solution] The information processing device according to the present invention comprises: a first acquisition unit that acquires behavioral attribute information relating to behavioral attributes that indicate the user's behavioral tendencies based on video information relating to a video in which the user has been captured; and a second acquisition unit that acquires behavioral analysis prompts, which are prompts corresponding to the user's behavioral attributes, that are prompts for analyzing the user's behavior in the video, based on the video information and the behavioral attribute information.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Conventionally, techniques for analyzing video data such as surveillance cameras are known. For example, techniques for analyzing the actions of people in video from surveillance cameras are known. For example, on a network that can communicate with a store, image data captured by a surveillance camera in the store is received, feature values related to a person are extracted from the image data, and based on whether the feature values correspond to rules related to shoplifting suspicion actions defined in advance using the feature values of the image data, a technique for detecting the occurrence of shoplifting suspicion actions by a person is known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Means for Solving the Problems

[0004] The information processing apparatus according to the present application includes a first acquisition unit that acquires action attribute information related to an action attribute indicating a tendency of an action of the user based on video information related to a video of the user, and a second acquisition unit that acquires an action analysis prompt for analyzing an action of the user in the video based on the video information and the action attribute information, the action analysis prompt being a prompt corresponding to the action attribute of the user.

Brief Description of the Drawings

[0005] [Figure 1] FIG. 1 is a diagram showing a configuration example of an information processing system according to an embodiment. [Figure 2]Figure 2 shows an example of the configuration of an information processing device according to the embodiment. [Figure 3] Figure 3 shows an example of information stored in the user database according to this embodiment. [Figure 4] Figure 4 is a diagram illustrating an example of information processing by the information processing device according to the embodiment. [Figure 5] Figure 5 shows an example of video information according to the present invention. [Figure 6] Figure 6 shows an example of user attribute information according to the embodiment. [Figure 7] Figure 7 shows an example of an attribute prompt according to the embodiment. [Figure 8] Figure 8 shows an example of a generation prompt according to the embodiment. [Figure 9] Figure 9 shows an example of a hardware configuration. [Modes for carrying out the invention]

[0006] The following describes in detail, with reference to the drawings, the embodiments for implementing the information processing device, information processing method, and information processing program according to the present application (hereinafter referred to as "embodiments"). Note that these embodiments do not limit the information processing device, information processing method, and information processing program according to the present application. Furthermore, the same parts are denoted by the same reference numerals in each of the following embodiments, and redundant descriptions are omitted.

[0007] (Embodiment) [1. Introduction] Conventional technologies for analyzing video data from surveillance cameras and the like are known. For example, technologies for analyzing the actions of people in video footage from surveillance cameras and the like are known. Hereafter, video may be referred to as video footage. For example, a technology is known that receives image data captured by a store's surveillance camera on a network that can communicate with the store, extracts feature values ​​related to people from the image data, and detects the occurrence of suspected shoplifting by a person based on whether or not the feature values ​​match rules related to suspected shoplifting that have been defined in advance using the feature values ​​of the image data (Patent Document 1). However, conventional technologies only analyze whether or not the actions of people in video footage match rules that have been defined in advance, and therefore cannot analyze the actions of people in video footage that do not match the rules that have been defined in advance. For this reason, conventional technologies could not be said to be able to efficiently analyze the actions of people in video footage. Hereafter, people in video footage may be referred to as "users".

[0008] Furthermore, in recent years, techniques have been developed to analyze the behavior of people in videos using multimodal large-scale language models (LLMs). For example, by inputting prompts for analyzing the behavior of people in a video along with the video into a multimodal large-scale language model, analytical information showing the results of the analysis of the behavior of people in the video is generated. In contrast, there is a need for a technique that can acquire prompts that can efficiently analyze the behavior of people in a video. In other words, there is a need for a technique that can acquire prompts that can efficiently analyze the user's behavior in a video.

[0009] For example, if the attributes of a user in a video are unknown, it is necessary to input prompts into the multimodal large-scale language model to analyze the user's behavior for each of several different attributes that may correspond to the user in the video. For example, if it is unknown whether the user in the video is an adult or a child, and whether they are a person of interest who might shoplift, it is necessary to input prompts such as, "Describe what the customer is picking up. Notify us if the customer is a child and appears to be in distress. Notify us if the customer is shoplifting." into the multimodal large-scale language model. In addition, if prompt #1 is input, the multimodal large-scale language model will output analytical information such as, "The customer is picking up A. They are hesitating whether to buy A, but ultimately put it back on the shelf. The customer does not appear to be a child. They do not appear to be in particular distress. The customer is picking up product A, but has put it back on the shelf and does not appear to be shoplifting." However, prompts to analyze the user's behavior for each of several different attributes that may correspond to the user in the video cannot be said to be prompts that can efficiently analyze the user's behavior in the video. Furthermore, prompts for analyzing user behavior across multiple different attributes tend to be lengthy because they contain multiple instructions that dictate the analysis of user behavior across multiple different attributes.

[0010] On the other hand, if the attributes of the user in the video are known, prompts corresponding to the user's attributes are input to the multimodal large-scale language model. For example, if it is known that the user in the video is a person of interest who is likely to shoplift, prompt #2, "If you are shoplifting, please notify us," which analyzes the behavior of a person likely to shoplift, is input to the multimodal large-scale language model. When prompt #2 is input, the multimodal large-scale language model outputs the analysis information, "I am not shoplifting." Thus, prompts corresponding to the user's attributes in the video can be said to be prompts that can analyze the user's behavior in the video more efficiently than prompts that analyze the user's behavior for multiple different attributes. Furthermore, prompts corresponding to the user's attributes in the video can be said to be efficient because they require fewer characters than prompts that analyze the user's behavior for multiple different attributes.

[0011] Furthermore, generally speaking, increasing the number of tokens input into a multimodal large-scale language model increases the processing time of the model. Also, the cost of using tokens increases with the number of tokens input. Additionally, there is an upper limit to the number of tokens that can be input into a multimodal large-scale language model at one time. Therefore, it is desirable to input fewer tokens into a multimodal large-scale language model. For example, it is desirable to use fewer characters in the prompt input to a multimodal large-scale language model.

[0012] In contrast, the information processing device according to this embodiment acquires behavioral attribute information relating to behavioral attributes that indicate the user's behavioral tendencies, based on video information relating to a video in which the user has been captured. The information processing device also acquires behavioral analysis prompts, which are prompts for analyzing the user's behavior in the video and are corresponding to the user's behavioral attributes, based on the video information and behavioral attribute information. In this way, the information processing device acquires prompts for analyzing the user's behavior in the video and are corresponding to the user's behavioral attributes, based on the user's behavioral attribute information in the video. As a result, the information processing device can acquire prompts that enable efficient analysis of the user's behavior in the video. Furthermore, the information processing device can efficiently analyze the user's behavior in the video by, for example, inputting prompts that enable efficient analysis of the user's behavior in the video into a language model.

[0013] Furthermore, as mentioned above, prompts that correspond to the user's behavioral attributes in a video generally require fewer characters than prompts that do not correspond to the user's behavioral attributes in a video. Therefore, the information processing device can reduce the number of tokens input to the language model. In addition, because the information processing device can reduce the number of tokens input to the language model, it can reduce the time required for the language model to analyze the user's behavior in the video and the cost of using tokens. In addition, because the information processing device can reduce the time required for the language model to analyze the user's behavior in the video, it can speed up the analysis process of the user's behavior in the video by the language model.

[0014] [2. Configuration of the Information Processing System] FIG. 1 is a diagram showing a configuration example of an information processing system 1 according to an embodiment. As shown in FIG. 1, the information processing system 1 according to the embodiment includes a terminal device 10 and an information processing device 100. The terminal device 10 and the information processing device 100 are communicably connected by wire or wirelessly via a predetermined communication network (network N). Note that the information processing system 1 may include a plurality of terminal devices 10 and a plurality of information processing devices 100.

[0015] The terminal device 10 is an information processing device equipped with a camera. The terminal device 10 may be a home appliance, a mobile device such as a smartphone owned by a person, an automobile, or the like. The camera may be a camera mounted on an information device such as a smartphone, a camera mounted on an industrial robot, an in-vehicle camera (drive recorder), a security camera, or a surveillance camera. Note that the terminal device 10 may be referred to as an edge device. The camera generates a video (image) regarding the environment around the camera. The terminal device 10 acquires the video from the camera. Hereinafter, the case where the terminal device 10 is a security camera installed in a store that sells goods will be described. The terminal device 10 generates a video obtained by imaging a user in the store that sells goods.

[0016] The information processing device 100 is an information processing device that performs information processing according to the embodiment. Specifically, the information processing device 100 acquires a video from the terminal device 10. In addition, the information processing device 100 acquires an action analysis prompt, which is a prompt for analyzing the actions of a user in the video of the user and is a prompt according to the action attributes of the user. For example, the information processing device 100 may perform the information processing according to the embodiment according to an information processing method implemented by an information processing program according to the embodiment. For example, the information processing device 100 corresponds to a cloud computer (server device).

[0017] 〔3. Configuration of Information Processing Device〕 FIG. 2 is a diagram showing a configuration example of the information processing device 100 according to the embodiment. The information processing device 100 according to the embodiment includes a communication unit 110, a storage unit 120, and a control unit 130.

[0018] (Communication unit 110) The communication unit 110 is connected to the network N either wired or wirelessly and transmits and receives information to and from the terminal device 10. For example, the communication unit 110 is realized by a NIC (Network Interface Card), an antenna, or the like.

[0019] (Memory unit 120) The memory unit 120 is realized, for example, by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. Specifically, the memory unit 120 stores an information processing program according to the embodiment. The memory unit 120 also stores video information related to videos. For example, the memory unit 120 stores videos. The memory unit 120 also stores information related to a machine learning model. For example, when a video is input and the user in the video is a new user, the memory unit 120 outputs information indicating that the user in the video is a new user, and when the user in the video is not a new user (an existing user), the memory unit 120 stores information related to a face recognition model, which is a machine learning model trained to output identification information (e.g., user ID) for identifying the user in the video. The memory unit 120 also stores information related to a user attribute model, which is a machine learning model trained to output user attribute information indicating the user attributes of the users included in the video when a video including users is input. The memory unit 120 also stores information related to a caption generation model, which is a machine learning model for generating captions corresponding to videos from videos. The memory unit 120 also has a user database 121.

[0020] (User database 121) The user database 121 stores various information about the user. Figure 3 shows an example of the information stored in the user database 121 according to this embodiment. As shown in Figure 3, the user database 121 stores information such as "user ID," "facial recognition information," "user attribute information," "behavioral attribute information," and "behavioral analysis prompts" in association with each other.

[0021] The "User ID" field stores identification information to identify the user. The "Facial Recognition Information" field stores information used for facial recognition of the user. For example, the "Facial Recognition Information" field stores the user's facial image or information indicating the characteristics of the user's facial image. The "User Attribute Information" field stores user attribute information related to the user's user attributes. Here, user attributes include demographic attributes such as the user's age and gender. User attributes may also include geographic attributes such as the user's place of residence. User attributes may also include psychographic attributes such as the user's interests, values, and hobbies. The "Behavioral Attribute Information" field stores behavioral attribute information related to the user's behavior. Here, behavioral attributes include behavioral attributes that indicate the user's behavioral tendencies. For example, behavioral attribute information may include information indicating the user's behavioral tendencies in a physical store where products are sold. For example, behavioral attribute information may include information indicating whether the user's behavior in a physical store where products are sold is normal or abnormal. Here, abnormal user behavior in a physical store selling products includes suspicious behavior or actions that suggest shoplifting. Normal user behavior in a physical store selling products means that the user's behavior is not abnormal. The "Behavioral Analysis Prompt" field contains prompts for analyzing the user's behavior in the video, and these prompts are tailored to the user's behavioral attributes.

[0022] (Control unit 130) The control unit 130 is a controller, and is realized, for example, by executing various programs stored in the memory device inside the information processing device 100 using RAM as the working area, using a CPU (Central Processing Unit) or MPU (Micro Processing Unit), etc. Alternatively, the control unit 130 is a controller and can be realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0023] The control unit 130 has a first acquisition unit 131, a second acquisition unit 132, a generation unit 133, and a notification unit 134 as functional units, and may realize or execute the information processing operations described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Figure 2, and other configurations are also possible as long as they perform the information processing described later. Also, each functional unit indicates the function of the control unit 130 and does not necessarily have to be physically separated.

[0024] (1st acquisition part 131) The first acquisition unit 131 acquires a video (hereinafter sometimes simply referred to as "video") of the user. Specifically, the first acquisition unit 131 acquires a video from the terminal device 10. For example, the first acquisition unit 131 acquires a video that includes the user's face. Furthermore, when the first acquisition unit 131 acquires a video, it performs personal authentication of the user in the video using known personal authentication technology. For example, when the first acquisition unit 131 acquires a video, it performs facial authentication of the user in the video using known facial recognition technology. For example, when the first acquisition unit 131 receives a video that includes the user's face as input, it acquires a facial recognition model, which is a machine learning model trained to determine whether the user in the video is a new user or not. For example, when the first acquisition unit 131 receives a video that includes the user's face as input, if it determines that the user in the video is a new user, it outputs information indicating that the user in the video is a new user and acquires a facial recognition model, which is a machine learning model trained to generate identification information (e.g., user ID) to identify the user in the video. Furthermore, if the user in the video is a new user, the first acquisition unit 131 associates the user's facial recognition information with identification information (e.g., user ID) that identifies the user in the video and stores it in the user database 121. Also, if a video containing a user's face is input, and the first acquisition unit 131 determines that the user in the video is not a new user (i.e., an existing user), it acquires a facial recognition model, which is a machine learning model trained to identify the user in the video. For example, if the first acquisition unit 131 determines that the user in the video is not a new user (i.e., an existing user), it acquires a facial recognition model, which is a machine learning model trained to output identification information (e.g., user ID) that identifies the user in the video.

[0025] Figure 4 is a diagram illustrating an example of information processing by the information processing device 100 according to the embodiment. In Figure 4, the camera of the terminal device 10 captures images of users U1 and U2 shopping at a store that sells goods (for example, clothing). The first acquisition unit 131 acquires video from the terminal device 10. The first acquisition unit 131 acquires video including users U1 and U2. The first acquisition unit 131 acquires video including the faces of users U1 and U2. When the first acquisition unit 131 acquires video, it refers to the storage unit 120 to acquire a face recognition model. When the first acquisition unit 131 acquires video and a face recognition model, it inputs the video into the face recognition model to perform face recognition of the users in the video.

[0026] In Figure 4, user U1 is not a new user (it is an existing user). The first acquisition unit 131 inputs a video containing user U1's face into the face recognition model and identifies that user U1 is not a new user (it is an existing user). For example, the first acquisition unit 131 identifies that user U1 is not a new user (it is an existing user) by performing face recognition on user U1 in the video by referring to the face image of user U1 included in the video and the face recognition information stored in the user database 121. The first acquisition unit 131 also refers to the face image of user U1 included in the video and the face recognition information stored in the user database 121 and outputs the user ID "U1", which is identification information that identifies user U1 in the video.

[0027] In Figure 4, user U2 is a new user. The first acquisition unit 131 inputs a video containing user U2's face into the face recognition model and identifies user U2 in the video as a new user. For example, the first acquisition unit 131 identifies user U2 as a new user by performing face recognition on user U2 in the video by referring to the face image of user U2 included in the video and the face recognition information stored in the user database 121. If the first acquisition unit 131 identifies user U2 as a new user, it generates a user ID "U2", which is identification information that identifies user U2 in the video. The first acquisition unit 131 also generates face recognition information for user U2 based on the face image of user U2 included in the video. The first acquisition unit 131 also associates the face recognition information of user U2 in the video with the user ID "U2", which is identification information that identifies user U2 in the video, and stores them in the user database 121.

[0028] Furthermore, when the first acquisition unit 131 acquires a video, it generates video information related to the video. Specifically, when the first acquisition unit 131 acquires a video, it refers to the storage unit 120 and acquires a caption generation model, which is a machine learning model that generates captions corresponding to the video from the video. For example, the first acquisition unit 131 acquires a caption generation model that has been trained to output a caption corresponding to the video when a video is input, based on a dataset of videos and captions corresponding to the videos. Also, when the first acquisition unit 131 acquires a video and a caption generation model, it inputs the video into the caption generation model and generates a caption corresponding to the video. For example, the first acquisition unit 131 acquires a caption generation model that has been trained to generate multiple captions at once for a single video. For example, when the first acquisition unit 131 acquires a video, it acquires a caption generation model that has been trained to extract multiple frames from the video at predetermined time intervals (e.g., 5 seconds, etc.) and output a caption corresponding to each of the extracted frames. Furthermore, the first acquisition unit 131 may call the caption generation model multiple times for a single video and generate multiple captions for a single video. The caption generation model may also be a multimodal large-scale language model (multimodal LLM). For example, the caption generation model may be Gemini-1.5-Pro ​​or MV-GPT, etc. Here, video information is information that includes captions corresponding to the video. For example, video information is information output from the caption generation model. For example, video information is information that includes data consisting of a timestamp, which is information about the time of the video, and a caption corresponding to the video. For example, the caption generation model may be a machine learning model that has been trained to output video information that includes data consisting of a timestamp, which is information about the time of the video, and a caption corresponding to the video, when a video is input. For example, the first acquisition unit 131 inputs a video to the caption generation model and generates video information that includes data consisting of a timestamp, which is information about the time of the video, and a caption corresponding to the video.

[0029] Figure 5 shows an example of video information according to the embodiment. In Figure 5, when the first acquisition unit 131 acquires a video, it inputs the video to the caption generation model to generate video information IN1 which includes data of a timestamp T11 and a caption CA11. Here, the timestamp T11 is information about the time of the video. For example, the timestamp T11 may be information indicating the time of each individual frame that makes up the video. Alternatively, the timestamp T11 may be information indicating the time for each group of multiple frames that make up the video. For example, the timestamp T11 may be information indicating the time of the earliest frame included in each group of multiple frames, or information indicating the time of the last frame. Alternatively, the timestamp T11 may be the average value of the times of the frames included in each group of multiple frames. The caption CA11 is a caption corresponding to the video. For example, the caption CA11 may be a caption corresponding to each individual frame that makes up the video (for example, a descriptive text that explains the content of each frame). Alternatively, the caption CA11 may be a descriptive text that explains the content of each group of multiple frames that make up the video. For example, caption CA11 may be a caption corresponding to a group of frames (for example, a descriptive text that explains the content of the group of frames).

[0030] Furthermore, when the first acquisition unit 131 generates video information, it generates behavioral attribute information that indicates the user's behavioral tendencies based on the video information. Specifically, the first acquisition unit 131 inputs an attribute prompt, which is a prompt instructing the language model to generate behavioral attribute information from the video information, and the video information into the language model to generate behavioral attribute information. More specifically, the first acquisition unit 131 generates information as behavioral attribute information that indicates the user's behavioral tendencies in the video. For example, the first acquisition unit 131 analyzes the user's behavioral tendencies in the video and inputs an attribute prompt into the language model that includes an instruction instructing the language model to generate information that indicates the user's behavioral tendencies in the video, thereby generating information as behavioral attribute information that indicates the user's behavioral tendencies in the video. Here, the language model can be any machine learning model that is capable of processing natural language, and it can be a large-scale or small-scale language model. Below, we will explain the case where the language model is a large-scale language model. A large-scale language model may be, for example, a GPT (Generative Pre-trained Transformer) model.

[0031] For example, the first acquisition unit 131 inputs the video information IN1, as explained in Figure 5, into a large-scale language model to generate behavioral attribute information. For example, the first acquisition unit 131 inputs the video information IN1, which includes data of timestamps T11 and captions CL11, into a large-scale language model to generate behavioral attribute information. The first acquisition unit 131 also acquires data of timestamps that are temporally consecutive. The first acquisition unit 131 acquires data of timestamps and captions that are temporally consecutive. The first acquisition unit 131 acquires multiple captions that are temporally consecutive. The first acquisition unit 131 inputs data of timestamps and captions that are temporally consecutive into a large-scale language model to generate behavioral attribute information. In this way, the first acquisition unit 131 inputs multiple captions that are temporally consecutive into a large-scale language model to generate behavioral attribute information. The first acquisition unit 131 also acquires behavioral attribute information by generating behavioral attribute information based on the video information. In this way, the first acquisition unit 131 acquires behavioral attribute information relating to behavioral attributes that indicate the user's behavioral tendencies, based on video information relating to a video of the user. For example, the first acquisition unit 131 acquires behavioral attribute information relating to behavioral attributes that indicate the user's behavioral tendencies in the video. The "Behavioral Attribute Information" item in the user database 121 stores information indicating behavioral attributes based on information such as the user's past shoplifting history or history of damaging goods. Alternatively, instead of generating behavioral attribute information, the first acquisition unit 131 may acquire information already stored in the "Behavioral Attribute Information" item in the user database 121 as behavioral attribute information. For example, if the first acquisition unit 131 determines that the user in the video is not a new user (i.e., an existing user), it may refer to the "Behavioral Attribute Information" item in the user database 121 and acquire information already stored in the "Behavioral Attribute Information" item in the user database 121 as behavioral attribute information.

[0032] Furthermore, if the first acquisition unit 131 acquires a video, it performs personal authentication of the user included in the video. For example, if the first acquisition unit 131 acquires a video, it performs facial recognition of the user included in the video. Also, if the first acquisition unit 131 performs personal authentication of the user included in the video, it acquires user attribute information regarding the user's attributes based on the result of the personal authentication. For example, if the first acquisition unit 131 performs facial recognition of the user included in the video, it acquires user attribute information regarding the user's attributes based on the result of the facial recognition.

[0033] For example, the first acquisition unit 131 performs facial recognition of a user included in the video, and if it identifies that the user is not a new user (i.e., an existing user), it acquires a user ID, which is identification information that identifies the user included in the video. For example, the first acquisition unit 131 acquires the user ID output from the facial recognition model. The first acquisition unit 131 also refers to the user database 121 based on the acquired user ID and acquires user attribute information of the user corresponding to the user ID that has been previously stored in the user database 121. In Figure 4, the first acquisition unit 131 performs facial recognition of user U1 included in the video, and if it identifies that user U1 is not a new user (i.e., an existing user), it acquires the user ID "U1" of user U1. The first acquisition unit 131 also refers to the user database 121 based on the user ID "U1" and acquires user attribute information corresponding to the user ID "U1" that has been previously stored in the user database 121.

[0034] Furthermore, the first acquisition unit 131 performs facial recognition of the user included in the video, and if it identifies the user as a new user, it obtains a user ID, which is identification information that identifies the user included in the video. For example, the first acquisition unit 131 obtains the user ID output from the facial recognition model. Also, if the first acquisition unit 131 identifies the user as a new user, it refers to the storage unit 120 and obtains a user attribute model, which is a machine learning model that has been trained to output user attribute information indicating the user attributes of the user included in the video when a video containing the user is input. Also, if the first acquisition unit 131 has obtained the video and the user attribute model, it inputs the video into the user attribute model to generate user attribute information of the user included in the video. In this way, the first acquisition unit 131 obtains user attribute information by generating user attribute information. Also, if the first acquisition unit 131 has generated user attribute information, it associates the generated user attribute information with the acquired user ID and stores it in the user database 121. In Figure 4, the first acquisition unit 131 performs facial recognition on user U2 included in the video. If it identifies user U2 as a new user, it acquires user U2's user ID "U2". If the first acquisition unit 131 identifies user U2 as a new user, it inputs the video containing user U2 into the user attribute model to generate user attribute information for user U2 included in the video. In this way, the first acquisition unit 131 acquires user attribute information for user U2 by generating user attribute information for user U2. The first acquisition unit 131 also associates the generated user attribute information for user U2 with the acquired user ID "U2" for user U2 and stores it in the user database 121.

[0035] Figure 6 shows an example of user attribute information according to the embodiment. In Figure 6, the first acquisition unit 131 acquires user attribute information IN2, which includes gender information A11 indicating the user's gender and age information A12 indicating the user's age. Note that the user attribute information shown in Figure 6 is just an example, and user attribute information is not limited to the user's gender and age. For example, user attribute information may include information on demographic attributes other than the user's age and gender. User attribute information may also include information indicating geographic attributes, such as the user's place of residence. Furthermore, user attribute information may include information indicating psychographic attributes, such as the user's interests, values, and hobbies.

[0036] Furthermore, the first acquisition unit 131 generates behavioral attribute information corresponding to the user's user attributes based on the user attribute information. For example, the first acquisition unit 131 generates behavioral attribute information by inputting an attribute prompt, which is a prompt that instructs the system to generate behavioral attribute information from user attribute information relating to the user's user attributes, and the user attribute information IN2, as explained in Figure 6, into a large-scale language model.

[0037] Figure 7 shows an example of an attribute prompt according to the embodiment. The first acquisition unit 131 inputs an attribute prompt P1 as shown in Figure 7 into a large-scale language model to generate behavioral attribute information. Specifically, the first acquisition unit 131 inputs the video information IN1 described in Figure 5, the user attribute information IN2 described in Figure 6, and the attribute prompt P1 shown in Figure 7 into a large-scale language model to generate behavioral attribute information. For example, the first acquisition unit 131 inputs an attribute prompt P1 into a large-scale language model that includes an instruction sentence instructing the model to generate behavioral attribute information showing the user's behavioral tendencies in the video, referencing the video information IN1 described in Figure 5 and the user attribute information IN2 described in Figure 6 as input data, and generates behavioral attribute information. In Figure 7, the first acquisition unit 131 inputs an attribute prompt P1 including the sentence P11 "Input prompt: Refer to the input data and determine the individual's attributes" into a large-scale language model to generate behavioral attribute information. Furthermore, the first acquisition unit 131 inputs an attribute prompt P1 containing the text P12, "### Attribute Determination Method Refer to [Gender":"Male", "Age":"35", ...], [{timestamp, Caption}, ...] from the input data, and analyze and output the individual's "Purchasing Tendencies" and "Behavioral Tendencies," to the large-scale language model to generate behavioral attribute information. In addition, the first acquisition unit 131 inputs an attribute prompt P1 containing an instruction sentence that instructs the large-scale language model to output the behavioral attribute information in a specific text format as output data to generate behavioral attribute information. In this way, the first acquisition unit 131 inputs an attribute prompt P1 containing information indicating a method for determining behavioral attributes that show the user's behavioral tendencies in the video to the large-scale language model to generate behavioral attribute information. Specifically, the first acquisition unit 131 inputs an attribute prompt P1 containing an instruction sentence that instructs the large-scale language model to analyze the user's behavioral tendencies in the video based on video information and user attribute information, and generates behavioral attribute information to generate behavioral attribute information. In Figure 7, the first acquisition unit 131 inputs an attribute prompt P1 containing the text P13 "###Output {"Gender":"", "Age":"", "Purchasing Tendency":"", "Behavioral Tendency":"", ...}" into a large-scale language model to generate behavioral attribute information.

[0038] Furthermore, the first acquisition unit 131 generates information as behavioral attribute information that indicates the trends in user behavior in videos at physical stores where products are sold. For example, the first acquisition unit 131 analyzes the trends in user behavior in videos at physical stores where products are sold, and inputs an attribute prompt P1 containing an instruction sentence to the large-scale language model instructing it to generate information that indicates the trends in user behavior in videos at physical stores where products are sold, thereby generating information as behavioral attribute information that indicates the trends in user behavior in videos at physical stores where products are sold.

[0039] Furthermore, the first acquisition unit 131 generates information as behavioral attribute information indicating whether the user's behavior in the video at the physical store selling the product is normal or abnormal. Here, abnormal user behavior in the video at the physical store selling the product includes the user acting suspiciously or the user taking actions that suggest they might shoplift. Normal user behavior in the video at the physical store selling the product indicates that the user's behavior is not abnormal. For example, the first acquisition unit 131 analyzes whether the user's behavior in the video at the physical store selling the product is normal or abnormal, and inputs an attribute prompt P1 into the large-scale language model, which includes an instruction to generate information indicating whether the user's behavior in the video at the physical store selling the product is normal or abnormal, thereby generating information as behavioral attribute information indicating whether the user's behavior in the video at the physical store selling the product is normal or abnormal.

[0040] Furthermore, the first acquisition unit 131 generates information as behavioral attribute information that indicates the purchasing behavior trends of users in videos at stores that sell products. For example, the first acquisition unit 131 analyzes the purchasing behavior of users in videos at stores that sell products and inputs an attribute prompt P1 into a large-scale language model that includes an instruction to generate information indicating the purchasing behavior trends of users in videos at stores that sell products, thereby generating information as behavioral attribute information that indicates the purchasing behavior trends of users in videos at stores that sell products.

[0041] Furthermore, when the first acquisition unit 131 generates behavioral attribute information, it associates the behavioral attribute information with identification information (for example, a user ID) that identifies the user in the video and stores it in the user database 121.

[0042] (Second Acquisition Part 132) The second acquisition unit 132 acquires behavioral analysis prompts, which are prompts corresponding to the user's behavioral attributes, for analyzing the user's behavior in the video based on the video information and behavioral attribute information. Specifically, the second acquisition unit 132 acquires behavioral analysis prompts by generating behavioral analysis prompts based on the video information and behavioral attribute information. More specifically, when behavioral attribute information is acquired by the first acquisition unit 131, the second acquisition unit 132 generates a behavioral analysis prompt by inputting a generation prompt, which is a prompt instructing the first acquisition unit 131 to generate a behavioral analysis prompt from the video information and behavioral attribute information, and the video information and behavioral attribute information into the language model. Furthermore, the second acquisition unit 132 generates a behavioral analysis prompt by inputting a generation prompt, which is a prompt instructing the first acquisition unit 131 to generate a behavioral analysis prompt from user attribute information relating to the user's user attributes, and the user attribute information into the language model. For example, the second acquisition unit 132 generates a behavioral analysis prompt by inputting a generation prompt, which includes an instruction sentence instructing the language model to generate a behavioral analysis prompt by referring to the video information, user attribute information, and behavioral attribute information as input data. For example, the second acquisition unit 132 inputs video information, user attribute information, behavioral attribute information, and generated prompts into a language model to generate behavioral analysis prompts.

[0043] Figure 8 shows an example of a generated prompt according to the embodiment. For example, the second acquisition unit 132 inputs the video information IN1 described in Figure 5, the user attribute information IN2 described in Figure 6, the behavioral attribute information described in Figure 7, and the generated prompt P2 shown in Figure 8 into a large-scale language model to generate a behavioral analysis prompt. In Figure 8, the second acquisition unit 132 inputs the video information, user attribute information, behavioral attribute information, and the generated prompt P2, which includes the sentence P21 "Input prompt: Refer to the input data and create the optimal prompt," into the large-scale language model as input data to generate a behavioral analysis prompt.

[0044] Furthermore, the second acquisition unit 132 inputs a generation prompt P2 containing information on how to create a behavioral analysis prompt into the large-scale language model to generate a behavioral analysis prompt. Specifically, the second acquisition unit 132 inputs a generation prompt P2 containing an instruction sentence that instructs the large-scale language model to generate a behavioral analysis prompt for analyzing the user's behavior in the video based on video information, user attribute information, and behavioral attribute information to generate a behavioral analysis prompt. In Figure 8, the second acquisition unit 132 inputs a generation prompt P2 containing the sentence P22 "### How to create a prompt From the input data, create a prompt for analyzing an individual's behavior from [Gender":"Male","Age":"35", ...], [{timestamp, Caption}, ...] and information from the user database to generate a behavioral analysis prompt.

[0045] Furthermore, the second acquisition unit 132 inputs a generated prompt P2 containing information indicating the perspective for analyzing the user's behavior in the video into the large-scale language model to generate a behavioral analysis prompt. In Figure 8, the second acquisition unit 132 inputs a generated prompt P2 containing the sentence P23, "Please analyze the user's behavior, especially in clothing stores, while shopping," into the large-scale language model to generate a behavioral analysis prompt.

[0046] For example, the second acquisition unit 132 generates a behavioral analysis prompt that instructs the system to notify the user of their stay time and location if the user's behavior in the video is normal, based on the user's behavioral attribute information. Also, the second acquisition unit 132 generates a behavioral analysis prompt that instructs the system to detect the user's abnormal behavior and notify the system if the user's behavior in the video is abnormal, based on the user's behavioral attribute information. For example, the user's abnormal behavior may be a criminal act such as shoplifting.

[0047] Furthermore, if the second acquisition unit 132 generates a behavioral analysis prompt, it associates the behavioral analysis prompt with identification information (e.g., user ID) that identifies the user in the video and stores it in the user database 121.

[0048] (Generation unit 133) The generation unit 133 inputs the video and behavioral analysis prompts to the language model to generate analysis information showing the results of the analysis of the user's behavior in the video. Here, the language model can be any language model that can process video in addition to text. For example, the language model may be a multimodal large-scale language model. For example, the language model may be Gemini-1.5-Pro ​​or MV-GPT, etc. For example, if a behavioral analysis prompt is generated by the second acquisition unit 132, the generation unit 133 inputs the video and behavioral analysis prompts to the multimodal large-scale language model to generate analysis information.

[0049] (Notification Department 134) The notification unit 134 notifies the video monitor of the analysis information. For example, if the generation unit 133 generates analysis information, the notification unit 134 transmits the analysis information to the information processing device used by the video monitor.

[0050] [4. Variations] In the embodiment described above, the case in which the second acquisition unit 132 acquires a behavioral analysis prompt by generating a behavioral analysis prompt was explained. In a modified example, the second acquisition unit 132 acquires a behavioral analysis prompt by selecting a behavioral analysis prompt. Specifically, the second acquisition unit 132 acquires a behavioral analysis prompt by selecting a behavioral analysis prompt based on video information and behavioral attribute information. More specifically, when the second acquisition unit 132 generates a behavioral analysis prompt, it stores the behavioral analysis prompt and the user behavioral attribute information in the video in association with each other in the storage unit 120. The second acquisition unit 132 stores multiple different behavioral analysis prompts corresponding to each of the multiple different behavioral attribute information in association with each of the multiple different behavioral attribute information in the storage unit 120. When behavioral attribute information is acquired by the first acquisition unit 131, the second acquisition unit 132 acquires a behavioral analysis prompt by referring to the storage unit 120 and selecting the behavioral analysis prompt corresponding to the behavioral attribute information from among multiple different behavioral analysis prompts associated with each of the multiple different behavioral attribute information by the first acquisition unit 131. In this way, the second acquisition unit 132 selects a behavioral analysis prompt that corresponds to the user's behavioral attributes from among a plurality of different behavioral analysis prompts associated with each of the plurality of different behavioral attribute information.

[0051] Furthermore, while the above-described embodiment explained the case where video information relating to a video of a user is a caption corresponding to the video, video information is not limited to captions corresponding to the video. For example, video information may be the video itself. For example, if the video information is a video, the language model may be a multimodal large-scale language model.

[0052] [5. Effects] As described above, the information processing device 100 according to the embodiment includes a first acquisition unit 131 and a second acquisition unit 132. The first acquisition unit 131 acquires behavioral attribute information relating to behavioral attributes that indicate the user's behavioral tendencies, based on video information relating to a video in which the user has been captured. The second acquisition unit 132 acquires behavioral analysis prompts, which are prompts corresponding to the user's behavioral attributes, based on the video information and behavioral attribute information, for analyzing the user's behavior in the video.

[0053] Thus, the information processing device 100 obtains prompts for analyzing the user's behavior in a video based on the user's behavior attribute information in the video, and obtains prompts corresponding to the user's behavior attributes. This allows the information processing device 100 to obtain prompts that enable efficient analysis of the user's behavior in the video. Furthermore, because the information processing device 100 can obtain prompts that enable efficient analysis of the user's behavior in the video, it can contribute to achieving Sustainable Development Goal (SDG) 9, "Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation." In addition, the information processing device 100 can efficiently analyze the user's behavior in a video by inputting prompts that enable efficient analysis of the user's behavior in the video into a language model, for example. Moreover, as mentioned above, prompts corresponding to the user's behavior attributes in a video generally require fewer characters than prompts where the user's behavior attributes are unknown. Therefore, the information processing device 100 can reduce the number of tokens input into the language model. Furthermore, because the information processing device 100 can reduce the number of tokens input into the language model, it can reduce the analysis time of the user's behavior in the video by the language model and the cost of using tokens. Furthermore, since the information processing device 100 can reduce the time required for the language model to analyze the user's behavior in the video, it is possible to speed up the analysis process of the user's behavior in the video using the language model.

[0054] Furthermore, the second acquisition unit 132 generates a generation prompt, which is a prompt that instructs the system to generate a behavioral analysis prompt from the video information and behavioral attribute information, and also inputs the video information and behavioral attribute information into the language model to generate a behavioral analysis prompt.

[0055] As a result, the information processing device 100 can efficiently generate prompts for analyzing the user's behavior in the video, using a language model, and these prompts are tailored to the user's behavioral attributes.

[0056] Furthermore, the second acquisition unit 132 generates a generation prompt, which is a prompt that instructs the system to generate a behavioral analysis prompt from user attribute information relating to the user's user attributes, and inputs the user attribute information into the language model to generate a behavioral analysis prompt.

[0057] As a result, the information processing device 100 can generate prompts for analyzing the user's behavior in the video based on the user's user attribute information, and these prompts are tailored to the user's user attributes. Furthermore, by using a language model, the information processing device 100 can efficiently generate prompts for analyzing the user's behavior in the video, tailored to the user's user attributes.

[0058] Furthermore, the first acquisition unit 131 inputs an attribute prompt, which is a prompt instructing the system to generate behavioral attribute information from the video information, and the video information into the language model to generate behavioral attribute information.

[0059] As a result, the information processing device 100 can appropriately generate behavioral attribute information corresponding to the video information based on the video information. Furthermore, by using a language model, the information processing device 100 can efficiently generate user behavioral attribute information.

[0060] Furthermore, the first acquisition unit 131 provides an attribute prompt, which is a prompt that instructs the system to generate behavioral attribute information from user attribute information relating to the user's user attributes, and inputs the user attribute information into the language model to generate behavioral attribute information.

[0061] As a result, the information processing device 100 can appropriately generate behavioral attribute information based on user attribute information. Furthermore, the information processing device 100 can efficiently generate user behavioral attribute information by using a language model.

[0062] Furthermore, video information includes captions corresponding to the video.

[0063] As a result, the information processing device 100 can efficiently acquire user behavior attribute information based on the captions corresponding to the video. Furthermore, the information processing device 100 can efficiently acquire prompts for analyzing the user's behavior in the video, based on the captions corresponding to the video, and prompts that are appropriate to the user's behavior attributes.

[0064] Furthermore, the second acquisition unit 132 selects a behavioral analysis prompt that corresponds to the user's behavioral attributes from among several different behavioral analysis prompts associated with each of several different behavioral attribute information.

[0065] As a result, the information processing device 100 can efficiently acquire prompts that analyze the user's actions in the video, and that correspond to the user's behavioral attributes.

[0066] The information processing device 100 also includes a generation unit 133 and a notification unit 134. The generation unit 133 inputs the video and behavioral analysis prompts to a language model and generates analysis information showing the results of the analysis of the user's behavior in the video. The notification unit 134 notifies the person monitoring the video of the analysis information.

[0067] This allows the information processing device 100 to efficiently analyze the user's actions in the video. Furthermore, the information processing device 100 can notify the video monitor of the analysis results, which are derived from the efficient analysis of the user's actions in the video. This reduces the monitoring burden on the video monitor to monitor the user in the video.

[0068] [6. Hardware Configuration] Furthermore, the information processing device 100 according to the above-described embodiment is realized by a computer 1000 having a configuration such as that shown in Figure 9. The following explanation will use the information processing device 100 as an example. Figure 9 is a diagram showing an example of the hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which an arithmetic unit 1030, a primary storage device 1040, a secondary storage device 1050, an output interface 1060, an input interface 1070, and a network interface 1080 are connected by a bus 1090.

[0069] The arithmetic unit 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, as well as programs read from the input device 1020, and executes various processes. The arithmetic unit 1030 can be implemented using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array).

[0070] The primary storage device 1040 is a memory device, such as RAM (Random Access Memory), that temporarily stores data used by the arithmetic unit 1030 for various calculations. The secondary storage device 1050 is a storage device where data used by the arithmetic unit 1030 for various calculations and various databases are registered, and can be implemented using ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, etc. The secondary storage device 1050 may be internal storage or external storage. The secondary storage device 1050 may also be a removable storage medium such as USB (Universal Serial Bus) memory or SD (Secure Digital) memory card. The secondary storage device 1050 may also be cloud storage (online storage), NAS (Network Attached Storage), file server, etc.

[0071] The output I / F 1060 is an interface for transmitting information to be output to output devices 1010, such as displays, projectors, and printers, and is implemented using connectors of standards such as USB (Universal Serial Bus), DVI (Digital Visual Interface), and HDMI (High Definition Multimedia Interface). The input I / F 1070 is an interface for receiving information from various input devices 1020, such as mice, keyboards, keypads, buttons, and scanners, and is implemented using, for example, USB.

[0072] Furthermore, the output interface 1060 and input interface 1070 may be wirelessly connected to the output device 1010 and input device 1020, respectively. In other words, the output device 1010 and input device 1020 may be wireless devices.

[0073] Furthermore, the output device 1010 and the input device 1020 may be integrated as a touch panel. In this case, the output I / F 1060 and the input I / F 1070 may also be integrated as an input / output I / F.

[0074] The input device 1020 may also be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), DVD (Digital Versatile Disc), or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0075] The network interface 1080 receives data from other devices via network N and sends it to the computing unit 1030, and also transmits data generated by the computing unit 1030 to other devices via network N.

[0076] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output interface 1060 and the input interface 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0077] For example, when computer 1000 functions as an information processing device 100, the arithmetic unit 1030 of computer 1000 realizes the functions of the control unit 130 by executing a program loaded onto the primary storage device 1040. Alternatively, the arithmetic unit 1030 of computer 1000 may load a program obtained from another device via the network interface 1080 onto the primary storage device 1040 and execute the loaded program. Furthermore, the arithmetic unit 1030 of computer 1000 may cooperate with other devices via the network interface 1080 and call and use program functions, data, etc., from other programs on other devices.

[0078] [7. Other] Although embodiments of the present invention have been described above, the present invention is not limited by the content of these embodiments. Furthermore, the aforementioned components include those that can be easily conceived by those skilled in the art, those that are substantially the same, and those that fall within the so-called equivalent range. Moreover, the aforementioned components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the gist of the embodiments described above.

[0079] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0080] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0081] For example, the information processing device 100 described above may be implemented using multiple server computers, and depending on the function, it may be implemented by calling external platforms, etc., via APIs (Application Programming Interfaces) or network computing, allowing for flexible configuration changes.

[0082] Furthermore, the embodiments and modifications described above can be combined as appropriate, provided that the processing content is not inconsistent. [Explanation of Symbols]

[0083] 1. Information Processing System 10 Terminal devices 100 Information Processing Devices 110 Communications Department 120 Storage section 121 User Database 130 Control Unit 131 First acquisition part 132 Second acquisition part 133 Generation part 134 Notification Department

Claims

1. A first acquisition unit acquires behavioral attribute information relating to behavioral attributes that indicate the user's behavioral tendencies based on video information relating to a video of the user, A second acquisition unit acquires a behavioral analysis prompt, which is a prompt corresponding to the user's behavioral attributes, for analyzing the user's behavior in the video based on the video information and the behavioral attribute information. An information processing device equipped with the following features.

2. The aforementioned second acquisition unit is, A generation prompt which is a prompt that instructs the generation of the behavioral analysis prompt from the video information and the behavioral attribute information, and a language model which inputs the video information and the behavioral attribute information to generate the behavioral analysis prompt. The information processing apparatus according to claim 1.

3. The aforementioned second acquisition unit is, The generation prompt is a prompt that instructs the system to generate the behavioral analysis prompt from user attribute information relating to the user attributes of the user, and the system inputs the user attribute information into the language model to generate the behavioral prompt. The information processing apparatus according to claim 2.

4. The first acquisition unit is, An attribute prompt is a prompt that instructs the system to generate the behavioral attribute information from the video information, and a language model is used to input the video information and generate the behavioral attribute information. The information processing apparatus according to claim 1.

5. The first acquisition unit is, The attribute prompt is a prompt that instructs the system to generate the behavioral attribute information from user attribute information relating to the user attributes of the user, and the system inputs the user attribute information into the language model to generate the behavioral attribute information. The information processing apparatus according to claim 4.

6. The aforementioned video information includes information that includes captions corresponding to the video. The information processing apparatus according to claim 1.

7. The aforementioned second acquisition unit is, From among multiple different behavioral analysis prompts associated with each of multiple different behavioral attribute information, a behavioral analysis prompt corresponding to the user's behavioral attribute is selected. The information processing apparatus according to claim 1.

8. A generation unit inputs the aforementioned video and the aforementioned behavioral analysis prompt into a language model to generate analysis information showing the results of the user's behavior in the aforementioned video. A notification unit that notifies the person monitoring the video of the aforementioned analysis information, The information processing apparatus according to claim 1, comprising:

9. An information processing method implemented by a program executed by an information processing device, A first acquisition step involves acquiring behavioral attribute information relating to behavioral attributes that indicate the user's behavioral tendencies, based on video information relating to a video of the user. A second acquisition step of acquiring a behavioral analysis prompt, which is a prompt corresponding to the user's behavioral attributes, for analyzing the user's behavior in the video based on the video information and the behavioral attribute information. Information processing methods including

10. A first acquisition procedure for acquiring behavioral attribute information relating to behavioral attributes that indicate the tendencies of the user's behavior, based on video information relating to a video of the user, A second acquisition procedure for acquiring a behavioral analysis prompt, which is a prompt corresponding to the user's behavioral attributes, based on the video information and the behavioral attribute information, and is a prompt for analyzing the user's behavior in the video. An information processing program that causes a computer to execute something.

Images