Image analysis device, image analysis method, and recording medium

Abstract

An image analysis device according to the present disclosure comprises a specifying unit and a detecting unit. The specifying unit specifies a target region within an in-store image in which an analysis target product appears. The detecting unit analyzes the in-store image using information indicating the target region, to detect, from within the in-store image, a person who has performed a predetermined action with respect to the analysis target product.

Description

Image analysis device, image analysis method, and recording medium

[0001] The present disclosure relates to an image analysis device, an image analysis method, and a program.

[0002] A technology related to this disclosure is disclosed in Patent Document 1. Patent Document 1 discloses a technology for detecting, at a relatively low cost, events that suggest the unauthorized removal of merchandise from a store. This technology uses image analysis to detect the removal of a large number of merchandise items at once. This technology then detects a person who appears in an image taken close to the time the removal was detected as a person who may have removed the merchandise.

[0003] Japanese Patent Application Laid-Open No. 2022-019797

[0004] Various technologies using images taken inside a store have been developed. If new technologies using images taken inside a store are realized, images taken inside the store can be used more effectively.

[0005] One example of the purpose of this disclosure is to provide a new technology using images taken inside a store.

[0006] According to this disclosure, an image analysis device is provided that has: an identification means for identifying a target area in an in-store image in which a product to be analyzed is captured; and a detection means for analyzing the in-store image using information indicating the target area to detect a person who has performed a predetermined action on the product to be analyzed from the in-store image.

[0007] Furthermore, according to this disclosure, an image analysis method is provided in which one or more computers identify a target area in an in-store image in which a product to be analyzed is captured, and analyze the in-store image using information indicating the target area, thereby detecting a person who has performed a predetermined action on the product to be analyzed from within the in-store image.

[0008] Furthermore, according to this disclosure, a program is provided that causes a computer to function as: an identification means for identifying a target area in an in-store image in which a product to be analyzed is captured; and a detection means for analyzing the in-store image using information indicating the target area to detect from the in-store image a person who has performed a predetermined action on the product to be analyzed.

[0009] According to one aspect of the present disclosure, a new technology is realized using images taken inside a store.

[0010] FIG. 1 is a diagram showing an example of a functional block diagram of an image analyzing device. FIG. 2 is a flowchart showing an example of a processing flow of the image analyzing device. FIG. 3 is a diagram showing an example of a hardware configuration of the image analyzing device. FIG. 4 is a diagram showing an example of an image processed by the image analyzing device. FIG. 5 is a diagram showing an example of a target region. FIG. 6 is a diagram showing an example of an observation region. FIG. 7 is a diagram showing an example of a method for determining an observation region. FIG. 8 is a flowchart showing another example of a processing flow of the image analyzing device.

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In this disclosure, the drawings relate to one or more embodiments. In all drawings, similar components are designated by similar reference numerals, and descriptions thereof will be omitted as appropriate.

[0012] <<First Embodiment>> Fig. 1 is a functional block diagram showing an overview of an image analyzing device 10. Fig. 2 is a flowchart showing an example of the flow of processing executed by the image analyzing device 10.

[0013] 1, the image analysis device 10 includes an identification unit 11 and a detection unit 12. These functional units execute the processing of the flowchart in FIG.

[0014] In S10, the identification unit 11 identifies a target area in the in-store image in which the analysis target product is captured. In S11, the detection unit 12 analyzes the in-store image using information indicating the target area identified in S10, thereby detecting a person who has performed a predetermined action on the analysis target product from within the in-store image.

[0015] In this way, the image analysis device 10 can detect "a person who has performed a predetermined action on a specific product (a product to be analyzed) among various products" from images of the store. The predetermined action is an action that can be identified by image analysis. Various actions can be defined as the predetermined action depending on the application. For example, the predetermined action may be, but is not limited to, an action of picking up the product to be analyzed or an action of picking up the product to be analyzed and taking it away.

[0016] Furthermore, the image analysis device 10 uses the "information indicating the target area in which the analysis target product appears" identified in the in-store image to detect a person who has performed a predetermined action on the analysis target product from within the in-store image. By using the information indicating the target area in which the analysis target product appears, it is possible to accurately detect a person who has performed a predetermined action on the analysis target product from within the in-store image.

[0017] Such an image analysis device 10 can be used for a variety of purposes.

[0018] For example, a product that has been lost due to fraudulent acts such as shoplifting can be set as a product to be analyzed, and a person who has performed a predetermined action on such a product to be analyzed can be detected as a person who may have committed fraud such as shoplifting.

[0019] In another example, newly released products, promotional products, etc. can be analyzed as products to be analyzed. Then, people who have performed a predetermined action on such products to be analyzed can be detected from in-store images, and the detection results can be used for marketing, etc. For example, based on the detection results, it is possible to calculate trends in the attributes (gender, age group, nationality, etc.) of people who performed the predetermined action, or trends in the timing (time, day of the week, month, etc.) when the predetermined action was performed.

[0020] The uses of the image analysis device 10 are not limited to the examples given here.

[0021] <<Second Embodiment>> <Overview> An image analysis device 10 according to a second embodiment is a specific implementation of the configuration of the image analysis device 10 according to the first embodiment. A detailed description will be given below.

[0022] <Hardware Configuration> First, an example of the hardware configuration of the image analysis device 10 will be described. Each functional unit of the image analysis device 10 is realized by any combination of hardware and software. Those skilled in the art will understand that there are various variations in the realization method and device. The software includes programs that are pre-loaded when the device is shipped, and programs downloaded from recording media such as CDs (Compact Discs) or servers on the Internet.

[0023] FIG. 3 is a block diagram illustrating an example of the hardware configuration of an image analysis device 10. As shown in FIG. 3, the image analysis device 10 has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The image analysis device 10 does not necessarily have to have the peripheral circuit 4A. Note that the image analysis device 10 may be composed of multiple devices that are physically and / or logically separated. In this case, each of the multiple devices can have the above hardware configuration.

[0024] The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to mutually transmit and receive data. The processor 1A is, for example, a central processing unit (CPU) or a graphics processing unit (GPU). The memory 2A is, for example, a random access memory (RAM) or a read-only memory (ROM). The input / output interface 3A includes interfaces for acquiring information from input devices, external devices, external servers, external sensors, cameras, etc., and interfaces for outputting information to output devices, external devices, external servers, etc. The input / output interface 3A also includes an interface for connecting to a communication network such as the Internet. Examples of input devices include a keyboard, mouse, microphone, physical buttons, and touch panel. Examples of output devices include a display, projection device, speaker, printer, and mailer. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

[0025] <Functional Configuration> Next, a detailed description will be given of the functional configuration of the image analysis device 10. Fig. 1 shows an example of a functional block diagram of the image analysis device 10. As shown in the figure, the image analysis device 10 has an identification unit 11 and a detection unit 12.

[0026] The identification unit 11 identifies a target area in the in-store image in which the analysis target product is captured.

[0027] An "in-store image" is an image generated by capturing an image of the inside of a store with a camera. Fig. 4 shows an example of an in-store image D. In the in-store image D of Fig. 4, a display shelf S, products P displayed on the display shelf S, and an aisle A are shown.

[0028] The "camera" detects visible light and creates an image. The camera may also detect other electromagnetic waves such as infrared light and create an image. The camera may capture moving images, may capture still images at a predetermined timing, or may continuously capture still images at predetermined time intervals. The camera may have a function to register the generated in-store images by linking them to the date and time of capture. The in-store images generated by the camera are input into the image analysis device 10 by real-time processing or batch processing. There are no restrictions on the means for inputting the in-store images generated by the camera into the image analysis device 10, and any method may be used.

[0029] In one example, the camera is a fixed camera that is installed at a predetermined position in the store and does not move. One or more cameras are installed at predetermined positions in the store. For example, the camera may be installed on the ceiling or wall of the store. Alternatively, the camera may be installed on fixtures (display shelves, POS registers, etc.) installed in the store.

[0030] The camera may be installed in a store in a position and orientation that captures images of merchandise displayed in the store. The camera may also be installed in a store in a position and orientation that captures images of customers moving around the store. The camera may also be installed in a store in a position and orientation that captures images of a payment area in the store. The payment area is an area where a point of sales (POS) register is installed and where payment processing is performed.

[0031] In another example, the camera is installed on a mobile object that moves within the store. The camera then takes images at various locations within the store to generate images of the store. The mobile object may be configured to move along the floor, walls, or ceiling. Alternatively, the mobile object may be an aerial vehicle that moves through the air. The mobile object may be configured to move in response to an operator's operation, or may be configured to be capable of autonomous movement. At least one of the mobile object and the camera may have a function to acquire current location information indicating the current location and register it in association with the current time.

[0032] The "analysis target product" is a product to be analyzed that is selected from a large number of products displayed in a store. Various products can be selected as the analysis target product depending on the application or purpose. In the second embodiment, the analysis target product is a product that has been the subject of fraudulent activity such as shoplifting, or a product that may have been the subject of fraudulent activity such as shoplifting.

[0033] The "target area" is a partial area in the in-store image, in which the product to be analyzed is captured.

[0034] As shown in Fig. 5, the identification unit 11 identifies a target area W, which is a partial area in the in-store image D and in which the analysis target product is captured. In the example of Fig. 5, the analysis target product is a product P labeled with the words "salted potato chips."

[0035] Here, the process of identifying the target region will be described.

[0036] The identification unit 11 displays the in-store image on a display device such as a display. Next, the identification unit 11 accepts a user input specifying a partial area of ​​the in-store image on the displayed in-store image. Then, the identification unit 11 identifies the partial area of ​​the in-store image specified by the user input as a target area. The identification unit 11 can accept the user input via any input device such as a touch panel, a mouse, a keyboard, a physical button, or a microphone.

[0037] For example, the identification unit 11 may display a frame superimposed on the in-store image and accept a user input to change the position, size, shape, etc. of the frame. Then, the identification unit 11 may identify the area surrounded by the frame as the target area.

[0038] Alternatively, the identification unit 11 may analyze the in-store image to detect objects or products, and display multiple frames surrounding each object or product superimposed on the in-store image. The identification unit 11 may then accept a user input specifying at least one of the multiple frames. The identification unit 11 can identify the area surrounded by the specified frame as the target area. Object detection and product detection can be achieved using any well-known technology. Product detection can be achieved using pre-registered appearance information of each product.

[0039] The identification unit 11 may further accept a user input specifying an in-store image to be displayed on the display device from among the multiple in-store images (i.e., an in-store image to be subjected to the process of specifying a target area).The identification unit 11 may then display the in-store image specified by the user input on the display device and accept a user input specifying a target area on the in-store image.The user input specifying the in-store image to be displayed may include an input specifying at least one from multiple cameras, an input specifying the shooting date and time, an input to move the displayed portion of the video forward or backward by operations such as fast forward, rewind, frame forward, playback, and slow playback, etc.

[0040] Based on various information such as the results of in-store patrols and inventory, the user recognizes products that have been or may have been subject to fraudulent activity such as shoplifting as products to be analyzed.The user then inputs information to specify the target area in the in-store image in which the recognized analysis target product is captured.

[0041] 1 , the detection unit 12 analyzes the in-store image using the information indicating the target area to detect from the in-store image a person who has performed a predetermined action on the product being analyzed. The detection unit 12 can perform this detection by analyzing past in-store images.

[0042] A "predetermined action" is an action that can be identified by image analysis. Various actions can be defined as the predetermined action depending on the application or purpose.

[0043] As described above, in the second embodiment, the analysis target product is a product that has been the subject of fraudulent activity such as shoplifting, or a product that may have been the subject of fraudulent activity such as shoplifting. The predetermined action in such a case may be, for example, an action showing interest in the analysis target product. For example, the predetermined action may include at least one of the following:

[0044] ・Action of picking up an object in the target area ・Action of picking up an object in the target area and taking it away ・Action of looking at the target area ・Action of being located in a part of the store image identified based on the target area ・Action of staying in a part of the store image identified based on the target area for more than a specified period of time ・Action of assuming a specified posture in a part of the store image identified based on the target area

[0045] Each action will be explained below.

[0046] "Action of picking up an object in the target area" The object in the target area is the product to be analyzed. Therefore, the action of picking up an object in the target area is the action of picking up the product to be analyzed.

[0047] The action of picking up an object in the target area can be detected using any well-known technology. For example, the detection unit 12 may detect a person's hand in the store image. Then, the detection unit 12 may determine that the person has picked up an object in the target area when the hand comes into contact with the object in the target area and the position of the object in the target area moves while maintaining the contact state.

[0048] "Action of picking up an object in the target area and taking it away" The object in the target area is the product to be analyzed. Therefore, the action of picking up an object in the target area and taking it away is the action of picking up the product to be analyzed and taking it away.

[0049] The action of picking up an object in the target area can be detected using any well-known technology. The action of taking away an object can also be detected using any well-known technology. For example, the detection unit 12 may determine that an object has been taken away if an object that has been picked up by a person and moved out of the target area satisfies a predetermined condition (such as being out of the frame or no longer being detected in the store image) without returning to the target area.

[0050] "Action to view target area" The target area contains the product to be analyzed. Therefore, the action to view the target area is the action to view the product to be analyzed.

[0051] The action of looking at a target area can be detected using any widely known technology. For example, the detection unit 12 detects a person in the store image and detects the person's gaze direction. The detection unit 12 can then detect the action of looking at a target area by determining whether the target area is located in the gaze direction. The detection of a person and the detection of the gaze direction can be realized using any widely known technology.

[0052] "Action Located in a Partial Area Within the Store Image Identified Based on the Target Area" First, the detection unit 12 identifies a partial area within the store image based on the target area. The "partial area within the store image identified based on the target area" is an area where a person interested in the analysis target product located in the target area will be located. Hereinafter, the partial area within the store image identified based on the target area will be referred to as the "observation area." For example, an area close to the target area, an area from which the target area can be seen, or an area from which the analysis target product located in the target area can be picked up may be identified as the observation area. The detection unit 12 identifies an observation area B within the store image D, for example, as shown in FIG. 6 .

[0053] Here, the process of specifying the observation area will be described.

[0054] In one example, an observation area is determined in advance by linking it to each of a plurality of product display areas in the in-store image, and observation area information indicating the observation area for each product display area is stored in a predetermined storage device. The observation area information indicates the area occupied by each of the plurality of product display areas in the in-store image, the area occupied by each of the plurality of observation areas in the in-store image, and the linking relationship between the product display areas and the observation areas. The predetermined storage device may be provided within the image analysis device 10, or may be provided in an external device communicatively connected to the image analysis device 10. The same assumption regarding the predetermined storage device applies hereinafter.

[0055] The detection unit 12 then refers to the observation area information to identify a product display area corresponding to the target area, and then identifies an observation area linked to the identified product display area. A "product display area corresponding to the target area" is a product display area that includes the target area, a product display area that is included in the target area, a product display area that partially overlaps with the target area, or a product display area that overlaps with the target area by a predetermined percentage or more.

[0056] In another example, the area occupied by the display shelves in the in-store image is stored in advance in a predetermined storage device. The area occupied by the display shelves in the in-store image may be specified and registered by a user, or may be registered using other methods. Then, as shown in FIG. 7 , the detection unit 12 extends a "straight line L passing through a predetermined point in the target area W and extending in the vertical direction of the in-store image D" downward in the in-store image D, and obtains an intersection C between the boundary of the area occupied by the display shelves S (the boundary between the bottom of the display shelves S and the floor) and the line L. The predetermined point in the target area W is, for example, the center of a side extending in the horizontal direction of the rectangular target area W, but is not limited to this. The detection unit 12 then identifies, as the observation area B, a circle with a predetermined radius centered on the obtained intersection C. The predetermined radius is a value registered in advance. The detection unit 12 may also identify, as the observation area B, an area obtained by removing the area occupied by objects such as the display shelves S from the circle with a predetermined radius centered on the intersection C.

[0057] After identifying the observation area B as shown in FIG. 6, the detection unit 12 detects the person located in the observation area B, thereby detecting the action.

[0058] "An action of staying in a partial area within the store image identified based on the target area for a predetermined period of time or longer" The detection unit 12 detects a person located within the observation area (a partial area within the store image identified based on the target area) as described above. The detection unit 12 then tracks the detected person within the video image (store image) and determines whether the person has stayed in the observation area for a predetermined period of time or longer, thereby detecting the action.

[0059] "An action of assuming a predetermined posture in a partial area of ​​the store image identified based on the target area" The detection unit 12 detects a person located in the observation area (a partial area of ​​the store image identified based on the target area) as described above. Then, the detection unit 12 detects the action by determining whether the posture of the detected person is a predetermined posture.

[0060] The person's posture is determined using posture detection technology such as OpenPose, MMPose, etc. The predetermined posture may be, but is not limited to, a posture of reaching out to a display shelf, a posture of picking up an item from a display shelf, etc.

[0061] After detecting a person who has performed a predetermined action on the commodity to be analyzed from within the in-store image, the detection unit 12 can perform at least one of the following.

[0062] - Processing to display images of the store in which the detected person appears - Processing to play back scenes in which the person who performed a specified action on the product being analyzed appears afterwards - Processing to register the detected person in a list

[0063] Each process will be explained below.

[0064] "Process for displaying an in-store image in which a detected person appears" The detection unit 12 displays an in-store image in which a person detected as a person who performed a predetermined action on a product being analyzed is appearing. For example, the detection unit 12 may display an in-store image in which the detected person appears on a display device provided in the image analysis device 10. Alternatively, the detection unit 12 may transmit the in-store image in which the detected person appears to a mobile device carried by a store clerk for display, or may transmit the in-store image in which the detected person appears to another terminal operated by the store clerk (such as a POS register or a terminal installed in the backroom) for display.

[0065] By checking the images of the store interior, store staff can recognize individuals who may have engaged in shoplifting or other fraudulent activities. When such individuals enter the store, store staff can speak to the individuals and pay close attention to their movements, thereby preventing shoplifting or other fraudulent activities from occurring.

[0066] "Process of playing back a scene in which a person who has taken a predetermined action on a product to be analyzed appears afterwards" "Afterwards" means "after a predetermined action has been taken on a product to be analyzed." By checking the behavior of a person who has taken a predetermined action on a product to be analyzed afterwards, it may be possible to determine whether that person has committed fraud such as shoplifting. Therefore, the detection unit 12 detects a scene (in-store image) in which a person who has taken a predetermined action on a product to be analyzed appears afterwards, and plays back the detected scene.

[0067] The detection unit 12 analyzes the in-store images and extracts appearance information of a person who has performed a predetermined action on a product to be analyzed. Then, the detection unit 12 detects the detected person from other in-store images based on the extracted appearance information. Then, the detection unit 12 can identify a scene in which the detected person appears based on the detection result. If there are multiple cameras capturing images of the inside of the store, the detection unit 12 can detect a scene in which the detected person appears from the video images (in-store images) generated by each of the multiple cameras.

[0068] For example, the detection unit 12 can cause the detected scene to be played back and displayed on a display device included in the image analysis device 10. Alternatively, the detection unit 12 can transmit the moving image (image of the store interior) to a mobile device carried by a store clerk and cause the detected scene to be played back and displayed on the mobile device. Alternatively, the detection unit 12 can transmit the moving image (image of the store interior) to another terminal operated by the store clerk (such as a POS register or a terminal installed in the back room) and cause the detected scene to be played back and displayed on the other terminal.

[0069] "Process of registering detected individuals in a list" As described above, in the second embodiment, the analysis target products are products that have been the subject of fraudulent acts such as shoplifting, or products that may have been the subject of fraudulent acts such as shoplifting. The detection unit 12 then detects individuals who have taken a predetermined action (picking up, taking away, etc.) that shows an interest in such analysis target products. It is highly likely that individuals who have committed fraud such as shoplifting are included among those who have taken such predetermined actions.

[0070] Therefore, the detection unit 12 registers the detected person (a person who performed a predetermined action on the product to be analyzed) in a list. For example, the detection unit 12 can register an image of the detected person (an image cut out from an in-store image) in the list. The detection unit 12 may also register, in the list, information indicating the product to be analyzed that was the target of a predetermined action by the detected person, the date and time when the predetermined action was performed, etc., linked to the detected person. The list is stored in a predetermined storage device.

[0071] The image analyzing device 10 may execute processing using the list generated in this manner. For example, the image analyzing device 10 may execute processing to detect a person registered on the list within an in-store image. This processing allows the image analyzing device 10 to detect a re-entry of a person registered on the list within an in-store image. When the image analyzing device 10 detects a person registered on the list within an in-store image, it can notify a store clerk or the like of this fact.

[0072] When notifying the store clerk, the clerk may be notified of an image of the detected person (a person registered on the list), information indicating the analysis target product that was the target of a predetermined action by the detected person, the date and time when the predetermined action was performed, etc. Notification to the store clerk is realized via a mobile terminal carried by the store clerk or another terminal operated by the store clerk (such as a POS register or a terminal installed in the back room). By receiving the notification, the store clerk can speak to the detected person or pay attention to the detected person's movements, thereby preventing shoplifting and other fraudulent acts from occurring.

[0073] The image analysis device 10 may perform the "process of detecting a person registered on the list in store images" in real time or in batch processing. In the case of batch processing, it is not possible to take action such as calling out to the person on the spot as described above. However, based on the notification from the image analysis device 10, the store clerk can recognize that a person registered on the list has returned to the store, and can check the store images to confirm the person's movements within the store when they return.

[0074] Next, an example of the flow of processing executed by the image analyzing device 10 will be described with reference to the flowchart of Fig. 8. Details of each process have been described above, so further description will be omitted here.

[0075] First, the image analysis device 10 identifies an area designated by a user in the in-store image as a target area in which the analysis target product appears in the in-store image (S20). Next, the image analysis device 10 analyzes the in-store image using information indicating the target area, thereby detecting a person who has performed a predetermined action on the analysis target product from within the in-store image (S21).

[0076] "Effects" According to the image analysis device 10 of the second embodiment, the same effects as those of the image analysis device 10 of the first embodiment are achieved.

[0077] Furthermore, the image analysis device 10 identifies an area designated by the user in the in-store image as a target area. The image analysis device 10 then detects, from within the in-store image, a person who has performed a predetermined action on a product to be analyzed that is located in the designated target area. By designating an area in the in-store image that shows the product to be analyzed as the target area, the user can obtain detection results for a person who has performed a predetermined action on the product to be analyzed.

[0078] Furthermore, the image analyzing device 10 can detect people who have performed the above-described characteristic predetermined action on the product being analyzed. The above-described characteristic predetermined action is an action that shows interest in the product being analyzed. There is a high possibility that people who have performed such a predetermined action on the product being analyzed include people who have committed fraudulent acts such as shoplifting. The image analyzing device 10 can detect people who may have committed fraudulent acts such as shoplifting with a high degree of accuracy.

[0079] Furthermore, after detecting a person who has performed a predetermined action on a product to be analyzed, the image analyzing device 10 can display an in-store image in which the detected person appears, play back a scene in which the detected person appears, or register the detected person in a list. Such an image analyzing device 10 allows a store to take measures to prevent the inconvenience of shoplifting or other fraudulent acts from occurring again.

[0080] <<Third Embodiment>> The image analysis device 10 of the second embodiment receives user input specifying a target area in an in-store image in which a product to be analyzed appears. The image analysis device 10 of the third embodiment receives user input of product identification information for the product to be analyzed. The image analysis device 10 then identifies a target area in the in-store image in which the product to be analyzed identified by the product identification information appears. This will be described in detail below.

[0081] The identification unit 11 acquires product identification information of the analysis target product. The product identification information is information for identifying multiple products from each other, such as product name, product code, etc., but is not limited to these.

[0082] The identification unit 11 acquires the product identification information input by the user as the product identification information of the product to be analyzed. The user may directly input the product identification information. Alternatively, the user may input by selecting predetermined product identification information from multiple pieces of product identification information displayed in a selectable manner. The identification unit 11 can accept the user input via a UI (user interface) screen that displays various UI components.

[0083] After acquiring the product identification information of the product to be analyzed, the identification unit 11 identifies a target area in the in-store image in which the product to be analyzed is captured, using at least one of display position information indicating the display position of each of the multiple products and appearance information of each of the multiple products. At least one of the display position information indicating the display position of each of the multiple products and the appearance information of each of the multiple products is stored in advance in a predetermined storage device.

[0084] The "display position information" may indicate on which display shelf each product is displayed. The display position information may also indicate on which shelf each product is displayed. The display position information may also indicate the position from the left (or the position from the right) on a certain shelf of the display shelf each product is displayed. The display position information may also indicate which fixed camera captures each display shelf in the in-store image.

[0085] The identification unit 11 uses such display position information to identify a target area in which the analysis target product is captured. For example, the identification unit 11 uses the display position information to identify a display shelf on which the analysis target product is displayed. Then, the identification unit 11 uses the display position information to identify a fixed camera that generates an in-store image in which the identified display shelf is captured. Then, the identification unit 11 identifies the in-store image generated by the identified fixed camera as an in-store image in which the analysis target product is captured.

[0086] The identification unit 11 then identifies a target area in the identified in-store image in which the analysis target product appears. For example, the identification unit 11 may identify a target area in the identified in-store image in which the analysis target product appears, using the "row on which the analysis target product is displayed" or the "display position on that row (number from the left or number from the right)" indicated in the display position information. Alternatively, the identification unit 11 may detect the analysis target product in the identified in-store image using pre-registered appearance information of the analysis target product. The identification unit 11 may then identify the area in which the detected analysis target product appears as the target area.

[0087] Alternatively, the identification unit 11 may identify the target area in which the analysis target product appears using only the appearance information of each of the multiple products, without using the display position information. In this case, the identification unit 11 performs a process of detecting the analysis target product in the in-store image using the appearance information of the analysis target product, with each of the multiple in-store images generated by the multiple cameras as the analysis target. Then, the identification unit 11 identifies the area in which the detected analysis target product appears as the target area.

[0088] The other configurations of the image analysis device 10 of the third embodiment are similar to those of the image analysis device 10 of the first and second embodiments.

[0089] According to the image analysis device 10 of the third embodiment, the same effects as those of the image analysis device 10 of the first and second embodiments are realized.

[0090] Furthermore, when the image analysis device 10 receives user input specifying the product identification information of a product to be analyzed, it identifies a target area in the in-store image that shows the specified product to be analyzed. The image analysis device 10 then detects, from the in-store image, a person who has performed a predetermined action on the product to be analyzed that is located in the identified target area. By specifying the product identification information of the product to be analyzed, the user can obtain a detection result of the person who performed the predetermined action on the product to be analyzed. This image analysis device 10 is advantageous because it reduces the input burden on the user.

[0091] <<Fourth Embodiment>> The image analysis device 10 of the third embodiment receives input from a user specifying a product to be analyzed. The image analysis device 10 of the fourth embodiment determines the product to be analyzed based on the inventory quantity managed based on sales data and the inventory quantity identified in the inventory process. The image analysis device 10 then identifies a target area in the in-store image that shows the determined product to be analyzed. This is described in detail below.

[0092] The identification unit 11 acquires the commodity identification information of the commodity to be analyzed, as in the third embodiment, and then identifies a target area in the in-store image in which the commodity to be analyzed is captured, using the method described in the third embodiment.

[0093] The identification unit 11 acquires product identification information of the analysis target product using a method different from that of the third embodiment. The identification unit 11 identifies the analysis target product based on the inventory quantity managed based on sales data and the inventory quantity identified in the inventory process. Specifically, the identification unit 11 compares the inventory quantity managed based on sales data with the inventory quantity identified in the inventory process for each product. The identification unit 11 then identifies, as the analysis target product, a product for which the inventory quantity managed based on sales data does not match the inventory quantity identified in the inventory process. The identification unit 11 acquires, as the analysis target product, the product identification information of a product for which the inventory quantity managed based on sales data does not match the inventory quantity identified in the inventory process.

[0094] The "inventory quantity managed based on sales data" is the number obtained by subtracting the number sold from the number received. The "inventory quantity identified in inventory processing" is the count of the number of products that actually exist (actual inventory quantity). In the case of products that have been subject to fraudulent acts such as shoplifting, a situation may arise in which the inventory quantity managed based on sales data does not match the inventory quantity identified in inventory processing. Note that the inventory quantity managed based on sales data and the inventory quantity identified in inventory processing are stored in advance in a specified storage device. The identification unit 11 uses this data to identify the product to be analyzed.

[0095] The other configurations of the image analysis device 10 of the fourth embodiment are similar to those of the image analysis device 10 of the first to third embodiments.

[0096] According to the image analysis device 10 of the fourth embodiment, the same effects as those of the image analysis devices 10 of the first to third embodiments are realized.

[0097] In addition, the image analysis device 10 identifies as products to be analyzed products whose inventory numbers managed based on sales data do not match the inventory numbers identified in the inventory process, i.e., products for which there is a possibility that fraudulent acts such as shoplifting have been committed.

[0098] With this image analysis device 10, the user does not need to input information to specify a target area or to input product identification information for a product to be analyzed, etc. This is preferable because the image analysis device 10 can reduce the input burden on the user.

[0099] Furthermore, with this image analysis device 10, it is possible to prevent the inconvenience of products that should be the subject of analysis (products that may have been subject to fraudulent acts such as shoplifting) being omitted from the products to be analyzed.

[0100] <<Fifth Embodiment>> In the image analyzing device 10 of the fifth embodiment, even if a person has performed a predetermined action on a product to be analyzed, if the person satisfies a predetermined avoidance condition, the person is excluded from the detection result (the detection result of a person who may have engaged in fraudulent activity such as shoplifting). This will be described in detail below.

[0101] The detection unit 12 detects, from within the in-store image, a person who has performed a predetermined action on the analysis target commodity and who does not satisfy the avoidance condition.

[0102] Specifically, the detection unit 12 detects a person who has performed a predetermined action on a product to be analyzed from within the in-store image using a method similar to that used in the first to fourth embodiments. Then, the detection unit 12 tracks the detected person within the in-store image after the person has performed the predetermined action, and determines whether the detected person satisfies the avoidance condition. Tracking within the in-store image is a concept that includes tracking within the in-store image generated by a single camera, and tracking across multiple in-store images generated by multiple cameras.

[0103] If the detected person satisfies the avoidance condition, the detection unit 12 removes (deletes) the person who satisfies the avoidance condition from the detection results of people who have taken a predetermined action on the product to be analyzed. On the other hand, if the detected person does not satisfy the avoidance condition, the detection unit 12 leaves the person who does not satisfy the avoidance condition in the detection results of people who have taken a predetermined action on the product to be analyzed.

[0104] The avoidance condition is a condition indicating that the detected person did not commit any fraudulent acts such as shoplifting and purchased the product to be analyzed. The avoidance condition may include, for example, at least one of the following.

[0105] - Passed through the payment area where payment processing was performed - Executed payment processing and registered the product to be analyzed as a payment target in the POS data

[0106] The "payment area" is an area where a POS register is installed and where payment processing is carried out. A part of the store is defined as the payment area in advance, and information indicating the payment area is registered in a predetermined storage device.

[0107] For example, the detection unit 12 may execute a process of detecting a person who has performed a predetermined action on the product to be analyzed from an in-store image generated by a camera installed at a position and orientation that captures the payment area. Then, when a person who has performed a predetermined action on the product to be analyzed is detected from the in-store image generated by the camera, the detection unit 12 may determine that the person has "passed through the payment area where payment processing is performed," i.e., satisfies the avoidance condition.

[0108] Alternatively, the detection unit 12 may execute a process of detecting a person who has performed a predetermined action on the product to be analyzed from an in-store image generated by a camera installed in a POS register. When a person who has performed a predetermined action on the product to be analyzed is detected from an in-store image generated by the camera, the detection unit 12 may determine that the person has "passed through a payment area where payment processing is performed," i.e., that the avoidance condition is satisfied.

[0109] If the POS register is designed to be operated by a customer, the camera may be installed in a position and orientation that captures the person operating the register. If the POS register is designed to be operated by a store clerk, the camera may be installed in a position and orientation that captures the person standing near the POS register waiting for the store clerk to complete the registration process.

[0110] Additionally, when the detection unit 12 detects a person who has performed a predetermined action on a product to be analyzed in an in-store image generated by a camera installed in a POS register, the detection unit 12 may acquire, from the POS system, the registration information registered by that person in the POS register. The detection unit 12 can acquire the registration information registered by that person in the POS register from the POS system using timestamp information, etc. That is, the detection unit 12 can acquire the registration information registered by that person based on the correspondence between the date and time the in-store image in which the person was detected (the in-store image generated by a camera installed in the POS register) was captured and the date and time the registration information was registered in the POS register. Then, when the registration information includes the product to be analyzed, the detection unit 12 may determine that the person "performed a payment process and the product to be analyzed was registered as a payment target in the POS data," i.e., satisfies the avoidance condition.

[0111] The other configurations of the image analysis device 10 of the fifth embodiment are similar to those of the image analysis device 10 of the first to fourth embodiments.

[0112] According to the image analysis device 10 of the fifth embodiment, the same effects as those of the image analysis devices 10 of the first to fourth embodiments are realized.

[0113] Furthermore, even if a person has performed a predetermined action on the product being analyzed, if that person satisfies a predetermined avoidance condition, the image analyzing device 10 will exclude that person from the detection results (detection results of people who may have engaged in fraudulent activities such as shoplifting).With this type of image analyzing device 10, it is possible to detect only people who may have engaged in fraudulent activities such as shoplifting with high accuracy.

[0114] <<Modifications>> <Modification 1> In the fifth embodiment, the image analyzing device 10 removes people who satisfy the avoidance condition from the detection results of people who have performed a predetermined action on the analysis target commodity.

[0115] As a variant example, the image analysis device 10 may display a list of images of multiple people who have performed a predetermined action on the product being analyzed (store images showing the people, or images of the people cut out from the store images).The image analysis device 10 may then distinguish between people who do not satisfy the avoidance conditions and people who satisfy the avoidance conditions in the list display. Alternatively, the image analysis device 10 may display images of people who do not satisfy the avoidance conditions at the beginning of the list display and images of people who satisfy the avoidance conditions at the end of the list display.In other words, the image analysis device 10 may display images of people who do not satisfy the avoidance conditions before images of people who satisfy the avoidance conditions in the list display.

[0116] Alternatively, the image analyzing device 10 may sequentially display images of multiple people who have performed a predetermined action on the product being analyzed (store images showing the people, or images of the people cut out from the store images).The image analyzing device 10 may first display images of people who do not satisfy the avoidance conditions, and then display images of people who satisfy the avoidance conditions.

[0117] The image analyzing device 10 may display images of multiple people who have performed a predetermined action on the product being analyzed on a display device included in the image analyzing device 10. Alternatively, the detection unit 12 may transmit the images to a mobile device carried by a store clerk for display, or may transmit the images to another terminal operated by a store clerk (such as a POS register or a terminal installed in the back room) for display.

[0118] <Modification 2> The image analyzing device 10 may detect a person holding a product to be analyzed from an in-store image. The product to be analyzed is specified in the same manner as in the first to fifth embodiments.

[0119] In the first to fifth embodiments, the image analysis device 10 detects a person who has performed a predetermined action on a product to be analyzed from within the in-store image by analyzing the in-store image using information indicating the target area.

[0120] In this modified example, the image analyzing device 10 detects a person holding the analysis target product from within the store image by analyzing the store image using appearance information of the analysis target product. In this detection, the image analyzing device 10 does not use information indicating the target area. The concept of a "person holding the analysis target product" includes a person holding the analysis target product in their hand, a person with the analysis target product in their shopping cart, etc.

[0121] <Modification 3> In the second to fifth embodiments, the image analyzing device 10 is used as a countermeasure against fraudulent acts such as shoplifting. However, the use of the image analyzing device 10 is not limited to this.

[0122] For example, a newly released product, a promotional product, etc. can be set as the product to be analyzed. For example, the target area in which such a product to be analyzed is captured can be identified using the methods described in the second and third embodiments. In this example, the "predetermined action with respect to the product to be analyzed" can also be the "action showing interest in the product to be analyzed" described in the second embodiment.

[0123] According to this modification, it is possible to detect from the in-store image a person who has taken an action that indicates an interest in the analysis target product, which may be a newly released product, a promotional product, etc. The detection result can then be used for marketing, etc.

[0124] For example, the image analysis device 10 may estimate the attributes of a person who has taken a predetermined action on the product being analyzed through image analysis. The attributes include gender, age group, nationality, etc. The image analysis device 10 may then statistically process the estimation results and output the results. The results of this statistical processing may allow for an understanding of the trends in the attributes of people who are interested in the product being analyzed.

[0125] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0126] In addition, in the flowcharts used in the above explanation, multiple steps (processes) are described in order. However, the order of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the drawings can be changed as long as it does not cause any problems in terms of the content.

[0127] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes. 1. An image analysis device comprising: an identification means for identifying a target area in an in-store image in which a product to be analyzed is captured; and a detection means for detecting a person from within the in-store image who has performed a predetermined action on the product to be analyzed by analyzing the in-store image using information indicating the target area. 2. The image analysis device described in 1, wherein the identification means identifies an area specified by a user in the in-store image as the target area. 3. The image analysis device described in 1, wherein the identification means acquires product identification information for the product to be analyzed, and identifies the target area in the in-store image in which the product to be analyzed is captured using at least one of display position information indicating the display position of each of a plurality of products and appearance information of each of the plurality of products. 4. The image analysis device described in 3, wherein the identification means identifies a product whose inventory count managed based on sales data does not match the inventory count identified in inventory processing, and acquires the product identification information of the identified product as the product identification information of the product to be analyzed. 5. 6. The image analyzing device according to any one of 1 to 5, wherein the identification means acquires commodity identification information input by a user as the commodity identification information of the commodity to be analyzed. 7. The image analyzing device according to any one of 1 to 5, wherein the predetermined action with respect to the commodity to be analyzed includes at least one of: an action of picking up an object in the target area, an action of picking up an object in the target area and taking it away, an action of looking at the target area, an action of positioning in a partial area within the in-store image identified based on the target area, an action of staying in a partial area within the in-store image identified based on the target area for more than a predetermined period of time, and an action of assuming a predetermined posture in a partial area within the in-store image identified based on the target area.7. The image analysis device according to any one of 1 to 6, wherein the detection means detects from the in-store image a person who has performed a predetermined action on the analysis target product and who does not satisfy an avoidance condition, the avoidance condition including at least one of: passing through a payment area where payment processing is performed, and performing payment processing and registering the analysis target product as a payment target in POS (point of sales) data. 8. The image analysis device according to any one of 1 to 7, wherein the detection means executes at least one of: registering the detected person in a list; displaying the in-store image in which the detected person appears; and playing back a scene in which the person who performed a predetermined action on the analysis target product appears afterwards. 9. An image analysis method in which one or more computers identify a target area in the in-store image in which the analysis target product appears, and analyze the in-store image using information indicating the target area to detect from the in-store image a person who has performed a predetermined action on the analysis target product. 10. A program that causes a computer to function as: an identification means that identifies a target area in an in-store image in which a product to be analyzed is captured; and a detection means that analyzes the in-store image using information indicating the target area to detect from the in-store image a person who has performed a predetermined action on the product to be analyzed.

[0128] Some or all of Supplements 2 to 8 that are dependent on the image analysis device of Supplement 1 described above may also be dependent on the image analysis method of Supplement 9 and the program of Supplement 10 in the same dependent relationship as Supplement 1 and Supplements 2 to 8. Furthermore, within the scope of each of the above-mentioned embodiments, some or all of the configurations described as Supplements can be realized in various hardware, software, various recording means for recording software, or systems.

[0129] This application claims priority based on Japanese Patent Application No. 2024-072255, filed April 26, 2024, the disclosure of which is incorporated herein in its entirety by reference.

[0130] 10 Information processing device 11 Identification unit 12 Detection unit 1A Processor 2A Memory 3A Input / output I / F 4A Peripheral circuit 5A Bus

Claims

1. An image analysis device having: an identification means for identifying a target area in an in-store image in which a product to be analyzed is captured; and a detection means for analyzing the in-store image using information indicating the target area to detect a person who has performed a predetermined action on the product to be analyzed from the in-store image.

2. The image analysis device according to claim 1, wherein the identification means identifies an area designated by a user within the store interior image as the target area.

3. The image analysis device of claim 1 or 2, wherein the identification means acquires product identification information of the product to be analyzed, and identifies the target area in the in-store image in which the product to be analyzed appears using at least one of display position information indicating the display position of each of the multiple products and appearance information of each of the multiple products.

4. The image analysis device according to claim 3, wherein the identification means identifies products whose inventory numbers managed based on sales data do not match the inventory numbers identified in the inventory process, and acquires the product identification information of the identified products as the product identification information of the products to be analyzed.

5. An image analysis device according to claim 3 or 4, wherein the identification means acquires product identification information input by a user as the product identification information of the product to be analyzed.

6. The image analysis device of any one of claims 1 to 5, wherein the predetermined action for the product to be analyzed includes at least one of the following: an action of picking up an object in the target area; an action of picking up an object in the target area and taking it away; an action of looking at the target area; an action of locating in a partial area within the in-store image identified based on the target area; an action of staying in a partial area within the in-store image identified based on the target area for more than a predetermined period of time; and an action of assuming a predetermined posture in a partial area within the in-store image identified based on the target area.

7. The image analysis device according to any one of claims 1 to 6, wherein the detection means detects from the in-store image a person who has performed a predetermined action on the product to be analyzed and who does not satisfy an avoidance condition, and the avoidance condition includes at least one of: passing through a payment area where payment processing is performed, and performing payment processing and registering the product to be analyzed as a payment item in POS (point of sales) data.

8. An image analysis device as claimed in any one of claims 1 to 7, wherein the detection means performs at least one of the following processes: registering the detected person in a list; displaying the in-store image in which the detected person appears; and playing back a scene in which a person who has performed a specified action on the product being analysed appears afterwards.

9. An image analysis method in which one or more computers identify a target area in an in-store image in which a product to be analyzed appears, and analyze the in-store image using information indicating the target area, thereby detecting a person who has performed a specified action on the product to be analyzed from the in-store image.

10. An image analysis method as described in claim 9, wherein in the process of identifying the target area, an area designated by a user within the store interior image is identified as the target area.

11. An image analysis method as described in claim 9 or 10, wherein the process of identifying the target area includes obtaining product identification information for the product to be analyzed, and using at least one of display position information indicating the display position of each of the multiple products and appearance information for each of the multiple products, identifying the target area in which the product to be analyzed appears within the in-store image.

12. The image analysis method according to claim 11, wherein the process of identifying the target area identifies products whose inventory numbers managed based on sales data do not match the inventory numbers identified in the inventory process, and obtains the product identification information of the identified products as the product identification information of the products to be analyzed.

13. An image analysis method according to claim 11 or 12, wherein in the process of identifying the target area, product identification information input by a user is obtained as the product identification information of the product to be analyzed.

14. An image analysis method according to any one of claims 9 to 14, wherein the predetermined action for the product to be analyzed includes at least one of the following: an action of picking up an object in the target area; an action of picking up an object in the target area and taking it away; an action of looking at the target area; an action of being located in a partial area within the in-store image identified based on the target area; an action of staying in a partial area within the in-store image identified based on the target area for more than a predetermined period of time; and an action of assuming a predetermined posture in a partial area within the in-store image identified based on the target area.

15. A recording medium having recorded thereon a program that causes a computer to function as: an identification means for identifying a target area in an in-store image in which a product to be analyzed is captured; and a detection means for detecting a person who has performed a specified action on the product to be analyzed from within the in-store image by analyzing the in-store image using information indicating the target area.

16. The recording medium according to claim 15, wherein the specifying means specifies an area designated by a user within the store interior image as the target area.

17. A recording medium as described in claim 15 or 16, wherein the identification means acquires product identification information of the product to be analyzed, and identifies the target area in the in-store image in which the product to be analyzed is captured using at least one of display position information indicating the display position of each of the multiple products and appearance information of each of the multiple products.

18. The recording medium according to claim 17, wherein the identification means identifies products whose inventory numbers managed based on sales data do not match the inventory numbers identified in the inventory process, and obtains the product identification information of the identified products as the product identification information of the products to be analyzed.

19. A recording medium according to claim 17 or 18, wherein the identification means acquires product identification information input by a user as the product identification information of the product to be analyzed.

20. A recording medium according to any one of claims 15 to 19, wherein the predetermined action for the product being analyzed includes at least one of the following: an action of picking up an object in the target area; an action of picking up an object in the target area and taking it away; an action of looking at the target area; an action of locating in a partial area within the in-store image identified based on the target area; an action of staying in a partial area within the in-store image identified based on the target area for a predetermined period of time or more; and an action of assuming a predetermined posture in a partial area within the in-store image identified based on the target area.

Images