System, information processing method, and program

A video analysis system for shopping carts addresses high costs and inefficiencies by accurately monitoring cart storage areas, reducing false alarms and optimizing collection efforts through consistent threshold-based notifications.

JP7954331B1Active Publication Date: 2026-11-19SAFIE INC
View PDF 12 Cites 0 Cited by

Patent Information

Application Number
JP2026071784
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2026-04-23
Publication Date
2026-11-19
Estimated Expiration
2046-04-23

AI Technical Summary

Technical Problem

Existing shopping cart monitoring systems incur high costs and operational inefficiencies due to the need for communication devices on each cart, leading to frequent false detections and unnecessary notifications, especially in environments with mobile carts and passersby.

Method used

A video analysis system that uses cameras to detect shopping cart occupancy, estimates the presence of people, and calculates occupancy rates, suppressing false detections by considering environmental conditions and requiring consistent threshold violations before issuing notifications.

Benefits of technology

Reduces operational burdens by accurately monitoring cart storage areas, minimizing false alarms, and optimizing collection efforts through reliable and efficient cart management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007954331000001_ABST
    Figure 0007954331000001_ABST
Patent Text Reader

Abstract

The system accurately determines the occupancy status of objects within a designated area based on video footage captured by the camera. [Solution] A system that analyzes video footage captured by a camera sets an area in the video, detects objects in the video, calculates the percentage of the area occupied by objects, determines whether the percentage is above an upper threshold or below a lower threshold, and outputs according to the result of the determination.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present invention relates to a system, an information processing method, and a program.

BACKGROUND ART

[0002] Patent Document 1 describes a shopping cart system that uses a shopping cart equipped with a communication device, a base station, and a server to detect the number of carts in a cart storage area, and notifies a staff member or the like when the detected number exceeds a predetermined value.

PRIOR ART DOCUMENTS

PATENT DOCUMENTS

[0003]

PATENT DOCUMENT 1

SUMMARY OF THE INVENTION

PROBLEMS TO BE SOLVED BY THE INVENTION

[0004] However, attaching a communication device to an existing shopping cart or replacing it with a new shopping cart equipped with a communication device incurs various costs such as expenses and man-hours, which is not realistic especially for stores with a large number of shopping carts or enterprises operating multiple stores.

MEANS FOR SOLVING THE PROBLEMS

[0005] According to one aspect of the present disclosure, a system for analyzing video captured by a camera is provided. The system sets an area in the video, detects an object in the video, calculates an occupancy rate of the object in the area, determines whether the occupancy rate is equal to or higher than an upper limit threshold or equal to or lower than a lower limit threshold, and executes an output according to a result of the determination.

BRIEF DESCRIPTION OF THE DRAWINGS

[0006] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of a monitoring system. [Figure 2] Figure 2 shows an example of the hardware configuration of a server device. [Figure 3] Figure 3 shows an example of the hardware configuration of a client device. [Figure 4] Figure 4 shows an example of the hardware configuration of an imaging device. [Figure 5] Figure 5 shows an example of the functional configuration of a server device. [Figure 6] Figure 6 shows an example of a flowchart of the monitoring process performed by the server device. [Figure 7] Figure 7 shows an example of a settings screen displayed on a client device. [Figure 8] Figure 8 shows an example of a monitoring screen displayed on a client device. [Figure 9] Figure 9 shows an example of a progress display screen shown on the client device. [Modes for carrying out the invention]

[0007] Embodiments of the present invention will be described below with reference to the drawings. The various features shown in the embodiments below (including modified examples; the same applies hereinafter) can be combined with each other.

[0008] Furthermore, the more specific problems addressed by the present invention are as follows: In stores and other establishments, it is desirable to use images obtained from an imaging device to ascertain the occupancy status of the shopping cart area and to provide notifications at an appropriate time according to that occupancy status.

[0009] Furthermore, from the perspective of understanding the occupancy status with greater reliability, if the target object is obscured by a person appearing within the detection area, or if the target object and person are superimposed in the image, or if the brightness state of the image changes due to changes in illumination or time of day, the calculation of the occupancy rate becomes unstable, which may lead to frequent false detections or unnecessary notifications.

[0010] In particular, cart storage areas are often located near store entrances, along aisles, or at parking lot entrances, creating environments where passersby and employees frequently appear within the detection area. Therefore, if the occupancy status within the image is determined without considering the presence of people, the occupancy rate may change abruptly due to temporary obstruction or overlap by people, potentially leading to frequent false detections and unnecessary notifications that deviate from the actual number of carts. Such false detections can result in wasted effort in on-site collection operations and unnecessary patrols, increasing the burden on actual operations.

[0011] Furthermore, since carts are mobile objects that are taken out or returned by users, their occupancy rate can increase or decrease rapidly within the same detection area, and passersby are likely to be captured in images near entrances and exits. For this reason, a simple occupancy rate determination, similar to that used for monitoring the remaining quantity of fixed items, may result in false detections or unnecessary notifications at an operationally unacceptable frequency.

[0012] This embodiment also includes provisions to solve the problems described above.

[0013] <Embodiment 1> 0. Definition of Terms In this embodiment, "object" broadly refers to all inorganic matter, including, for example, shopping carts in stores, vehicles in parking lots, river water, and snow piled up outdoors. Furthermore, the term "living organism" broadly refers to all organic matter or similar entities, including, for example, humans, animals such as dogs and cats, and robots that possess artificial consciousness. Furthermore, "mobile object" refers to anything that can move autonomously or with human intervention, regardless of whether it is inorganic or organic, including, for example, carts, vehicles, people, animals, and robots.

[0014] 1. System Configuration Diagram Figure 1 is a diagram illustrating an example of the system configuration of a monitoring system 1000. As shown in Figure 1, the monitoring system 1000 includes a server apparatus 100, a client apparatus 110, and an imaging apparatus 120 as its system configuration. The server apparatus 100, the client apparatus 110, and the imaging apparatus 120 are communicably connected via a network 150. The network 150 includes any one of a WAN, a LAN, and the Internet, or any arbitrary combination thereof, and may include both a wired network and a wireless network.

[0015] The monitoring system 1000 is a system that monitors the state (full or insufficient) of a cart storage area based on an image obtained by imaging the cart storage area, and outputs a notification according to the determination result. The monitoring system 1000 may be configured as an example of a so-called cloud camera system. The monitoring system 1000 is an example of a system.

[0016] The server apparatus 100 is an apparatus that executes monitoring processing. For example, the server apparatus 100 calculates the occupancy rate of a target object in a detection area set in the image based on the image acquired from the imaging apparatus 120, determines whether the occupancy rate is equal to or higher than an upper limit threshold or equal to or lower than a lower limit threshold, and outputs a notification based on the determination result. The server apparatus 100 executes the main processing of the embodiments described later. The server apparatus 100 may store various types of setting information such as notification destinations, notification types, thresholds, and determination conditions (predetermined time interval, consecutive predetermined number of times).

[0017] The imaging apparatus 120 is an imaging apparatus (camera) installed to image the cart storage area. The imaging apparatus 120 is fixedly installed at a position that includes the cart storage area in its imaging range, for example, near the entrance / exit of a store, at the front of the cart storage area, or at an overhead position. The imaging apparatus 120 is not limited to a visible light camera, and may be an infrared camera or the like. In Figure 1, only one imaging device 120 is shown for simplification, but the monitoring system 1000 may include a plurality of imaging devices 120. For example, in a store with a plurality of cart storage areas, or a store with a plurality of store entrances and exits, a plurality of imaging devices 120 may be installed corresponding to each storage area or each entrance and exit.

[0018] The client device 110 is an example of an information terminal device used for checking monitoring results or making settings. The client device 110 may be a PC, a tablet terminal, a smartphone, or the like. The client device 110 may provide a user interface that performs, for example, detection area setting, upper threshold / lower threshold setting, notification enable / disable, setting of a predetermined time interval and a predetermined number of consecutive times, and display of monitoring results (such as a transition graph display of occupancy rate). In Figure 1, only one client device 110 is shown for simplification, but a plurality of client devices may be connected to the network 150.

[0019] The server device 100 may estimate a person region appearing in a detection area set in an image through image processing, exclude pixels corresponding to the person region from area counting for the target object, and then calculate the occupancy rate of the target object in the detection area. Accordingly, even when a passerby or an employee appears in the detection area, monitoring can be performed while suppressing destabilization of the occupancy rate and erroneous determination caused by the appearance of the person.

[0020] In the present embodiment, the target object is a shopping cart. Hereinafter, unless otherwise specified, the target object will be described as a shopping cart. Note that although the target object is described as a shopping cart in the present embodiment, the target object is not limited thereto. The target object may be, for example, a shopping basket, a handcart, a transport cage trolley, a pallet, a container, or a moving body or article that is collected and returned to a predetermined storage place in a store or facility.

[0021] Even if the target object is not a cart, the target object region estimation unit, as described later, can estimate the target object region by changing or selecting the estimation model or parameters used for segmentation or object detection according to the appearance, shape, and imaging conditions of the target object. For example, a trained model may be switched depending on the type of target object, or thresholds or preprocessing parameters may be switched for each type of target object.

[0022] Furthermore, the server device 100 may be configured to switch detection logic and models according to environmental conditions (day / night, lighting conditions). For example, in dark places at night, applying a low-light model and parameters can prevent detection failures of objects that appear dark.

[0023] Furthermore, the server device 100 may calculate the occupancy rate for multiple frames obtained at predetermined time intervals, and if the occupancy rate is above an upper threshold or below a lower threshold for a predetermined number of consecutive times corresponding to the predetermined time interval, a determination may be made, and a notification may be output when the determination is made. This can suppress unnecessary notifications caused by temporary reflections or noise, and improve the reliability of notifications.

[0024] In determining the occupancy rate, it is preferable to determine it only when the occupancy rate across multiple frames obtained at predetermined time intervals satisfies the threshold condition for a predetermined number of consecutive times, rather than determining it based on a single frame. This suppresses notifications caused by fluctuations in the occupancy rate due to momentary passage of people, imaging noise, or temporary occupancy, and allows notifications to be issued only for actual, continuous full or under-occupied states.

[0025] In this specification, "predetermined time interval" refers to the time interval at which frames (images) used for calculating and determining occupancy rates are acquired or extracted, and can be set in the range of, for example, a few seconds to a few minutes. More specifically, from the viewpoint of balancing responsiveness in store operations and suppression of false positives, it may be set to an interval of about one minute, but is not limited to this.

[0026] In this specification, "a predetermined number of consecutive occurrences" means that the occupancy rate obtained at predetermined time intervals satisfies the threshold condition (above or below the upper threshold or below the lower threshold) for multiple consecutive times over time. The predetermined number of occurrences is, for example, two or more, and as a specific example, three consecutive occurrences may be used as the condition for the determination to be valid. This makes it possible to suppress the determination being valid due to transient fluctuations in the occupancy rate caused by the instantaneous passage of a person or imaging noise, etc.

[0027] The monitoring system 1000 may have a setting to enable at least one of full-capacity detection (upper threshold) or under-capacity detection (lower threshold) for each detection area. Alternatively, each detection area or imaging device 120 may be assigned a receiving area where carts are picked up, or a return area where carts are placed, and under-capacity detection may be enabled for detection areas assigned to the receiving area, while full-capacity detection may be enabled for detection areas assigned to the return area.

[0028] In this specification, "main storage area" refers to the primary return location designated as the collection point for shopping carts within a store. On the other hand, "sub-storage area" refers to a location where carts temporarily returned or abandoned by users accumulate. Sub-storage areas tend to accumulate excessive amounts of carts compared to main storage areas, potentially obstructing aisles and compromising safety; therefore, they are subject to priority collection or removal.

[0029] For example, multiple detection areas to be monitored may be set up, such as a detection area corresponding to a sub-parking area (a place to drop off the carts) (first detection area) and a detection area corresponding to a main parking area (a place where carts are collected) (second detection area). The second detection area may be set up to correspond to the main parking area that customers are directed to return their carts to in the store.

[0030] Furthermore, the server device 100 may associate each imaging device 120 with a store and store the total number of shopping carts in each store (hereinafter also simply referred to as the total number of carts). The server device 100 may calculate a first aggregate value, which is an aggregate of the changes in the occupancy rate of the detection area group assigned to the receiving area, and a second aggregate value, which is an aggregate of the changes in the occupancy rate of the detection area group assigned to the return area, and display these as a trend graph on the client device 110. Furthermore, the server device 100 may calculate and display the trend of at least one of the sum of the first aggregate value and the second aggregate value, and the value obtained by subtracting the sum from the total number of carts.

[0031] The form of output is not limited and may include notification to the user. For example, it may include mobile push notification to client device 110, notification by email, transmission to a designated chat / messaging service, message transmission to a designated terminal, display on a display device, or notification by sound or light. Furthermore, the output is not limited to notifications; it may also be an output that displays a list of thumbnail images of the video or frame images at the time the judgment is made or at the time corresponding to that judgment. This allows the operator to easily check the corresponding actual video while understanding when the event occurred. Furthermore, the output may also be an output that clearly displays the point in time when the determination was made on a graph showing the trend of the occupancy rate. For example, the point in time may be distinguished from other points in time by a marker, icon, color coding, line type change, flag display, or annotation display. Furthermore, the output may be a CSV file that allows downloading of information corresponding to the point in time when a determination is made, along with related information. Related information may include, for example, time information, imaging device ID, detection area ID, detection area name, occupancy rate, determination result, threshold information, and reference information to a representative image or video. The operator may be able to select the notification type (e.g., email notification / mobile push notification) and set the notification address or notification device.

[0032] Here, the monitoring system described in the claims may consist of multiple devices or a single device. For example, when the monitoring system consists of a single device, one example is a server device 100 (or including an edge device). When the monitoring system consists of multiple devices, one example is a server device 100 and a client device 110, or a cloud server consisting of multiple server devices that provide the functions of server device 100. Furthermore, some or all of the processing, such as occupancy rate calculation and person area estimation, may be performed on the imaging device 120 side (edge ​​side).

[0033] 2. Hardware Configuration (1) Hardware configuration of server device 100 Figure 2 shows an example of the hardware configuration of server device 100. As shown in Figure 2, the server device 100 includes, as a hardware configuration, a control unit 210, a storage unit 220, a communication unit 230, and an internal bus 240. The control unit 210, the storage unit 220, and the communication unit 230 are electrically connected via the internal bus 240.

[0034] The control unit 210 is one of the following: CPU (Central Processing Unit), MPU (Micro Processing Unit), SoC (System-on-a-Chip), FPGA (Field-Programmable Gate Array), ASIC (Application Specific Integrated Circuit), GPU (Graphics Processing Unit), or at least two of these, and controls the entire server device 100 and performs various processes in response to images or image-based data input from the imaging device 120, for example. The control unit 210 performs processes such as calculating the occupancy rate of target objects in the detection area set in the image, estimating the area of ​​people captured in the detection area, excluding pixels corresponding to the area of ​​people from the area calculation, determining the threshold of the occupancy rate, generating and sending notifications based on the determination results, and assigning receiving areas / returning areas, calculating aggregate values ​​per store, and generating data for displaying trend graphs.

[0035] The storage unit 220 is one of the following: HDD (Hard Disk Drive), ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drive), or a combination of at least two of these, and stores the program and the data used by the control unit 210 when it executes processing based on the program.

[0036] Examples of data used by the control unit 210 when executing processing include images (or a part thereof, or feature quantities) acquired from the imaging device 120, detection area setting information (coordinates, shape, area ID, etc.), models or parameters used for identifying target objects (e.g., estimation models for segmentation or object detection), models or parameters used for estimating human areas, settings used for calculating occupancy rates (area calculation method, exclusion conditions, etc.), upper and lower thresholds, predetermined time intervals and determination conditions for a predetermined number of consecutive detections, settings for enabling full / under-occupancy detection for each detection area, assignment information for receiving / returning areas for each detection area or camera, correspondence information between cameras and stores, total number of carts per store, notification destination information (destination terminal, notification method, etc.), historical data (time-series data) of occupancy rates and aggregated values, and graph data to be displayed on the user interface.

[0037] The storage unit 220 is an example of a non-temporary recording medium (storage medium) that is readable by a computer on which a program is recorded. In this specification, the data used by the control unit 210 when executing processing based on the program is described as being stored in the storage unit 220, but it may also be stored in the storage unit of another device that can communicate with the server device 100. In other words, the data may be stored in the storage unit of any device as long as the control unit 210 can refer to and / or retrieve it.

[0038] The control unit 210 executes processing based on the program stored in the storage unit 220, thereby realizing the functional configuration of the server device 100 (for example, the person area estimation unit, exclusion processing unit, occupancy rate calculation unit, threshold determination unit, notification control unit, setting management unit, aggregate value calculation unit, display data generation unit, etc.) and the processing shown in various flowcharts, which will be described later.

[0039] The communication unit 230 is a communication interface that connects the server device 100 to the network 150 and mediates communication with other devices. The communication unit 230 can be implemented using hardware that enables wired communication (e.g., Ethernet-compliant network interface card (NIC), LAN (Local Area Network) adapter, optical transceiver, etc.) and / or wireless communication (e.g., wireless communication modules such as Wi-Fi (Wireless Fidelity), Bluetooth (registered trademark), 5G (Fifth Generation), LTE (Long Term Evolution), NFC (Near Field Communication)), as well as circuits that control them.

[0040] Based on instructions from the control unit 210, the communication unit 230 performs data transmission and reception processing over the network 150 according to a predetermined communication protocol (e.g., TCP (Transmission Control Protocol) / IP (Internet Protocol), UDP (User Datagram Protocol), HTTP (Hypertext Transfer Protocol), MQTT (Message Queuing Telemetry Transport), etc.). For example, the communication unit 230 receives an image or image-related data from the imaging device 120 and transmits monitoring results, notifications, or display data (occupancy rate, aggregated values, trend graph data, etc.) to the client device 110.

[0041] Furthermore, the hardware configurations of the control unit 210, storage unit 220, and communication unit 230 are not limited to one. For example, multiple control units may be included in the server device 100. Furthermore, some or all of the processing performed by the server device 100 may be distributed and executed by a cloud server composed of multiple server devices. In addition, some processing, such as estimation of the human area, estimation of the target object area, and calculation of the occupancy rate, may be performed on the imaging device 120 side or on the edge device side within the store, and the server device 100 may be configured to receive the results of such processing, make a determination, and notify.

[0042] (2) Hardware configuration of client device 110 Figure 3 shows an example of the hardware configuration of the client device 110. As shown in Figure 3, the client device 110 includes, as a hardware configuration, a control unit 310, a storage unit 320, an input unit 330, an output unit 340, a communication unit 350, and an internal bus 360. The control unit 310, the storage unit 320, the input unit 330, the output unit 340, and the communication unit 350 are electrically connected via the internal bus 360.

[0043] The control unit 310 is a CPU, MPU, SoC, FPGA, ASIC, GPU, or a combination of at least two of these, and controls the entire client device 110 and performs various processes in response to, for example, operator input or data received from the server device 100. The control unit 310 performs, for example, display control based on monitoring results (occupancy rate, judgment results, notification content, etc.) and display data (data for trend graphs, etc.), as well as generating and transmitting various setting information such as detection area settings, upper / lower thresholds, full / insufficient detection activation, predetermined time intervals and predetermined number of consecutive times, pick-up / return area assignment, store settings, and total number of carts.

[0044] The storage unit 320 stores programs and data used by the control unit 310 when it executes processing based on the programs, using one of the following: HDD, ROM, RAM, SSD, etc., or any combination thereof. Examples of stored data include connection information to the server device 100, operator account information, selection information for the store or camera to be monitored, setting information such as detection area settings and various thresholds (including data in the process of being entered), history of received monitoring results and notifications, and display data used for displaying trend graphs, etc.

[0045] The input unit 330 is a device that inputs information to the client device 110 in response to the operator's operation. The input unit 330 receives operation input made by the operator. The operation input is transferred to the control unit 310 via the internal bus 360 as a command signal. The control unit 310 can, if necessary, perform predetermined control and / or calculations based on the transferred command signal. The input unit 330 may be included in the housing of the client device 110 or it may be external. For example, the input unit 330 may be implemented as a touch panel display integrated with the output unit 340. When the input unit 330 is implemented as a touch panel display, the operator can input tap operations, swipe operations, etc. to the input unit 330. Instead of a touch panel display, the input unit 330 can be a switch button, mouse, trackpad, keyboard, etc. The operator may input, for example, the detection area (specifying a region on the image), enable full / under-fill detection, assign receiving / returning areas, set thresholds and judgment conditions, or specify the display period via the input unit 330.

[0046] The output unit 340 is a display unit, such as a display, that outputs (displays) information as a graphical user interface (GUI) screen that can be operated by the operator. The output unit 340 may be included in the housing of the client device 110 or it may be external. More specifically, the output unit 340 may be implemented as a display device such as a liquid crystal display, an organic EL (Electro-Luminescence) display, or a plasma display. The output unit 340 may display, for example, the occupancy rate for each detection area, the judgment result (full / insufficient), the notification content, a first aggregate value for the detection area group assigned to the receiving area, a second aggregate value for the detection area group assigned to the return area, and graphs of their trends.

[0047] The communication unit 350 is a communication interface that connects the client device 110 to the network 150 and mediates communication with other devices. The communication unit 350 can be implemented, for example, by hardware that realizes wired communication (e.g., Ethernet-compliant NIC, LAN adapter, optical transceiver, etc.) and / or wireless communication (e.g., wireless communication modules such as Wi-Fi, Bluetooth®, 5G, LTE, NFC, etc.), as well as circuits that control them.

[0048] The communication unit 350, based on instructions from the control unit 310, performs data transmission and reception processing via the network 150 in accordance with a predetermined communication protocol (e.g., TCP / IP, HTTP, MQTT, etc.). For example, the communication unit 350 receives monitoring results, notifications, or display data from the server device 100, and also transmits various setting information based on operator input to the server device 100.

[0049] In this specification, a personal computer (computer) is used as an example of the client device 110. However, the client device 110 may be a notebook PC, smartphone, tablet computer, etc. The client device 110 can be any device that can display a screen as described later and accept operations from an operator via the screen, etc.

[0050] (3) Hardware configuration of the imaging device 120 Figure 4 shows an example of the hardware configuration of the imaging device 120. As shown in Figure 4, the imaging device 120 includes, as a hardware configuration, a control unit 410, a storage unit 420, an imaging unit 430, a communication unit 440, and an internal bus 450. The control unit 410, the storage unit 420, the imaging unit 430, and the communication unit 440 are electrically connected via the internal bus 450.

[0051] The control unit 410 is a CPU, MPU, SoC, FPGA, ASIC, GPU, or a combination of at least two of these, and controls the entire imaging device 120 and performs processing according to, for example, images acquired by the imaging unit 430 or control information input from the server device 100. The control unit 410 performs processing such as controlling imaging conditions (exposure, frame rate, resolution, etc.), encoding or compressing images, adding time information to captured images, and generating images or image-related data to be transmitted to the server device 100 as needed.

[0052] The storage unit 420 is one of the following: HDD, ROM, RAM, SSD, or at least two of these, and stores the program and data used by the control unit 410 when it executes processing based on the program. Examples of data used by the control unit 410 when it executes processing include setting values ​​for imaging conditions (exposure, frame rate, resolution, etc.), connection information to the network 150, destination information to the server device 100, temporary storage data of the captured image (or a part thereof, or encoded data), time information, and operation logs.

[0053] The storage unit 420 is an example of a non-temporary recording medium (storage medium) that is readable by a computer on which a program is recorded. In this specification, the data used by the control unit 410 when executing processing based on the program is described as being stored in the storage unit 420, but it may also be stored in the storage unit of another device that can communicate with the imaging device 120. In other words, the data may be stored in the storage unit of any device as long as the control unit 410 can refer to and / or retrieve it.

[0054] The imaging unit 430 is an imaging unit that includes an image sensor and an optical system for imaging subjects included in the imaging range, including the cart storage area. The imaging unit 430 may include, for example, an image sensor such as a CMOS or CCD, a lens, and infrared illumination as needed. The imaging device 120 may be a fixed camera, for example, installed in a position that allows imaging of the cart storage area from above or from the front. The imaging unit 430 may also be configured to acquire depth information. For example, the imaging unit 430 may be equipped with a stereo camera, a ToF sensor, or a LiDAR to acquire depth information indicating the distance to the subject along with the image. This can improve the robustness of the estimation of the human area or the target object area.

[0055] The communication unit 440 is a communication interface that connects the imaging device 120 to the network 150 and mediates communication with other devices. The communication unit 440 can be implemented, for example, by hardware that realizes wired communication (e.g., Ethernet-compliant NIC, LAN adapter, optical transceiver, etc.) and / or wireless communication (e.g., wireless communication modules such as Wi-Fi, Bluetooth®, 5G, LTE, NFC, etc.), and circuits that control them. The communication unit 440, based on instructions from the control unit 410, performs data transmission and reception processing over the network 150 in accordance with a predetermined communication protocol (e.g., TCP / IP, HTTP, MQTT, etc.). For example, the communication unit 440 may transmit images or image-related data acquired by the imaging unit 430 to the server device 100, and may also receive setting information or control information from the server device 100.

[0056] The imaging device 120 may be configured to include an edge processing function, perform some processing such as estimating the human area or the target object area, and transmit the results (mask information, area information, occupancy rate, etc.) to the server device 100.

[0057] 3. Functional Configuration Figure 5 shows an example of the functional configuration of the server device 100. As shown in Figure 5, the server device 100 includes, as a functional configuration, an image acquisition unit 510, a setting management unit 520, a person area estimation unit 530, a target object area estimation unit 540, an exclusion processing unit 550, an occupancy rate calculation unit 560, a determination unit 570, a notification control unit 580, a storage unit 595, and an aggregation / display data generation unit 590.

[0058] (Image acquisition unit 510) The image acquisition unit 510 acquires images (frames) or image-related data of the cart storage area from the imaging device (camera) 120. The image acquisition unit 510 may acquire images from multiple imaging devices 120. The image acquisition unit 510 may also add or associate metadata such as time information, imaging device ID, and store ID to the acquired images and store them in the storage unit 595.

[0059] (Settings management section 520) The setting management unit 520 manages various setting information used for monitoring. Examples of setting information include the setting of detection areas within the image (position, shape, area ID, etc.), the setting to enable at least one of upper limit detection and lower limit detection for each detection area, upper limit threshold and lower limit threshold, predetermined time interval, determination conditions for a predetermined number of consecutive occurrences, and allocation information for receiving area / return area for each detection area or imaging device. Here, upper limit detection detects when the occupancy rate is above the upper limit threshold, and may correspond to the detection of a full state, for example. Lower limit detection detects when the occupancy rate is below the lower limit threshold, and may correspond to the detection of a shortage state, for example. The setting management unit 520 may receive setting changes input from the client device 110 and update the settings stored in the storage unit 595. Furthermore, the settings management unit 520 may manage information that associates each imaging device with a store, as well as the total number of shopping carts for each store. Furthermore, the setting management unit 520 may manage setting information (such as an exclusion processing enable flag) indicating whether or not to enable exclusion processing to reduce the impact of human areas for each detection area or imaging device.

[0060] (Human area estimation unit 530) The person area estimation unit 530 estimates the person area within the detection area based on the image acquired by the image acquisition unit 510. The person area estimation unit 530 may estimate the person area by means of, for example, person detection, segmentation, etc. The person area estimation result may be output as a person area mask, contour information of the person area, or a set of pixels corresponding to the person area.

[0061] (Target object region estimation unit 540) The target object region estimation unit 540 estimates the target object region corresponding to the target object based on the image acquired by the image acquisition unit 510. The estimation may use a model switched according to the image brightness and time of day (for example, a nighttime model). For example, brightness information may be calculated based on the image's average brightness, brightness histogram, exposure value, or gain value. Time information may indicate the imaging time, the period before and after sunset, or business hours. Based on this information, the target object region estimation unit 540 may switch the applicable estimation model or the threshold or preprocessing parameters used for the estimation. The target object may be, for example, a shopping cart. The target object region estimation may be performed by segmentation or object detection. The target object region estimation result may be output as a target object region mask, a bounding box of the target object, or a set of pixels corresponding to the target object region. The target object area estimation unit 540 may be omitted. In this case, the occupancy rate calculation unit 560 may determine which pixels to be counted as the target object by, for example, for pixels within the detection area, by foreground extraction based on the difference with the background model, threshold processing based on a predetermined range of color or brightness, or extraction based on the edge amount or texture amount, and calculate the occupancy rate based on the number of such pixels.

[0062] (Exclusion processing unit 550) The exclusion processing unit 550 performs a process to exclude pixels corresponding to the human area from the area calculation of the target object, based on the estimation result of the human area estimation unit 530. For example, the exclusion processing unit 550 may mask the pixels corresponding to the human area and exclude them from the area calculation. Furthermore, the exclusion process is not limited to disabling pixels; it may also involve subtracting the total area (denominator) of the detection area according to the area of ​​the person's region, or multiplying it by a correction coefficient according to the degree of overlap of the people. Furthermore, if the area of ​​a person overlaps with the area of ​​a target object, the exclusion processing unit 550 may be configured to exclude pixels corresponding to the portion of the area of ​​the person that overlaps with the target object from the area calculation.

[0063] (Occupancy rate calculation unit 560) The occupancy rate calculation unit 560 calculates the occupancy rate of the target object in the detection area based on the acquired image, and if exclusion processing is performed, it may calculate the occupancy rate based on the information after the exclusion processing (for example, an image that reflects the result of excluding pixels corresponding to the human area from the area calculation as a target object, or an exclusion mask that shows the exclusion result). The occupancy rate calculation unit 560 may store the calculated occupancy rate in the storage unit 595 in association with time information, detection area ID, imaging device ID, store ID, etc. Here, "occupancy rate" as used herein may be defined as the number of pixels that are counted as the area of ​​the target object within the detection area divided by the total number of pixels within the detection area, or it may be defined as the number of effective pixels after excluding pixels corresponding to the human area (the total number of pixels within the detection area minus the number of pixels corresponding to the human area). The number of pixels whose area is calculated as the target object may be the number of pixels belonging to the target object region estimated by the target object region estimation unit 540.

[0064] Furthermore, "occupancy rate" is a ratio index defined with respect to the detection area, and is calculated using at least (i) a numerator representing the amount that is counted as the area of ​​the target object within the detection area, and (ii) a denominator representing the standard amount of the detection area. The numerator may be, for example, the number of pixels that are counted as the area of ​​the target object, or the number of pixels belonging to the target object area estimated by the target object area estimation unit 540. The denominator may be, for example, the total number of pixels within the detection area, or the number of effective pixels after excluding pixels corresponding to the human area.

[0065] Furthermore, the index representing the occupancy status may be defined as a ratio based on count values ​​such as the number of units, and for convenience, this index may be called the "occupancy rate." For example, the occupancy rate may be the value obtained by normalizing the number of detections obtained by the target object area estimation unit 540 through object detection by the maximum capacity set for each detection area.

[0066] In this specification, the term "sufficiency rate" may be used in the user interface. This "sufficiency rate" is an indicator representing the occupancy status of target objects within the detection area and is synonymous with the "occupancy rate" described above. For example, in application screens, the term "sufficiency rate" may be used for ease of understanding by the user.

[0067] Furthermore, in this specification, "status" or "status information" means information including at least (i) the occupancy rate (sufficiency rate) associated with the detection area, (ii) the determination result based on the occupancy rate (type such as full / insufficient / normal), and (iii) time information corresponding to the calculation or determination. It may also include the imaging device ID, detection area ID, detection area name, and representative image (frame image at the time of determination).

[0068] Specific examples of how the occupancy rate is calculated (for example, using the total number of pixels A in the detection area, the number of pixels H corresponding to the human area, and the number of pixels O that are counted as the area of ​​the target object) will be described later.

[0069] In this specification, “occupancy rate” is defined as an index representing the occupancy rate of a target object within a detection area, wherein the index increases as the amount of target objects within the detection area increases. The occupancy rate may be calculated, for example, based on (i) the area ratio of the target object area to the detection area, (ii) the number of pixels determined to be target objects within the detection area, (iii) the number of target objects detected, or (iv) a combination thereof.

[0070] Furthermore, the "target object region" refers to the region in the acquired image that is estimated to correspond to the target object, and may be represented as a mask obtained by segmentation, a bounding box obtained by object detection, or a set of pixels.

[0071] "Determination" is a process that determines whether the state of the target object in the detection area is in a predetermined state (e.g., full or insufficient) by comparing the occupancy rate with an upper threshold and / or a lower threshold. The determination may be based on the occupancy rate of a single frame, or it may be based on the occupancy rates of multiple frames obtained at predetermined time intervals, and the determination may be established when a predetermined number of consecutive occupancy rates are met.

[0072] "Notification" refers to an example of output that presents the judgment result to the user, and may be, for example, a push notification to a mobile device, email, or display on a user interface (including status display and history display). The notification content may include, for example, identification information of the imaging device or detection area, whether it is full or insufficient, and recommended actions (collection, movement to collection point, etc.). Furthermore, the term "output" in this specification is not limited to the above-mentioned notification, and may also include displaying a thumbnail list of video or frame images corresponding to the time when the determination is made, displaying the identification of that time in the occupancy rate trend graph, or making CSV data containing related information corresponding to that time available for download.

[0073] Here, the upper threshold is a threshold for detecting when the occupancy rate exceeds a predetermined value, for example, a threshold for detecting a full state where there is an excess of the target object. The lower threshold is a threshold for detecting when the occupancy rate falls below a predetermined value, for example, a threshold for detecting a shortage state where there is a shortage of the target object. In this specification, "upper limit detection" means detecting a predetermined state based on the occupancy rate being equal to or greater than an upper limit threshold. "Lower limit detection" means detecting a predetermined state based on the occupancy rate being equal to or less than a lower limit threshold. Upper limit detection may be performed, for example, as full capacity detection, and lower limit detection may be performed, for example, as insufficient capacity detection. The determination unit 570 may make a determination by applying at least one of upper limit detection and lower limit detection according to the activation setting for each detection area.

[0074] For example, in a detection area where only full-occupancy detection is effective, it is possible to determine whether the occupancy rate is above or below an upper threshold, and if this condition is met, it is determined to be full. On the other hand, in a detection area where only under-occupancy detection is effective, it is possible to determine whether the occupancy rate is below or below a lower threshold, and if this condition is met, it is determined to be under-occupied.

[0075] Furthermore, if both full-capacity detection and under-capacity detection are effective, the determination unit 570 may determine whether the occupancy rate is above the upper threshold and below the lower threshold, respectively, and output a notification according to the type that is met. In addition, to suppress chattering of determinations near the thresholds, hysteresis may be provided by making the threshold for meeting a condition and the threshold for clearing a condition different.

[0076] (Judgment unit 570) The determination unit 570 determines whether the occupancy rate is above an upper threshold or below a lower threshold, based on the occupancy rate calculated by the occupancy rate calculation unit 560. The determination unit 570 may selectively apply at least one of upper limit detection or lower limit detection depending on the activation settings for each detection area managed by the setting management unit 520. Upper limit detection may be used, for example, to detect when the occupancy rate of the target object is high, and lower limit detection may be used, for example, to detect when the occupancy rate of the target object is low. Furthermore, the determination unit 570 may be configured to use the occupancy rate for multiple frames obtained at predetermined time intervals, and to establish a determination when the occupancy rate satisfies the threshold condition a predetermined number of consecutive times.

[0077] (Notification control unit 580) The notification control unit 580 outputs a notification based on the determination result of the determination unit 570. The notification control unit 580 may also control the notification destination (e.g., client device 110, management terminal, or predetermined messaging destination), notification content, and notification method (push notification, screen display, message transmission, etc.) according to the settings of the setting management unit 520. Furthermore, if the determination state (full or insufficient) is not resolved even after a predetermined time (e.g., 10 minutes) has elapsed since outputting the notification, the notification control unit 580 may output the notification again (reminder notification). Furthermore, if multiple detection areas are set, and full capacity is detected in the first detection area, which corresponds to a drop-off location, the system may be configured to output a notification prompting the user to move to a second detection area, which is a different detection area from the first detection area and corresponds to a destination for consolidating carts.

[0078] (Data aggregation and display generation unit 590) The aggregation and display data generation unit 590 calculates a first aggregate value, which aggregates the changes in occupancy rates in the detection area groups assigned to the receiving area, and a second aggregate value, which aggregates the changes in occupancy rates in the detection area groups assigned to the return area, and generates display data to be used for graph display on the user interface. Furthermore, the aggregation and display data generation unit 590 may estimate the number of uncollected carts that are being used by customers or have not been returned to the designated location within the store by subtracting the sum of the first aggregate value (total for receiving area) and the second aggregate value (total for return area) from the total number of carts set for each store, and generate the changes as display data. This allows the manager to grasp the operating status of carts throughout the store and the risk of shortages. In addition, the aggregation and display data generation unit 590 may calculate a first aggregate value, which aggregates the occupancy rates in the area groups where upper limit detection is enabled, and a second aggregate value, which aggregates the occupancy rates in the area groups where lower limit detection is enabled, and display them on the user interface.

[0079] The aggregation and display data generation unit 590 is not limited to classification by receiving area / return area, but may also group and aggregate detection area groups based on attributes assigned to the detection area. For example, it may calculate a first aggregate value which aggregates the changes in occupancy rates in detection area groups assigned a first attribute, and a second aggregate value which aggregates the changes in occupancy rates in detection area groups assigned a second attribute, and display them as graphs on the user interface.

[0080] Furthermore, the aggregation and display data generation unit 590 may calculate and display the trend of at least one of the following: the sum of the first aggregated value and the second aggregated value, and the value obtained by subtracting the sum of the first aggregated value from the total number of carts stored for each store. This makes it possible to understand the relationship between the amount of carts in the storage area, which is understood as the first attribute group and the second attribute group, and the amount of carts outside the storage area (in use, in transit, etc.) in a time series.

[0081] (Storage unit 595) The storage unit 595 stores various information used by the server device 100 when it performs processing. For example, the storage unit 595 may store setting information managed by the setting management unit 520, images (or a part thereof) acquired by the image acquisition unit 510, person area estimation results, target object area estimation results, calculated occupancy rates, judgment results, notification history, and aggregated values ​​and display data generated by the aggregation / display data generation unit 590. Note that the storage unit 220 shown in Figure 2 represents a storage resource as part of the hardware configuration of the server device 100, while the storage unit 595 shown in Figure 5 represents a storage function as part of the functional configuration of the server device 100. The storage unit 595 may be implemented, for example, by the storage unit 220 (or an equivalent storage resource). Furthermore, the storage unit 595 may be configured to include an external storage device or cloud storage that can communicate with the server device 100.

[0082] 4. Information Processing The information processing of Embodiment 1 will be described below.

[0083] (1) Overview of the process The monitoring system 1000 is a system that monitors the cart storage area based on images captured by the imaging device 120. The image acquisition unit 510 of the server device 100 acquires images from the imaging device 120. The occupancy rate calculation unit 560 calculates the occupancy rate of target objects in the detection area set in the image. The determination unit 570 determines whether the occupancy rate is above an upper threshold or below a lower threshold. The notification control unit 580 outputs a notification based on the determination result. In one embodiment, the person area estimation unit 530 may estimate the person area corresponding to a person appearing within the detection area. The exclusion processing unit 550 may perform exclusion processing to reduce the influence of the person area; for example, pixels corresponding to the person area may be excluded from the area calculation as a target object. The occupancy rate calculation unit 560 may determine the occupancy rate based on the information after the exclusion processing.

[0084] By performing this process, the system can determine the occupancy status (full or insufficient) of the target object in the detection area based on the image obtained from the imaging device 120, and output a notification corresponding to that occupancy status. Furthermore, in a configuration that performs person area estimation and exclusion processing as in one embodiment, even in situations where a person is captured within the detection area or where the target object is obscured or superimposed by a person during imaging, it is possible to suppress the incorrect area calculation of the target object or unstable fluctuations in occupancy rates due to the influence of the person. As a result, the accuracy of full / insufficient determination based on occupancy rate threshold judgment is improved, reducing the occurrence of unnecessary notifications (false notifications), and enabling more appropriate output of notifications according to the status of the cart storage area.

[0085] (2) Details of the process (explanation using flowcharts and UI) Figure 6 shows an example of a flowchart of the monitoring process performed by the server device 100. The processing procedure will be described below with reference to Figure 6. Note that the following processing may be repeated for each detection area if multiple detection areas are set. In this embodiment, various settings and monitoring results may be confirmed via a UI screen (for example, Figures 7 to 9) displayed on the client device 110. The transitions for Yes / No in each judgment step are shown in Figure 6. For example, if no people are included, the process proceeds to occupancy rate calculation. If the threshold condition or consecutive count condition is not met, the process proceeds to aggregation and display data generation without issuing a notification.

[0086] (Step S600) First, the operator selects the store or imaging device 120 to be monitored using the setting screen 700 shown in Figure 7, which is displayed on the client device 110, and sets the detection area on the image obtained from the imaging device 120. For example, the operator may register the detection area by specifying a rectangle or polygon on the image and assign a region name or region ID to each detection area.

[0087] The settings screen 700 may also include a list area 710 for displaying a list of configured detection areas, an operator for creating a new detection area (e.g., a "Create New" button 720), and an operator for deleting or editing a detection area (e.g., a "Delete" button 730). When the operator selects a detection area in the list area 710, that detection area is highlighted on the image, and the operator may be able to change the position or shape of the detection area by dragging on the image. If the detection area is set as a polygon, the operator may register the polygon based on an operation to sequentially specify vertices on the image, or change its shape by moving or adding vertices.

[0088] Furthermore, in the settings screen 700, at least one of upper limit detection and lower limit detection can be enabled for each detection area. For both upper limit detection and lower limit detection, the threshold and the judgment condition "detection is determined when X consecutive occurrences occur at X minute intervals (the time can be seconds, etc., instead of minutes)" can be set. Here, upper limit detection may correspond to, for example, the detection of a full state, and lower limit detection may correspond to, for example, the detection of a low state. Furthermore, the operator may assign either a receiving area or a return area (or both) as an attribute for each detection area or imaging device 120, and enable lower limit detection for detection areas assigned to the receiving area and upper limit detection for detection areas assigned to the return area. In this embodiment, the receiving area is an example of a detection area to which a first attribute, which indicates a tendency for the number of target objects to decrease, is assigned, and the return area is an example of a detection area to which a second attribute, which indicates a tendency for the number of target objects to increase, is assigned. Furthermore, in the settings screen 700, a display color may be set for each detection area, and the display may be shown in correspondence with that display color on the monitoring screen 800 or the trend display screen 900. Additionally, you can set the type of cart as an attribute for each area. For example, for supermarket carts, if you're looking for a basket-type cart (for general use), there are options such as: (1) Single-tier cart: Only the top tier holds a basket. It's easy to maneuver and is the most common type for everyday shopping. (2) Double-tier cart: Baskets can be placed on both the top and bottom tiers. This is suitable for bulk purchases or for separating frozen foods and heavy items on the bottom tier. (3) Compact and lightweight cart: Suitable for stores with narrow aisles or small stores. And so on. For functional and purpose-specific carts, there are options such as: (4) Child-carrying cart: Equipped with a seat so children can sit safely while shopping. (5) Wheelchair-accessible shopping cart: Can be attached to or mounted on the front of a wheelchair. (6) Large and multi-functional cart: Types like the Royal Cart that can hold three baskets, or sturdy types for carrying heavy items. And so on. Additionally, you may set area-specific attributes such as location (store / floor / indoor / outdoor / parking lot, etc.). The aforementioned area-specific attributes (receive / return / cart type / area location, etc.) may also be set collectively for each camera. In that case, the settings made for a camera will be reflected as the settings for each area corresponding to that camera. Furthermore, if various settings are made for multiple cameras / multiple areas, the status of each area may be aggregated and statistics such as graphs may be output, or used to determine whether the area is full or understaffed, as described above. If multiple attributes are set for each area as described above, one or more of these attributes may be selected and cross-tabulated before outputting the results.

[0089] When setting the detection area, it is preferable to refer to an image or video of the target cart storage area in its "maximum capacity state" (for example, the upper limit beyond which it becomes dangerous or requires retrieval in operation) and set the detection area to include the area occupied by the target object in that maximum capacity state. This makes it easier for the occupancy rate (fullness rate) within the detection area to intuitively reflect the "degree of fullness" in operation, and makes it easier to adjust the threshold settings (upper threshold / lower threshold).

[0090] For example, the operator may determine the shape of the detection area (e.g., a polygon) based on video footage of the maximum occupancy state, and then set an upper threshold for full-occupancy detection or a lower threshold for under-occupancy detection (e.g., upper threshold 0.4-0.6, lower threshold 0.1-0.2). These values ​​are examples and can be changed as appropriate depending on the store layout, storage capacity, traffic volume, etc.

[0091] In this specification, "First Attribute" and "Second Attribute" refer to attributes assigned to select a detection type (full detection / insufficient detection) according to the increasing or decreasing trend of target objects in the detection area. The First Attribute is an attribute that may be assigned to a detection area where the number of target objects tends to decrease over time, for example, it may be assigned to a pick-up area where the occupancy rate tends to decrease as users take out carts. The Second Attribute is an attribute that may be assigned to a detection area where the number of target objects tends to increase over time, for example, it may be assigned to a return area where the occupancy rate tends to increase as users return carts.

[0092] Once the operator confirms the settings, the settings are sent to the server device 100 and stored and managed by the settings management unit 520. In this way, the system accepts the settings desired by the operator and stores them internally, which means that the settings are performed by the system. Note that the processing in step S600 may be performed prior to the monitoring process or as appropriate during monitoring.

[0093] (Step S601) Next, the system waits until a predetermined time has elapsed. Here, the predetermined time is the value of the judgment condition "X-minute interval" set in step S600 (Figure 7).

[0094] (Step S610) Next, the image acquisition unit 510 acquires the latest frame image from the video captured by the imaging device 120 and associates the imaging device ID and time information with the image. Here, a frame image is a part of the video, and the video is composed of multiple frame images. Steps S630 and S640 below are executed if it is determined in step S620 that exclusion processing should be performed. If it is determined in step S620 that exclusion processing should not be performed, steps S630 and S640 may be omitted, and in this case, step S650 may be executed after step S610.

[0095] (Step S611) Next, the control unit 210 (for example, the person area estimation unit 530 and the target object area estimation unit 540) analyzes the video (acquired frame images) and detects objects and people. Here, "objects" mainly refer to carts, but may also include other moving objects such as vehicles. Similarly, "people" mainly refer to humans, but may also include other living beings such as animals. These detections can be performed using AI or the like. Once objects and people are detected, bounding boxes or the like may be assigned to each.

[0096] (Step S612) Next, the control unit 210 determines whether or not a person is included in the area. This process determines that a person is included in the detection area if the bounding box of a person is included in the set detection area at or above a predetermined threshold (any value greater than or equal to 0 / which can be arbitrarily set in advance by the operator). Note that the operator may arbitrarily set whether or not to perform this determination (whether or not to perform the determination at all) in advance. If this determination is OFF, step S612 can be skipped. If it is determined in step S612 that a person is included, the process returns to step S601; otherwise, the process proceeds to step S620.

[0097] (Step S620) The control unit 210 (for example, the setting management unit 520) determines whether or not to perform exclusion processing (processing to reduce the impact of human areas) on the detection area based on the setting information managed by the setting management unit 520 in the storage unit 595. For example, if the setting information indicating whether the exclusion processing is enabled or disabled (exclusion processing enabled flag) stored in association with the imaging device ID and the detection area ID is enabled, it is determined to perform exclusion processing (Yes) and the system proceeds to step S630. If the setting information is disabled, it is determined not to perform exclusion processing (No) and the system proceeds to step S650. If the human presence determination in step S612 is set to OFF, steps S620 to S640 can be skipped regardless of the settings related to exclusion processing. Furthermore, the decision of whether or not to perform exclusion processing may be made based on the exclusion processing enabled flag, as well as the detection type (full detection by upper threshold / underutilization detection by lower threshold) enabled for each detection area. For example, in a detection area where shortage detection is effective, the presence or obstruction of people not holding carts can temporarily decrease the occupancy rate and easily lead to a false determination of shortage. Therefore, the exclusion process may be enabled (steps S630 and S640 are executed). On the other hand, in a detection area where only full-capacity detection is effective, even if the occupancy rate decreases due to the presence of people, a false determination of full-capacity is less likely to occur. Therefore, the exclusion process may be disabled (transition to step S650).

[0098] The determination in step S620 may be made based on imaging condition information associated with the acquired image, either in lieu of or in addition to the setting information. The imaging condition information may be, for example, the time of acquisition, the brightness information of the image, or time zone classifications based on these. For example, it may be determined that exclusion processing should be performed during time zones when congestion is expected and people are likely to be captured in the image, or during time zones when the average brightness of the image is below a predetermined threshold, and that exclusion processing should not be performed during other time zones.

[0099] The determination in step S620 may be made based on a simple estimation of the possibility of a person being captured in the image. For example, the image acquisition unit 510 or the preprocessing unit (not shown) may reduce the acquired image to generate a low-resolution image or a reduced image of the detection area. For example, the person area estimation unit 530 may perform a simple person detection on the reduced-resolution image or the reduced image of the detection area to estimate the presence or absence of a person, and the control unit 210 may determine to perform exclusion processing if it is estimated that a person is present. Alternatively, the system may refer to the estimation result of the presence or absence of a person in the previous frame and determine to perform exclusion processing if it is estimated that a person was present in the previous frame.

[0100] (Step S630) The person region estimation unit 530 estimates the person region captured within the detection area based on the acquired image. The estimation of the person region may be performed by either person detection or person segmentation.

[0101] (Step S640) The exclusion processing unit 550 excludes pixels corresponding to the person region obtained in step S630 from the area calculation of the target object. For example, the exclusion processing unit 550 invalidates the pixels using a person region mask. Furthermore, if the human area overlaps with the target object area within the detection area, the exclusion processing unit 550 may exclude the pixels corresponding to the overlapping portion of the human area with the target object area from the area calculation. The target object area may be the one estimated in step S650, or it may be the one estimated prior to step S650 (for example, the estimation result of the previous frame). The exclusion processing unit 550 may determine whether or not there is an overlap between the human area and the target object, for example, based on the estimation result of the target object area estimation unit 540 (target object area mask or bounding box) and the human area estimation result.

[0102] (Step S650) The occupancy rate calculation unit 560 calculates the occupancy rate based on the acquired image if it is determined in step S620 not to perform exclusion processing, and calculates the occupancy rate based on the information after the exclusion processing in step S640 if it is determined to perform exclusion processing. The target object region estimation result by the target object region estimation unit 540 may be obtained as a target object region mask, a bounding box, or a set of pixels corresponding to the target object region. The occupancy rate may be calculated as the area ratio of the target object region to the detection area. For example, it may be calculated using the number of pixels included in the target object region within the detection area as the numerator and the number of pixels or effective pixels within the detection area as the denominator.

[0103] (Step S660) The determination unit 570 compares the calculated occupancy rate with thresholds (upper and lower thresholds) managed by the setting management unit 520 to determine whether the unit is full or underutilized. The determination unit 570 may also selectively enable at least one of full detection or underutilization detection depending on the settings for each detection area (for example, the activation settings specified on the setting screen in Figure 7). This determination is performed at predetermined time intervals in conjunction with S601. If the occupancy rate meets the threshold condition, proceed to step S670; otherwise, proceed to step S690.

[0104] (Step S670) Furthermore, the determination unit 570 may determine if a threshold condition is met a predetermined number of times consecutively, based on the occupancy rate for multiple frames obtained at predetermined time intervals. Here, the predetermined number of times is the value of the determination condition "X times consecutively" among the contents set in S610 (Figure 7). For example, the determination unit 570 refers to the occupancy rate history for the most recent N times and determines whether the condition has been met N times consecutively. The predetermined number of times N may be, for example, 3 times. If the predetermined number of consecutive occurrences occurs, the process proceeds to step S680; otherwise, the process proceeds to step S690.

[0105] (Step S680) If the determination is successful (step S670: Yes), the notification control unit 580 outputs a notification based on the determination result. The notification may be output as, for example, a push notification to the client device 110, an alert display on the monitoring screen 800 displayed on the client device 110, or a message sent to an external system. On the monitoring screen 800, the detection area frame 810, the detection area name 820, the occupancy rate 830, and the determination result 840 (full or insufficient) may be superimposed on the image, and when the determination is successful, an alert may be displayed for the detection area, and the notification content (for example, full, replenishment required, prompt to move, etc.) may be displayed. Furthermore, even if a notification has already been sent, if a predetermined time (for example, 10 minutes) has elapsed since the previous notification and the determination conditions continue to be met, the notification control unit 580 will output a re-notification. Furthermore, the monitoring screen 800 may be provided with an operator (e.g., a toggle switch) for switching the display of superimposed displays such as the detection area frame 810, occupancy rate 830, and judgment result 840. In addition, the output from the notification control unit 580 is not limited to notifications, and may also be a list of video or frame images corresponding to the time when the judgment is successful, displayed as thumbnails on the notification history screen or monitoring screen 800. The operator may, in response to selecting a thumbnail, transition to the corresponding video playback screen and check the video including the time before and after that point. Furthermore, the notification control unit 580 or the aggregation / display data generation unit 590 may display the point in time when the determination was made in the occupancy rate trend graph in an identifiable manner. For example, a marker may be placed on the plot corresponding to that point in time, or a vertical line, flag, or annotation corresponding to that point in time may be superimposed. Furthermore, the notification control unit 580 or the client device 110 may make information corresponding to the time when the determination is made available for download as CSV data, along with related information. Related information may include the notification time, imaging device identification information, detection area name, occupancy rate, determination result, threshold information, reference information to a representative image, etc. On the other hand, if the determination is not met (step S670: No), no notification is output, and the process proceeds to step S690.

[0106] Furthermore, the monitoring screen 800 or the notification history screen may display a list of past notifications, and if a re-notification is issued, it may be clearly indicated that it is a re-notification. This allows the operator to easily identify detection areas that remain unaddressed.

[0107] In the notification history list display, at least the notification time, imaging device (camera) identification information, detection area name, notification type (full / insufficient, etc.), and a representative image at the time of notification (or reference information to the representative image) may be displayed in association with each other. Representative images may be displayed as thumbnails corresponding to multiple notifications or judgment success points, and the operator may view the corresponding actual video by selecting a thumbnail. Furthermore, the notification history screen may allow users to download relevant information corresponding to each notification or the point at which a decision is made as CSV data. This makes it easier for operators to confirm the situation at the time the notification was issued and to prioritize retrieval actions or conduct post-incident analysis (such as which storage locations experience frequent incidents).

[0108] Furthermore, the notification content in step S680 may be changed depending on the type of detection area. For example, if the first detection area, which corresponds to a drop-off location, becomes full, a notification may be issued prompting the user to move to the second detection area (e.g., "Return to the main storage area to the right of the entrance (second detection area)").

[0109] (Step S690) The aggregation and display data generation unit 590 calculates a first aggregate value, which aggregates the changes in the occupancy rate in the detection area group assigned to the receiving area, and a second aggregate value, which aggregates the changes in the occupancy rate in the detection area group assigned to the return area, and displays them as a graph on the client device 110 as a trend display screen 900. Furthermore, the system may calculate and display the trend of at least one of the sum of the first aggregate value and the second aggregate value, and the value obtained by subtracting the sum from the total number of carts stored for each store. Furthermore, on the trend display screen 900, reference lines corresponding to the upper and lower threshold limits may be superimposed on the graph, and an operator may be provided to switch whether or not to display the reference lines. In addition, it may be possible to select the detection area group to be displayed, and the series corresponding to each detection area may be displayed in a way that allows identification in correspondence with the display color set on the settings screen 700. Furthermore, the progress display screen 900 may display the point in time when the judgment is successful in an identifiable way on the graph. For example, a marker, flag, label, annotation, vertical line, or color change may be added to the position corresponding to that point in time. In addition, depending on the selection of such identification display, the system may transition to a thumbnail list display or video playback screen corresponding to that point in time. This allows operators to get an overview of trends in the distribution of carts, as well as signs of shortages or overflows, and to easily check the actual video footage to understand when an event occurred.

[0110] (Step S695) The control unit 210 (for example, the image acquisition unit 510) determines whether or not to terminate the monitoring process. For example, the determination may be made based on input of a termination instruction by the operator, or the elapsed time of a set monitoring period. If it is determined that the monitoring process should not be terminated (step S695: No), the process returns to step S610, the image acquisition unit 510 acquires the image of the next frame, and the subsequent processing is repeated. On the other hand, if it is determined that the monitoring process should be terminated (step S695: Yes), the control unit 210 (for example, the image acquisition unit 510) terminates the series of processes shown in this flowchart.

[0111] (Effects and Benefits) According to Embodiment 1, the occupancy rate of the target object in the detection area is calculated based on the image obtained from the imaging device, and a notification is output based on the threshold determination of the occupancy rate, so that a notification can be output according to the occupancy status (full / insufficient) of the cart storage area. Furthermore, by configuring the system to perform person area estimation and exclusion processing, it is possible to suppress instability and miscalculation of the occupancy rate caused by the inclusion or occupancy of people, thereby reducing unnecessary notifications (false notifications) and enabling notifications to be output at the appropriate time.

[0112] In particular, as in this embodiment, by estimating the area of ​​a person that appears within the detection area and using an occupancy rate that reduces the influence of that area, and by configuring the system so that a determination is made only when the occupancy rate satisfies a threshold condition for a predetermined number of consecutive times, false detections and unnecessary notifications can be significantly suppressed even in real environments where people frequently appear in the images. As a result, wasted effort in retrieval work and unnecessary patrols can be reduced, and highly reliable notification control suitable for actual operation can be realized.

[0113] Furthermore, by calculating the occupancy rate for multiple frames obtained at predetermined time intervals and configuring the system to establish a determination when the occupancy rate meets a threshold condition for a predetermined number of consecutive times, it is possible to suppress determinations caused by temporary noise, image fluctuations, momentary inclusion of passersby, etc. This improves the stability and reliability of notifications and reduces unnecessary operational responses.

[0114] The predetermined time interval and the number of consecutive scans can be adjusted according to the monitored environment (traffic volume, imaging device field of view, degree of lighting fluctuations, etc.). For example, the number of scans can be increased in environments where many pedestrians are captured in images, and the predetermined time interval can be shortened in environments where responsiveness is important. This allows for a balance between suppressing false detections and providing immediate notifications, while achieving judgment stability that is suitable for the operating environment.

[0115] Furthermore, by enabling at least one of full-capacity detection and low-capacity detection for each detection area, and by configuring the system to automatically apply low-capacity detection in the receiving area and full-capacity detection in the returning area in conjunction with the assignment of receiving / returning areas, monitoring and notification that are consistent with the purpose of the detection area can be easily achieved, reducing the configuration and operational burden.

[0116] Furthermore, by estimating the target object area through segmentation or object detection and calculating the occupancy rate, the system is less susceptible to the effects of background fluctuations, lighting changes, shadows, etc., and can easily obtain an occupancy rate consistent with the presence of the target object. In addition, by excluding the overlapping portion from the area calculation when the human area and the target object area overlap, variations in judgment can be suppressed and misjudgments can be reduced even in situations where partial occlusion occurs due to a person.

[0117] Furthermore, by configuring the system to output a notification prompting movement to the second detection area when the first detection area, which corresponds to a drop-off location, becomes full, congestion and uneven distribution in specific areas can be suppressed, and the efficiency of collection and redistribution throughout the store can be improved.

[0118] Furthermore, by associating the imaging device with each store and maintaining the total number of carts for each store, and by aggregating and displaying the changes in the occupancy rate of the pick-up area group and the return area group in graph form, it is possible to grasp the changes in status for each type of location over time. In addition, by displaying the trends in the sum of the first aggregated value and the second aggregated value, and the value obtained by subtracting this sum from the total number of carts, it is possible to grasp the degree of uneven distribution and congestion of carts at a glance, which can support decisions regarding collection, replenishment, and rearrangement.

[0119] (modified version) Although embodiments of the present invention have been described above, the present invention is not limited to the above embodiments. Hereinafter, Modifications 1 to 12 are illustrated. Note that the following modifications can be combined in any way that is not contradictory to one another.

[0120] (Modification 1: Specific example of determining whether to execute exclusion processing) The control unit 210 may determine whether or not to perform exclusion processing based on an exclusion processing enable flag set for each detection area or imaging device. Furthermore, the determination may be made according to the time of imaging, the brightness information of the image, or the time period classification based on these. For example, exclusion processing may be enabled during peak hours or low-light hours, and disabled during other times. Furthermore, the determination may be made by referring to a simplified estimation result obtained from low-resolution person detection, or the estimation result of whether or not a person was present in the previous frame. According to Modification 1, the execution of exclusion processing can be switched on or off depending on the detection area, imaging device, or imaging conditions such as time and brightness. This allows for adjustment of the balance between suppressing false detections and processing load (computational load) or responsiveness according to the operating conditions.

[0121] (Variation 2: Specific example of how to calculate occupancy rate) The occupancy rate calculation unit 560 may calculate the occupancy rate as O / (A-H), where A is the total number of pixels in the detection area, H is the number of pixels corresponding to the human area, and O is the number of pixels whose area is calculated as the target object. This ensures that even if the effective area decreases due to a person appearing in the detection area, the occupancy rate index is less likely to be affected by the amount of the person appearing. According to specific example 1 of variation 2, by subtracting the area of ​​the person from the denominator, even when the effective area fluctuates due to the inclusion of a person, the occupancy rate can be prevented from being excessively influenced by the amount of the person, thereby improving the stability of the judgment.

[0122] (Variation 2: Specific example of how to calculate occupancy rate) On the other hand, the occupancy rate calculation unit 560 may calculate the occupancy rate as O / A. For example, if the size of the detection area differs between stores, and it is desired to compare the degree of occupancy based on the absolute area of ​​the detection area, it is preferable to fix the denominator. In this case as well, pixels corresponding to the human area are excluded from the area calculation (numerator) as the target object. According to specific example 2 of variation 2, by fixing the denominator, it becomes easier to compare the degree of occupancy across multiple stores and locations with different detection area dimensions, which contributes to the formulation and analysis of cross-store operational standards.

[0123] (Modification 3: A specific example of exclusion based on confidence level) When the person region estimation unit 530 outputs a confidence level for the person region estimation, the exclusion processing unit 550 may, instead of uniformly excluding pixels corresponding to the person region, change the percentage of pixels excluded from the area calculation according to the confidence level. For example, the strength of exclusion may be weakened when the confidence level is low, and strengthened when the confidence level is high. According to Modification 3, even when there is uncertainty in the estimation of the person's domain, the strength of exclusion can be adjusted according to the confidence level. This suppresses excessive correction of the occupancy rate due to incorrect person estimation and improves the robustness of the occupancy rate calculation.

[0124] (Modification 4: Specific example of excluding overlapping portion) If the human area overlaps with the target object area within the detection area, the exclusion processing unit 550 may exclude only the pixels corresponding to the overlapping portion of the human area with the target object area from the area calculation. For example, pixels included in the logical AND of the target object area mask and the human area mask may be excluded. The target object area may be the one estimated in step S650, or it may be one estimated prior to step S650 (for example, the estimation result of the previous frame). According to Modification 4, even if superposition occurs, such as when a person is positioned in front of the target object, by excluding only the superimposed portion, errors originating from the person can be suppressed without excessively reducing the contribution of the target object, and judgment stability can be improved, especially near the threshold.

[0125] (Variation 5: Specific example 1 of target object region estimation) The target object region estimation unit 540 may estimate a region mask corresponding to the target object by segmentation and calculate the occupancy rate based on the region mask. In this case, area-based calculation becomes possible even in scenes where the target objects are densely clustered. According to specific example 1 of modified example 5, even when the target objects are densely packed and it is difficult to distinguish individual outlines, the occupancy rate can be calculated based on area information, making it easier to reliably detect conditions such as full capacity.

[0126] (Variation 5: Specific example 2 of target object region estimation) The target object area estimation unit 540 may estimate the number of target objects detected within the detection area by object detection and calculate the occupancy rate as a value normalized by the maximum capacity set in advance for each detection area. For example, if the number of storage slots in the storage area is clear, the occupancy rate based on the number of units is easier to understand in operation. According to specific example 2 of variation 5, in a storage area with a clearly defined number of available slots, the occupancy rate can be expressed on a unit basis, providing an indicator that is intuitive for the operator, thus improving the ease of setting thresholds and understanding the status.

[0127] (Variation 6: Specific example 1 of pickup / return area allocation) The configuration management unit 520 may, based on the operator's selection on the configuration screen 700 displayed on the client device 110, collectively assign the detection area group closer to the entrance side as the receiving area and the detection area group closer to the exit side as the return area. For example, if the operator selects either the entrance side or the exit side, the receiving area / return area may be automatically assigned based on that selection, according to the relative positions of each detection area. According to specific example 1 of variation 6, the operator can easily assign multiple detection areas at once, thereby reducing the initial setup time and optimizing detection (insufficient / full) according to the intended use (receive / return).

[0128] (Modification 6: Specific example 2 of pickup / return area allocation) The configuration management unit 520 may estimate the direction of movement of a person near the detection area from the image and assign the area where the person is moving towards the store as the receiving area and the area where the person is moving towards the store as the return area. For example, the assignment may be updated by using person tracking to determine that the flow is predominantly from outside to inside at the entrance and from inside to outside at the exit. According to specific example 2 of variation 6, since the allocation can be updated based on actual foot traffic, it is easy to keep the distinction between pick-up and return in line with the current situation even if the layout changes or the flow of people changes.

[0129] (Modification 7: Specific example of guidance notification) If the first detection area, which corresponds to the drop-off area, becomes full, the notification control unit 580 may output a notification that prompts the user to move to the second detection area, as well as a notification that includes the destination second detection area or installation location (for example, "Return to the main storage area (second detection area) to the right of the entrance"). According to Modification 7, the notification goes beyond simply notifying customers that the containers are full; it includes specific destinations to encourage retrieval or redeployment, thereby accelerating the resolution of uneven distribution.

[0130] (Variation 8: Specific example of integration with in-store signage) The notification control unit 580 may, in lieu of or in addition to notifying the client device 110, transmit a control signal to the in-store digital signage, indicator lights, or voice guidance device to display or output a guidance message when the store is full. Furthermore, the display or output may be canceled when the store is no longer full. According to Modification 8, since guidance can be provided to users using in-store equipment, it is easier to encourage the movement of carts without relying on the actions of collection staff, which contributes to reducing congestion and improving safety.

[0131] (Variation 9: Specific example of deterring repeated notification) The notification control unit 580 may, after outputting a notification, suppress further notifications of the same type (full / insufficient) for the same detection area for a predetermined period of time. For example, it may store the notification time and refrain from outputting further notifications of the same type until a predetermined period of time has elapsed since the initial notification. According to specific example 1 of modification 9, it is possible to suppress the frequent sending of notifications while the same state continues, thereby reducing notification fatigue and the increased burden on operators.

[0132] (Modification 9: Specific examples of resolution notices and hysteresis) The notification control unit 580 may output a resolution notification when the full or insufficient state is resolved. In addition, the threshold for full capacity and the threshold for releasing the full capacity may be set differently, or the threshold for insufficient capacity and the threshold for releasing the insufficient capacity may be set differently (hysteresis) to suppress chattering of judgments near thresholds. According to specific example 2 of modification 9, it becomes possible to understand the resolution and manage state transitions. Furthermore, hysteresis can suppress chattering near the threshold, improving the stability of notifications.

[0133] (Variation 10: Specific example of estimating uncaptured items based on the total number of carts) The aggregation and display data generation unit 590 may estimate the number of uncaptured carts (for example, total number of carts minus the sum of According to Modification 10, it is possible to grasp the estimated number of carts located outside the detection area, thereby making operational issues such as carts accumulating outside the field visible at an early stage and prompting countermeasures (such as strengthening collection).

[0134] (Example 11: Specific example of missing information) If the image acquisition unit 510 is unable to acquire an image for a predetermined period, the determination unit 570 may suspend the determination of the detection area and display a status such as "not acquired" or "communication interrupted" on the monitoring screen 800. In addition, the occupancy rate during the missing period may be recorded as missing, and the missing interval may be distinguished and displayed on the trend display screen 900. According to Modification 11, even if image loss or communication interruption occurs, the loss status can be visualized while suppressing incorrect judgments and notifications, thereby improving operational quality (reliability).

[0135] (Variation 12: Variation of screen output control) In the above embodiment, it was explained that the server device 100 generates information to be used for displaying UI screens (for example, a settings screen 700, a monitoring screen 800, and a progress display screen 900) displayed on the client device 110 using the notification control unit 580 and / or the aggregation / display data generation unit 590, and controls the output (display) to the client device 110. However, depending on the processing capacity or communication environment of the client device 110, output control (display control) may be performed in any of the following manners or a combination thereof. According to Modification 12, the display control method can be selected according to constraints such as communication volume, rendering load, and terminal performance, making it easier to ensure the responsiveness and operability of the user interface.

[0136] • Screen data transmission type (server-side rendering): In this configuration, the server device 100 generates a rendering result of the entire screen, including occupancy rate, judgment results, superimposed display of detection areas, alert displays, or trend graphs (for example, an HTML snapshot, bitmap, or video stream), and sends it to the client device 110 for display. This configuration is suitable, for example, when displaying on a terminal with low rendering capabilities, or when it is not desirable to store detailed display logic or configuration information on the client device 110 side.

[0137] • Configuration data transmission type (client-side rendering): In this configuration, the server device 100 transmits structured data (JSON, XML, etc.) that describes the definition of the detection area (coordinates, area ID, etc.), threshold and activation settings, occupancy rate and judgment status, and time-series data for displaying changes, and the receiving application (client device 110) interprets this data and performs rendering. This configuration is suitable, for example, when it is desired to improve responsiveness or to perform smooth screen transitions and display updates.

[0138] • Command transmission type: The server device 100 sends trigger control commands to the client device 110, such as "Display full alert," "Display low alert," "Transition to progress display screen," or "Update display," and the specific display content and format are output based on templates or assets held by the client device 110. This configuration is effective when you want to reduce communication traffic or when you want to perform routine display updates or notifications.

[0139] (Variation 13: Specific example of lighting conditions) If estimation accuracy decreases due to nighttime or backlighting, the image acquisition unit 510 or preprocessing unit (not shown) may input an image to which image correction such as gamma correction, noise reduction, or exposure control has been applied for person area estimation and target object area estimation. According to Modification 13, the decrease in estimation accuracy caused by fluctuations in lighting conditions can be suppressed, making it easier to achieve monitoring that is less dependent on time of day or environment.

[0140] (Variation 14: Specific example of the execution location) As described above, at least part of the estimation of the person area, the estimation of the target object area, the exclusion process, and the calculation of the occupancy rate may be performed by the imaging device 120 or an edge device in the store instead of the server device 100. In this case, the server device 100 may receive only the occupancy rate, the judgment result, or the notification data to perform the judgment and notification. According to Modification 14, processing can be divided according to the terminal and network configuration, which contributes to reducing latency, reducing communication load, or enabling autonomous operation within the store.

[0141] (Variation 15: Method, Program, Medium) The processes relating to this disclosure may be implemented as methods, or they may be provided as programs that perform each process. Furthermore, such programs may be provided on a computer-readable non-temporary recording medium, or they may be distributed via a communication line. According to Modification 15, the same technological concept can be implemented in various forms as methods, programs, and media, thereby increasing the flexibility of implementation.

[0142] <Note> The product may be provided in any of the following embodiments. (Note 1) A system that analyzes images captured by a camera, In the aforementioned video, an area is defined, In the aforementioned video, an object is detected, Determine the occupancy rate of the object in the area, Determine whether the occupancy rate is above the upper threshold or below the lower threshold. Depending on the result of the above determination, the output is executed. system.

[0143] According to the configuration described in Appendix 1, an area is defined in the video, objects are detected, the occupancy rate of objects within that area is determined, and a decision is made based on the relationship with an upper or lower threshold. Output is then executed according to the result, thereby enabling an appropriate understanding of whether there is an excess or deficiency of objects within the area, and providing information or facilitating appropriate responses in accordance with the situation.

[0144] (Note 2) The above determination is performed at predetermined time intervals. The system described in Appendix 1.

[0145] According to the configuration described in Appendix 2, the judgment is performed at predetermined time intervals, allowing for continuous monitoring of the video and the ability to grasp changes in state at regular intervals. This makes it easier to capture the state progression of the monitored object over time.

[0146] (Note 3) The output is executed when the determination that the value is above the upper threshold or below the lower threshold occurs a predetermined number of times consecutively. The system described in Appendix 1.

[0147] According to the configuration described in Appendix 3, the output is generated when the threshold condition is met a predetermined number of times consecutively, thus suppressing unwanted outputs caused by temporary fluctuations or noise. This improves the reliability of the output.

[0148] (Note 4) If the area contains people, the determination is skipped. The system described in Appendix 1.

[0149] According to the configuration described in Appendix 4, the detection process is skipped if a person is present in the area, thus suppressing false detections caused by people being captured in the image. This enables stable monitoring that is less susceptible to human influence.

[0150] (Note 5) An exclusion process is performed to reduce the impact of people within the aforementioned area. The system described in Appendix 1.

[0151] According to the configuration described in Appendix 5, exclusion processing is performed to reduce the influence of people within the area, so even if occupancy or overlapping by people occurs, the occupation status of objects can be evaluated more appropriately. This improves the accuracy of the judgment.

[0152] (Note 6) If the condition remains true after the above determination is met and the output is executed, and a predetermined time has elapsed, the output will be executed again. The system described in Appendix 1.

[0153] According to the configuration described in Appendix 6, output is executed again if the condition remains valid, so even if the initial output is overlooked or there is a delay in response, the system can re-recognize the continued condition. This makes it easier to prevent the condition from becoming prolonged.

[0154] (Note 7) For each area, it is possible to enable at least one of the upper limit detection using the upper limit threshold or the lower limit detection using the lower limit threshold. The system described in Appendix 1.

[0155] According to the configuration described in Appendix 7, at least one of upper limit detection or lower limit detection can be enabled for each area, allowing for the selection of an appropriate detection mode according to the use or operational purpose of each area. This enables necessary detection while suppressing unnecessary detections.

[0156] (Note 8) In areas where the number of objects tends to decrease, the lower limit detection is enabled. In areas where the number of the aforementioned objects tends to increase, the upper limit detection is enabled. The system described in Appendix 7.

[0157] According to the configuration described in Appendix 8, lower limit detection is enabled in areas where the number of objects tends to decrease, and upper limit detection is enabled in areas where the number of objects tends to increase, allowing for detection in accordance with the direction of changes in the area's state. This enables monitoring that is in line with actual operational conditions.

[0158] (Note 9) If the upper limit detection is achieved in the first area, the movement of the object to the second area is prompted. The system described in Appendix 7.

[0159] According to the configuration described in Appendix 9, when an upper limit is detected in the first area, the movement of objects to the second area is encouraged, making it easier to correct congestion or uneven distribution in specific areas. This helps to optimize the overall layout.

[0160] (Note 10) Each camera is associated with a store and stored in memory, and the total number of the aforementioned objects is stored for each store. The system described in Appendix 1.

[0161] According to the configuration described in Appendix 10, each camera is associated with a specific store and the total number of objects in each store is stored, making it easier to manage and analyze data on a store-by-store basis. This helps to understand the operational status of each store.

[0162] (Note 11) The system calculates a first aggregate value, which is the occupancy rate in the area group where the upper limit detection is enabled, and a second aggregate value, which is the occupancy rate in the area group where the lower limit detection is enabled. The system described in Appendix 7.

[0163] According to the configuration described in Appendix 11, a first aggregate value is calculated for the area group where upper limit detection is enabled, and a second aggregate value is calculated for the area group where lower limit detection is enabled, making it easier to understand the status trends for each detection type. This facilitates management across multiple areas.

[0164] (Note 12) The system calculates the trend of at least one of the sum of the first aggregate value and the second aggregate value, and the value obtained by subtracting the sum of the first aggregate value from the total number of objects. The system described in Appendix 11.

[0165] According to the configuration described in Appendix 12, the system calculates the trend of at least one of the following: the sum of the first and second aggregate values, and the value obtained by subtracting this sum from the total number of objects. This makes it easier to understand the relationship between the amount of objects within the area that can be tracked and the amount of objects estimated to exist outside the area in a time series. This supports the analysis of uneven distribution or uncaptured conditions.

[0166] (Note 13) An information processing method performed by a system that analyzes images captured by a camera, In the aforementioned video, an area is defined, In the aforementioned video, an object is detected, Determine the occupancy rate of the object in the area, Determine whether the occupancy rate is above the upper threshold or below the lower threshold. Depending on the result of the above determination, the output is executed. Information processing methods.

[0167] According to the configuration described in Appendix 13, the processing to be implemented as a system can be understood as a method, allowing the same technical concept to be implemented regardless of the device configuration. This increases the flexibility of the embodiments.

[0168] (Note 14) A computer analyzes the images captured by the camera. The process of setting an area in the aforementioned video, The process of detecting an object in the aforementioned video, A process for determining the occupancy rate of the object in the area, A process to determine whether the occupancy rate is above an upper threshold or below a lower threshold, Depending on the result of the above determination, the process of executing output is performed, A program to execute.

[0169] According to the configuration described in Appendix 14, the system can be provided as a program that allows a computer to perform each process of video analysis, occupancy rate calculation, judgment, and output execution, making it easy to install as software on existing computing resources. This improves scalability and implementation flexibility.

[0170] While various embodiments of the present invention have been described, these are presented as examples only and are not intended to limit the scope of the invention. Novel embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. Embodiments and variations thereof are included in the scope and spirit of the invention, as well as in the claims and their equivalents. Each variation may be combined in any way desired. [Explanation of Symbols]

[0171] 100: Server device 110: Client device 120: Imaging device 150: Network 210: Control Unit 220: Storage section 230: Communications Department 1000: Surveillance system

Claims

1. A system that analyzes images captured by a camera, In the aforementioned video, an area is set that corresponds to a designated location in a store or facility where objects that are mobile or physical items are collected or returned. In the aforementioned video, the object is detected, Determine the occupancy rate of the object in the area, If the occupancy rate is equal to or greater than the upper threshold, it is determined that the area is full, and a notification is output including the identification information of the area and the fact that it is full. If the occupancy rate is below a lower threshold, the system determines that the occupancy status of the object in the area is insufficient, and outputs a notification including the identification information of the area and the fact that it is insufficient. system.

2. The occupancy rate is determined at predetermined time intervals, and a determination is made at predetermined time intervals as to whether the container is full or understaffed. The system according to claim 1.

3. The notification is output when the determination that the container is full or that it is insufficient is made a predetermined number of times consecutively. The system according to claim 2.

4. If the area contains people, the determination of whether it is full or understaffed is skipped. The system according to claim 1.

5. An exclusion process is performed to reduce the influence of people within the area, and the occupancy rate is determined based on the results of the exclusion process. The system according to claim 1.

6. If the full state or the insufficient state persists even after a predetermined time has elapsed since the notification was output, the notification is output again. The system according to claim 1.

7. A plurality of areas are set as the area, For each of the aforementioned multiple areas, the system accepts a setting to enable at least one of the detection of a full state based on the upper threshold or the detection of a shortage state based on the lower threshold. The system according to claim 1.

8. In the above setting, the detection of the shortage state is enabled in areas where the number of objects tends to decrease, and the detection of the full state is enabled in areas where the number of objects tends to increase. The system according to claim 7.

9. The plurality of areas include a first area and a second area, If it is determined that the first area is full, a notification prompting the object to move to the second area is output. The system according to claim 7.

10. Each camera is associated with a store or facility and stored in memory; the total number of the aforementioned objects is stored for each store or facility. The system according to claim 1.

11. A first aggregate value obtained by aggregating the occupancy rate in the area group in which detection of the full state is enabled, and a second aggregate value obtained by aggregating the occupancy rate in the area group in which detection of the shortage state is enabled, The system according to claim 7.

12. Store the total number of the above-mentioned objects for each store or facility, The system calculates the trend of at least one of the sum of the first aggregate value and the second aggregate value, and the value obtained by subtracting the sum of the first aggregate value from the total number of objects. The system according to claim 11.

13. An information processing method performed by a system that analyzes images captured by a camera, In the aforementioned video, an area is set that corresponds to a designated location in a store or facility where objects that are mobile or physical items are collected or returned. In the aforementioned video, the object is detected, Determine the occupancy rate of the object in the area, If the occupancy rate is equal to or greater than the upper threshold, it is determined that the area is full, and a notification is output including the identification information of the area and the fact that it is full. If the occupancy rate is below a lower threshold, the system determines that the occupancy status of the object in the area is insufficient, and outputs a notification including the identification information of the area and the fact that it is insufficient. Information processing methods.

14. A computer analyzes the images captured by the camera. In the aforementioned video, the process of setting an area corresponding to a designated location in a store or facility where objects that are mobile or physical items are collected or returned, In the aforementioned video, the process of detecting the object is performed, A process for determining the occupancy rate of the object in the area, If the occupancy rate is equal to or greater than the upper threshold, the process determines that the area is full and outputs a notification including the identification information of the area and the fact that it is full. If the occupancy rate is below a lower threshold, the process determines that the occupancy status of the object in the area is insufficient, and outputs a notification including the identification information of the area and the fact that it is insufficient. A program to execute.

Citation Information

Patent Citations

  • Device for detecting abnormality conveyor controller

    JP1979000358A

  • Merchandise monitoring system

    JP1993081552A

  • Image processing system, image processing program, and image processing method

    JP2006048468A

  • Handling equipment and handling method for carton

    JP2015074482A

  • Merchandise monitoring device, merchandise monitoring system, and merchandise monitoring method

    JP2016057952A