System for store management using multiple cameras and method for its operation

Abstract

A method of operation of a store management system operated by at least one processor, wherein initial images are received from a plurality of cameras installed in a store, and based on the initial images, the shooting ranges of the plurality of cameras are masked on a floor image of the store so that the shooting ranges of the plurality of cameras do not overlap. Depending on the type of service mapped to each of the plurality of cameras, a region of interest is set in the shooting range captured by the plurality of cameras, images captured by the plurality of cameras are received, the images are masked to generate a de-identified image, and a service is provided in the store based on the de-identified image.

Description

Technology Field

[0001] The present invention relates to a store management system using multiple cameras that assists employees and manages the store in real time at a food and beverage store using multiple cameras, and a method of operation thereof. Background Technology

[0002] Food and beverage establishments, such as cafes or fast-food restaurants, employ staff to provide service. In addition, visiting customers utilizing the services provided by the establishment, as well as various equipment (e.g., kiosks) and furniture (e.g., tables, chairs, counters, dish return areas, etc.) equipped within the establishment, constitute the food and beverage space.

[0003] In order to manage the interior of such food and beverage establishments, traditionally, employees would walk around the store to organize dishes piled up on the dish return rack, clear trash from tables, or adjust the position of chairs. Consequently, due to manual management by employees, it takes a long time to clean up the store when it is crowded or when there are few staff members.

[0004] Furthermore, in order for customers visiting a food and beverage establishment to secure a table for service, they must walk around the store checking for occupancy to find an empty seat. Consequently, when the store is crowded, it takes a long time for customers to secure a table, and there is the inconvenience of having to repeatedly check the interior until an empty table is found.

[0005] Meanwhile, traditional retailers prioritize cost minimization when determining store layout and operational methods. To achieve this, they seek to accurately understand the behavior and preferences of visiting customers; however, the process of collecting and analyzing data such as customer behavior and preferences is complex and costly.

[0006] To address this, there are increasing attempts to reduce the number of employees performing additional routine tasks by utilizing artificial intelligence technology to continuously monitor the situation within the store. However, in the case of multi-story food and beverage establishments, employees are physically moving around the various floors to check seat occupancy and personally inspect the dishes on the tables that need to be collected.

[0007] In particular, it is difficult to perform tasks such as informing customers about seat availability or clearing dishes from the return area during busy times. Furthermore, due to the Personal Information Protection Act, it is impossible to manage the store interior by referring to CCTV footage.

[0008] Furthermore, due to the nature of the multi-story store structure, it is difficult to assess the overall store situation using CCTV footage. Additionally, simply combining the results obtained from individual footage captured by multiple CCTVs leads to inaccurate findings due to overlapping areas between the cameras. The problem to be solved

[0009] Accordingly, the present invention provides a store management system using multiple cameras and a method of operation thereof, which analyzes the store situation based on video footage collected inside the store using multiple cameras and CCTV installed inside the store, and manages the store or provides services to visiting customers based on the analyzed store situation. means of solving the problem

[0010] A method of operation of a store management system operated by at least one processor, which is one feature of the present invention for achieving the technical problem of the present invention, comprising: receiving initial images from a plurality of cameras installed in a store and, based on the initial images, masking the shooting ranges of the plurality of cameras on a drawing image of the store so that the shooting ranges of the plurality of cameras do not overlap; setting a region for calculating the complexity of the store and a region for measuring the contamination level of the store in the shooting range captured by the plurality of cameras as regions of interest, respectively, according to the type of store management service mapped to each of the plurality of cameras; and masking an image including the regions of interest captured by the plurality of cameras to generate a de-identified image, and providing the store management service based on the de-identified image.

[0011] The step of masking the shooting range may include the step of identifying reference points set in the drawing image, the step of extracting a plurality of points corresponding to the reference points from the initial images, and the step of deriving a perspective transformation matrix so that the initial images are projected onto the drawing image by mapping the plurality of points to the reference points.

[0012] The step of masking the shooting range may include the step of masking the shooting area captured by the plurality of cameras on the drawing image, and the step of setting one of the plurality of cameras to capture the overlapping shooting areas when a portion of the shooting areas captured by the plurality of cameras overlap.

[0013] The step of setting the area of ​​interest may further include the step of setting the area of ​​an object including a dish return rack or a plurality of tables installed in the store as the area of ​​interest for measuring the contamination level.

[0014] The step of providing the store management service may include inputting images captured by the plurality of cameras into a de-identification module trained to extract face regions from an image to extract face regions of the store customers or store employees from the image, and de-identifying the face regions using the de-identification module to generate the de-identified image.

[0015] The step of providing the store management service may include extracting the location of a store customer and the locations of a plurality of seats placed in the store from the de-identified image, mapping the location of the customer to the seat closest to the customer's location among the plurality of seats, and providing information on non-preempted seats among the plurality of seats in the store that have not been preempted by the store customer to a management terminal installed in the store.

[0016] The step of providing the above store management service may include the step of extracting customers who have visited the store from the above video to determine the number of customers who have visited the store, and the step of calculating the congestion level of the store based on the number of customers and the number of maximum personnel previously set for the store, and providing congestion level information corresponding to the calculated congestion level.

[0017] The step of providing the above-mentioned store management service may include the step of obtaining information on the gender, age group, and time of visit of customers who visited the store in the above-mentioned video as store operation data.

[0018] The step of providing the store management service may include: a step of extracting and dividing the object from an initial image capturing a region of interest set to measure the contamination level of the store and storing it as a reference division area; a step of extracting and dividing the object from images capturing the regions of interest, and calculating the sum of the number of labels dividing the object based on the reference division area and the pixel area values ​​of the labels; a step of calculating the complexity of each of the regions of interest based on the number of labels and the sum of the pixel area values; and a step of transmitting the contamination level of the region of interest to a management terminal within the store if the calculated complexity is higher than a preset threshold.

[0019] As another feature of the present invention for achieving the technical problem of the present invention, a store management system comprises an interface linked with a plurality of cameras already installed in a store and a processor, wherein the processor receives initial images from a plurality of cameras installed in a store, masks the shooting ranges of the plurality of cameras on a drawing image of the store based on the initial images so that the shooting ranges of the plurality of cameras do not overlap, sets a region of interest in the shooting ranges captured by the plurality of cameras according to the type of store management service mapped to each of the plurality of cameras, receives images captured by the plurality of cameras, generates a de-identified image by masking the images, and then provides the store management service based on the de-identified image.

[0020] The processor can identify reference points set in the drawing image, extract a plurality of points corresponding to the reference points from the initial images, and then mark the plurality of points on the reference points to derive a perspective transformation matrix.

[0021] The processor can mask the shooting area captured by the plurality of cameras on the drawing image, and if a portion of the shooting areas captured by the plurality of cameras overlap, it can be configured so that one of the plurality of cameras captures the overlapping shooting areas.

[0022] The processor may set an area for calculating the complexity of the store as a region of interest, and may set an area of ​​an object including a dish return rack or a plurality of tables installed in the store as a region of interest to measure the contamination level of the store.

[0023] The processor can input images captured by the plurality of cameras into a de-identification module trained to extract face regions from an image, extract face regions of the store customers or store employees from the image, and de-identify the face regions using the de-identification module to generate the de-identified image.

[0024] The processor can extract the location of a store customer and the locations of a plurality of seats placed in the store from the de-identified image, map the location of the customer to the seat closest to the customer's location among the plurality of seats, and then provide information on the non-preempted seats among the plurality of seats in the store that have not been preempted by the store customer to a management terminal installed in the store.

[0025] The processor can extract customers who have visited the store from the image, determine the number of customers who have visited the store, calculate the congestion level of the store based on the number of customers and the number of maximum personnel set for the store, and provide congestion level information corresponding to the calculated congestion level.

[0026] The processor extracts and divides the object from an initial image captured for a region of interest set to measure the contamination level of the store and stores it as a reference division area; extracts and divides the object from images captured of the regions of interest; calculates the number of labels that divide the object based on the reference division area and the sum of the pixel area values ​​of the labels; calculates the complexity of each of the regions of interest based on the number of labels and the sum of the pixel area values; and if the calculated complexity is higher than a preset threshold, transmits the contamination level for the region of interest to a management terminal within the store. Effects of the invention

[0027] According to the present invention, information can be provided so that store employees can easily understand the situation inside the store by utilizing existing infrastructure, such as CCTV cameras, without additional construction or infrastructure building.

[0028] In addition, by utilizing customer behavior analysis based on computer vision algorithms, store managers can make decisions, thereby maximizing the efficiency of store operations.

[0029] In addition, information such as store congestion and area of ​​interest complexity can be processed in real time to enable immediate operational adjustments, and complex data processing, such as calculating the number of customers entering and exiting and estimating customer information, can be performed in real-time or non-real-time.

[0030] In addition, by calculating the congestion level by floor of the store and generating a heatmap, it enables employees to respond flexibly to crowded areas. Brief explanation of the drawing

[0031] FIG. 1 is an example of an environment in which a store management system according to an embodiment of the present invention is applied. Figure 2 is a structural diagram of a store management system according to an embodiment of the present invention. FIG. 3 is a flowchart illustrating the operation of a store management system according to an embodiment of the present invention. FIGS. 4 to 7 are exemplary diagrams of services provided using de-identified images according to embodiments of the present invention. FIG. 8 is an exemplary diagram of a store management service provided according to an embodiment of the present invention. FIG. 9 is a hardware configuration diagram of a computing device according to an embodiment of the present invention. Specific details for implementing the invention

[0032] Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement them. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0033] Throughout the specification, when a part is described as "including" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0034] Hereinafter, a store management system and a method of operation according to an embodiment of the present invention will be described in detail with reference to the drawings.

[0035] FIG. 1 is an example of an environment in which a store management system according to an embodiment of the present invention is applied.

[0036] As illustrated in FIG. 1, the store management system (100) interacts with a plurality of cameras (200: 200-1 to 200-5) already installed in the store (10) and collects images captured by each camera (200). Additionally, the store management system (100) can interact with a POS terminal (300) installed in the store (10). Furthermore, the store management system (100) can interact with various devices or viewers and can provide store management results to a separate display or device. However, for convenience of explanation, the embodiment of the present invention describes the case where the POS terminal (300) receives the analyzed results.

[0037] In the embodiment of the present invention, five cameras (200-1 to 200-5) are installed in a single-story store (10), but this is not necessarily limited to this. That is, even if the store (10) is a building composed of multiple floors, the store management system (100) can receive images from multiple cameras (200) installed on each floor and provide store management services (e.g., guidance on unoccupied seats, measurement of contamination levels of dish return counters or seats, provision of store congestion levels, store management, etc.).

[0038] Each camera (200) is an IP camera, and the explanation is given as an example where identification information is assigned to each camera (200).

[0039] The store management system (100) is described as being implemented using edge computing (Edge PC) to minimize space utilization and analysis costs suitable for the store (10). Here, edge computing is a technology for processing data on local devices (e.g., smartphones, tablets, IoT devices, etc.). Since edge computing is already known, a detailed description is omitted in the embodiments of the present invention.

[0040] The store management system (100) analyzes the video captured by each camera (200). At this time, the store management system (100) divides the cases where the video must be analyzed in real time and the cases where the video must be analyzed non-real time, and collects and stores the results of the video analysis for the protection of personal information.

[0041] For example, the store management system (100) analyzes video in real time when measuring the level of congestion or the presence of occupied seats within the store (10), or the level of contamination of unoccupied seats, dish return areas, or self-service bars. On the other hand, to optimize store operations, the number of customers who visited the store (10) on the day can be calculated from the video by providing a cumulative number of people in real time or by performing it non-real-time, allowing for calculation in separate time intervals.

[0042] To this end, the store management system (100) receives video from each camera (200) in real time via a Real Time Streaming Protocol (RTSP). Then, to receive video from multiple cameras (200) simultaneously, the store management system (100) applies multithreading to receive video asynchronously at a defined period for each camera (200), as an example.

[0043] The store management system (100) can check if there are any unoccupied seats in the store (10) and inform the store staff whether there are any tables not occupied by customers. Additionally, the store management system (100) can check the contamination level of the dish return area or the self-service bar and inform the store staff how many dishes need to be cleaned up are piled up on the dish return area.

[0044] Additionally, the store management system (100) can determine the level of congestion of the store (10). Additionally, the store management system (100) can calculate the number of visiting customers who have visited the store (10) and collect this as store operation data. A manager managing the store (10) can use the store operation data provided by the store management system (100) to adjust marketing strategies (e.g., adjusting table locations, etc.) and provide appropriate services (e.g., notifications of empty seats, etc.) to customers based on real-time information.

[0045] To this end, the store management system (100) performs various functions such as image calibration, computer vision algorithms, congestion calculation, heat map generation, and region of interest complexity calculation. The functions of the store management system (100) will be explained in detail later.

[0046] In this environment, the structure of a store management system (100) that calculates the level of congestion within the store (10), determines whether there are unoccupied seats, checks whether there are dishes that need to be cleaned, and collects store operation data based on images collected by each camera (200) is described with reference to FIG. 2.

[0047] Figure 2 is a structural diagram of a store management system according to an embodiment of the present invention.

[0048] As illustrated in FIG. 2, the store management system (100) includes a video analysis unit (110), a seat occupancy determination unit (120), a congestion calculation unit (130), a store operation data collection unit (140), and a database (150).

[0049] The image analysis unit (110) interacts with the plurality of cameras (200) through a closed network to receive images from the plurality of cameras (200). Since there are various ways for the image analysis unit (110) to interact with the plurality of cameras (200), the embodiment of the present invention is not limited to any one method.

[0050] The image analysis unit (110) receives images from the linked cameras (200). At this time, the image analysis unit (110) is described as receiving images from multiple cameras (200) asynchronously by applying multithreading, but it is not necessarily limited to this. Here, multithreading is a known technology, so a detailed description is omitted in the embodiment of the present invention.

[0051] The video analysis unit (110) receives initial images asynchronously received from the camera (200) when the store management system (100) is first started. At this time, the initial images refer to images collected by the camera (200) immediately after the store management system (100) is first started.

[0052] The store management system (100) processes an initial image and detects a perspective transformation matrix. Here, the perspective transformation matrix refers to a matrix for projecting the shooting range captured by each camera (200) onto a drawing image corresponding to the store (10).

[0053] The image analysis unit (110) detects a perspective transformation matrix to convert a rectangular image into an arbitrary polygonal square shape corresponding to the drawing image of the store (10), so that multiple images taken from multiple angles are projected onto the drawing image corresponding to the store (10). At this time, since there are various methods for the image analysis unit (110) to detect the perspective transformation matrix from the images, the embodiment of the present invention is not limited to any one method.

[0054] At this time, at least four matching points are required to derive the transformation matrix. As a criterion for point matching, anything is possible as long as the same location can be identified in the drawing and the image collected from the camera (200). Accordingly, in the embodiment of the present invention, the image analysis unit (110) is described by using a table location as an example to automate the process and ensure accuracy.

[0055] To this end, the image analysis unit (110) detects the location of the table from the initial image. Then, the image analysis unit (110) identifies the location of the table that is pre-registered in the drawing image corresponding to the store (10) and aligns it by comparing it with the table location detected in the image. Through this, the image analysis unit (110) can derive the location where the drawing and the image intersect, as well as the scale, rotation, and perspective changes.

[0056] At this time, in order to respond to the changed seats even if the positions of the seats arranged in the store (10) change dynamically during business hours, the image analysis unit (110) can periodically perform the process of calculating the projection matrix between the drawing of the store (10) and the image. If the perspective transformation matrix is ​​missing or needs to be modified during the periodic execution process, the image analysis unit (110) can again perform the work of setting matching points and deriving the perspective transformation matrix.

[0057] For example, every day before the store opens, all tables are detected in each CCTV video and the positions of the tables are projected onto a drawing using a perspective transformation matrix derived in advance. Excluding tables located in masked areas among the projected tables, the table position information of all videos is merged to update the drawing image of the store (10) to be used in subsequent processes.

[0058] The image analysis unit (110) masks the shooting area captured by each camera (200) onto the drawing image in order to remove overlapping areas between multiple images. Here, the drawing image is an image corresponding to the store (10), and is described using an example that is pre-stored in the database (150).

[0059] Additionally, when the video analysis unit (110) receives a video, it can mask (e.g., blur, replace, mosaic, etc.) the face areas of people in the video so that customers or employees cannot be distinguished in the received video. After masking the face areas, the video analysis unit (110) generates a de-identified video. This is to protect the personal information of customers appearing in the video.

[0060] Additionally, the image analysis unit (110) may mask the entire received image to generate a de-identified image. The image analysis unit (110) generates the image as a de-identified image in various ways, and the embodiment of the present invention is not limited to any one method, and various methods capable of protecting personal information may be used.

[0061] The video analysis unit (110) detects visiting customers in the video before generating a de-identified video. To this end, the video analysis unit (110) is described as including an artificial intelligence model trained with training data to detect people in the video and to calculate the number of customers visiting the store (10) in real time or to sum them up in batches.

[0062] The seat occupancy determination unit (120) extracts objects, namely tables and chairs, within a location pre-set as a region of interest in the de-identified image generated by the image analysis unit (110). Then, based on the state of the tables and chairs, it determines whether the corresponding seat is occupied.

[0063] To this end, the seat occupancy determination unit (120) clusters markers for each visiting customer to each seat based on the coordinates of each seat and the visiting customer detected in the de-identified image. Through this, the seat occupancy determination unit (120) matches a seat to each visiting customer. At this time, the seat occupancy determination unit (120) is described as an example of matching the visiting customer to an unoccupied seat among the seats closest to the visiting customer's location.

[0064] The seat occupancy determination unit (120) determines that a seat not matched with a visiting customer is in an unoccupied state. The seat occupancy determination unit (120) provides occupancy / unoccupied result information to the POS terminal (300) in the store (10), which has identification information already mapped to the unoccupied seat.

[0065] The congestion calculation unit (130) extracts customers from the de-identified image generated by the image analysis unit (110). Then, the congestion calculation unit (130) checks the number of coordinates of the extracted customers and calculates the congestion level by calculating the ratio of the maximum number of people set in the store (10) to the number of current customers.

[0066] In an embodiment of the present invention, congestion is described as being classified into four stages (e.g., smooth, normal, congested, very congested, etc.), but it is not necessarily limited to this. For example, in an embodiment of the present invention, the congestion calculation unit (130) is described as being classified as smooth if the ratio of the number of maximum people to the number of current coordinates is 100% or less, normal if the ratio is 100% to 130%, congested if the ratio is 130% to 150%, and very congested if the ratio is 150% or more.

[0067] Additionally, the congestion calculation unit (130) performs a perspective transformation on the coordinates of the table detected to be identical to the customer's coordinates, converting them into coordinates on the drawing. Then, the congestion calculation unit (130) visualizes the table in the form of a box at the corresponding location on the drawing. After that, the congestion calculation unit (130) accumulates the customer coordinates transformed into perspective on the drawing in fixed time units (e.g., in 'seconds') and displays them in the form of a heatmap.

[0068] Additionally, the congestion calculation unit (130) periodically performs complexity calculations to check the contamination level of areas of interest, such as dish return areas, self-service bars, and unoccupied seats.

[0069] That is, the congestion calculation unit (130) detects the location of a person within the de-identified image. As a method for the congestion calculation unit (130) to detect a person in the de-identified image, additional processing steps may be added, such as using an object detection model that has been trained to label only the area where a person is located within the image, or undergoing a re-verification process through comparison with previous frames after using the object detection model.

[0070] In an embodiment of the present invention, the result detected by the congestion calculation unit (130) is expressed as the coordinates of a rectangular box in an image, and the central coordinates of the box in the detection result are projected onto a drawing as an example. If the coordinates after projection are an area excluded by masking, they are not reflected in the detection result, and the coordinates outside the excluded area are recorded as the detection result.

[0071] The store operation data collection unit (140) obtains the number of customers who visited the store as store operation data based on customer characteristics obtained from the video analysis unit (110). In the embodiment of the present invention, the store operation data collection unit (140) is described as counting the number of visiting customers, but additional information such as the age group or gender of the visiting customers and the time of day they mainly visited can be obtained as store operation data.

[0072] The store operation data collection unit (140) provides the acquired store operation data to the store manager through the POS terminal (300), thereby enabling the provision of appropriate services to customers visiting the store (10) or the adjustment of marketing strategies.

[0073] The operation method of such a store management system (100) is explained with reference to FIG. 3.

[0074] FIG. 3 is a flowchart illustrating the operation of a store management system according to an embodiment of the present invention.

[0075] As illustrated in FIG. 3, the store management system (100) is linked to a plurality of cameras (200) already installed in the store (10) in a closed network (S100). Since the store management system (100) can be linked to a plurality of cameras (200) using known methods, the embodiment of the present invention is not limited to any one method.

[0076] The store management system (100) receives images captured by each camera (200) after linking with each camera (200) and derives a perspective transformation matrix (S110). To do this, the store management system (100) derives a perspective transformation matrix based on images captured by each camera (200) and drawings of the store (10) stored in the database (150).

[0077] That is, reference points at specific locations on the drawing and points corresponding to the reference points in the image are marked, and then a perspective transformation matrix is ​​derived. In an embodiment of the present invention, a store management system (100) detects the location of a table in an initial image and identifies the location of a table that is pre-registered in a drawing image corresponding to the store (10).

[0078] Next, the store management system (100) compares and aligns the table positions detected in the image to derive the position where the drawing and the image intersect, as well as the scale, rotation, and perspective changes, thereby deriving a single perspective transformation matrix. In this case, the embodiment of the present invention describes deriving the perspective transformation matrix once from an initial image collected immediately before operating the store (10), but the perspective transformation matrix may also be derived periodically as needed while operating the store (10) (e.g., changing the perspective transformation matrix due to changes in table positions).

[0079] After deriving the perspective transformation matrix, the store management system (100) masks the shooting area captured by each camera (200) on the drawing (S120). This may be such that some of the shooting areas captured by each camera (200) are set as shooting areas that are captured by at least two or more cameras (200) in duplicate.

[0080] Accordingly, in an embodiment of the present invention, the shooting area captured by each camera (200) is masked in the drawing so that the perspective conversion area of ​​each camera (200) does not overlap. And, if there is a part of the masked area that overlaps with the plurality of cameras (200), the store management system (100) can set one of the plurality of cameras (200) to shoot the corresponding area.

[0081] At this time, the store management system (100) may set the number of people or seats that can be accommodated in the store (10), or set an area of ​​interest for calculating complexity (e.g., a dish return area, a table, a self-service bar, etc.). In an embodiment of the present invention, the number of people or seats that can be accommodated is described as being input by a manager who manages the store (10). Additionally, the store management system (100) may set an area of ​​interest for which complexity can be calculated using a store management service, such as a camera (200) that photographs the entrance door of the store (10), a camera (200) that photographs the area around the display stand, and a camera (200) that photographs an area where seats are densely packed.

[0082] The store management system (100) masks the shooting area of ​​each camera (200) on a drawing, receives the image captured by each camera (200), and generates a non-identification image with the area of ​​interest masked (S130). To this end, the store management system (100) includes a non-identification module and generates a non-identification image by masking the face area so that the faces of employees or visiting customers who have visited the store (10) in the image are not identified. Since there are various methods for the store management system (100) to mask the face area of ​​visiting customers or employees in the image, the embodiment of the present invention is not limited to any one method.

[0083] The store management system (100) uses the de-identified image generated in step S130 to check whether a seat is occupied, calculate the level of congestion within the store, or collect store operation data (S140). Here, the service provided by the store management system (100) using the de-identified image in step S140 is described with reference to FIGS. 4 to 7.

[0084] FIGS. 4 to 7 are exemplary diagrams of services provided using de-identified images according to embodiments of the present invention.

[0085] First, as illustrated in FIG. 4, the store management system (100) masks the images on a drawing mapped to the store (10) so that the areas monitored based on the images captured by each camera (200) do not overlap. To do this, specific points are extracted from the images captured by a specific camera (200) and masked on the drawing of the store (10) shown in FIG. 4 (a), as shown in FIG. 4 (b).

[0086] In an embodiment of the present invention, when the store management system (100) is first operated, specific points (e.g., table locations, etc.) extracted from an image are mapped onto a drawing, and the shooting area handled by the corresponding camera (200) is masked onto the drawing of the store (10) using the mapped specific points. However, a new camera (200) may be added, or the shooting area handled by each camera (200) may be masked onto the drawing of the store (10) at a preset interval.

[0087] To this end, the store management system (100) detects the location of a table in an image by utilizing an artificial intelligence model, such as an object detection model or an object separation model. Then, the store management system (100) can derive the location where the drawing and the image intersect, as well as the scale, rotation, and perspective change, by comparing and aligning the location (T) of the table pre-registered in the drawing of the store (10) with the location of the table detected in the image. Based on the location, scale, rotation, and perspective change obtained in this way, the store management system (100) can automatically or semi-automatically derive a perspective transformation matrix.

[0088] And, the store management system (100) de-identifies areas where personal information is likely to be exposed, such as the faces of visiting customers or employees, in the video captured by each camera (200) through a de-identification module. Alternatively, the store management system (100) can de-identify the video itself through a de-identification module as shown in FIG. 5 to generate a de-identified video.

[0089] Additionally, the store management system (100) can separately collect and store only the de-identified images to be used for non-real-time services among the images from the camera (200) collected in real time. Then, when there is spare computing resources after the business hours of the store (10) have ended, the images collected for non-real-time services are processed to generate store operation data.

[0090] Next, the store management system (100) utilizes a perspective transformation matrix, as shown in FIG. 6, to project the detected visiting customers and seat locations within the image onto the drawing of the store (10) as coordinates.

[0091] That is, the store management system (100) represents the coordinates of a rectangular box (①) in an image as a customer or seat location as shown in FIG. 6 (a), and projects the center coordinates of the box in the detection result onto a drawing as shown in FIG. 6 (b). After projection, if the coordinates are in an area excluded by masking, they are not reflected in the detection result, and the coordinates outside the excluded area are recorded as the detection result.

[0092] In this way, the store management system (100) accumulates the locations of people detected over a certain frame or time period, and the accumulated area is considered to be an area occupied by customers.

[0093] If the area occupied by the customer is near an area of ​​interest, such as a table or a dish return area, the store management system (100) updates the occupancy status of that area. Conversely, if the area of ​​interest previously occupied by the customer disappears, the system also updates it so that the store management system (100) notifies the POS terminal (300) of the store (10) of the current occupancy status of the areas of interest.

[0094] At this time, the store management system (100) represents an area of ​​interest that is not occupied and is available as shown in (c) of FIG. 6 using a first display means (②), and removes the first display means (②) from an area of ​​interest that is already in use and displays it using a second display means (③).

[0095] Additionally, the store management system (100) can count the number of coordinates of customers projected onto each drawing to derive the current number of people per floor, and calculate the ratio of the current number of people to the predefined maximum number of people per floor to record the congestion level. Then, the store management system (100) accumulates the locations of visiting customers that have been transformed in perspective over multiple frames and visualizes the accumulated amount as a heatmap on the drawing image.

[0096] At this time, the store management system (100) can determine whether the people extracted from the store (10) are visiting customers or store employees. To do this, the store management system (100) uses a previously trained object extraction model to explain, for example, that among the objects extracted as people, store employees are not accumulated as current personnel.

[0097] Additionally, the store management system (100) can calculate the complexity of the region of interest using an object partitioning algorithm.

[0098] To explain using a display stand in a store (10) as an example, the store management system (100) extracts unique characteristics of the image before segmenting the objects in the image. First, the store management system (100) extracts edge information of objects within a region of interest extracted from an initial image using an edge detection algorithm (e.g., Sobel-filter or Canny-edge, etc.). The edge information extracted in this way is added as an input to an object segmentation model as an image feature map and is used to train and infer the object segmentation model.

[0099] And, the store management system (100) assigns a label to each of the multiple divided objects constituting the shelf image using an object division algorithm. For example, an empty dish return rack and the result of executing it with an object division algorithm are as shown in FIG. 7 (a).

[0100] The tabletop and beverage dispenser of the dish return rack, detected when there are no dishes on it, are partition objects that should not be used for actual complexity measurement. Therefore, the partition results measured in the empty state without dishes are stored as the reference set.

[0101] Meanwhile, as shown in Fig. 7(b), when the dish return rack is filled with dishes such as plates and cups, an object segmentation algorithm is applied, not only cups and trays but also unnecessary background segmentation objects such as table tops are detected. At this time, the area is compared with the previously stored reference segmentation area, and only objects that can affect the actual complexity, such as plates or cups placed on the dish return rack, are separated and stored.

[0102] At this time, the store management system (100) separates and stores only the objects that can affect the actual complexity, such as plates or cups placed on a shelf, as illustrated in FIG. 7 (b), by comparing the area with a previously stored reference division area. The store management system (100) uses the result of an IoU (Intersection over Union) operation to compare each division object with the reference division area. The IoU operation is as follows: Equation 1.

[0103]

[0104] The store management system (100) outputs the total number of divided labels and the sum of the pixel area values ​​of those labels as the final object division result. The store management system (100) calculates the complexity using the total number of divided labels output and the pixel area values ​​of the corresponding labels. If the calculated complexity is higher than a preset threshold, the store management system (100) sends a notification message to a terminal, such as a POS terminal provided in the store, indicating that the complexity of the dish return counter is currently high.

[0105] At this time, the calculation formula for the store management system (100) to calculate the complexity is as shown in Table 1 below.

[0106] roi_region_interest = pixel area of ​​the region of interest within the image term1_base = maximum number of detectable objects (value obtained by dividing region_of_interest by the average number of pixels of a single detected object (=average area of ​​one unit)) number of segments = number of detected objects

[0107] Here, α is the weight value of term1, which determines the extent to which the "count" value is reflected in the complexity calculation. And, β is the weight value of term2, which determines the extent to which the "area" value is reflected in the complexity calculation. Based on experimental results, the weight values ​​are explained using an example where α is set to 2.5 and β to 1.5. This calculation formula allows the manager to set different weight values ​​according to the characteristics of the store, thereby enabling the complexity to be calculated with optimized values ​​for each store.

[0108] roi_region_ineterest corresponds to a region of interest set as the area for calculating complexity in a display stand or a space for storing tableware. The store management system (100) determines how many tableware items are filled within the area and transmits a value corresponding to the complexity to a POS terminal (300) managing the store (10).

[0109] To explain using the example illustrated in FIG. 7(c) and FIG. 7(d), term1_base is the value obtained by dividing the region_of_interest, which is the upper area of ​​the dish return rack represented by the third display means (④) in FIG. 7(c), by the pixel area of ​​one glass. That is, the number of glasses required to fill the dish return rack becomes the term1_base value.

[0110] region_of_interest is 900, average area of ​​one unit is 20, term1_base Let's assume there are 45 people.

[0111] And, assuming that the total number of detected segments (number_of_segments) in Fig. 7 (c) is 25, term1 becomes α * (45+25) / 45 = 1.56α.

[0112] Meanwhile, term2, which is a value related to area, is the area of ​​the currently filled portion relative to the total area of ​​the dish return area. So, when region_of_interest is 900 and segmented_area is 700, term2 becomes β * (900+700) / 900 = 1.78β.

[0113] Since deriving the value for clearing dishes from the actual area of ​​interest using a mathematical model is a very complex process, the embodiments of the present invention utilize multiple store data to use a transcendental function including a hyperbolic tangent.

[0114] And, the store management system (100) can set a threshold for when to clear dishes according to the level of congestion in the store.

[0115] For example, when there are few customers, such as during weekday daytime or evening hours, the rate at which dishes are refilled at the dish return counter is slow and the count is low. Therefore, if the threshold is set to 60% or 70% during these times, the alarm may not sound for a long time, which can affect store cleanliness. Accordingly, in situations with few people, the threshold is lowered to ensure that the store is cleaned periodically.

[0116] On the other hand, during peak hours when there are many customers, the speed at which dishes are filled in the area of ​​interest is fast and the complexity level is high. Therefore, the store management system (100) can send a complexity alarm by raising the threshold so that the staff can clean at once when it is most needed and not too much.

[0117] FIG. 8 is an exemplary diagram of a store management service provided according to an embodiment of the present invention.

[0118] As illustrated in FIG. 8, when the complexity calculated by the store management system (100) is higher than a certain threshold, the store management system (100) sends a notification message (⑤) to a terminal, such as a POS terminal installed in the store (10), indicating that the complexity of the dish return counter is currently high.

[0119] The store management system (100) transmits the congestion level derived through video analysis and the visualized image to the POS terminal so that store employees can view them. Through this, by transmitting real-time image logs regarding the contamination level of the dish return area or the contamination level of another area of ​​interest, the store employees are enabled to respond immediately.

[0120] FIG. 9 is a hardware configuration diagram of a computing device according to an embodiment of the present invention.

[0121] As illustrated in FIG. 9, a store management system (100) operated by at least one processor may be implemented as a computing system (400) and executes a program containing instructions described to execute the operation of the present invention in the computing system (400). The program may be stored in a computer-readable storage medium and may be distributed.

[0122] The hardware of the computing system (400) may include at least one processor (410), memory (420), storage (430), and communication interface (440), and may be connected via a bus. In addition, hardware such as input devices and output devices may be included. The computing system (400) may be equipped with various software, including an operating system capable of running programs.

[0123] The processor (410) is a device that controls the operation of the computing system (400) and may be a various type of processor that processes instructions included in a program, for example, a CPU (Central Processing Unit), MPU (Micro Processor Unit), MCU (Micro Controller Unit), GPU (Graphic Processing Unit), etc.

[0124] The memory (420) loads the corresponding program so that instructions described for executing the operation of the present invention are processed by the processor (410). The memory (420) may be, for example, ROM (read only memory), RAM (random access memory), etc. The storage (430) stores various data, programs, etc. required to execute the operation of the present invention. The communication interface (440) may be a wired / wireless communication module.

[0125] Although embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims also fall within the scope of the present invention.

Claims

Claim 1 A method of operation of a store management system operated by at least one processor, comprising: receiving initial images from a plurality of cameras installed in a store and, based on the initial images, masking the shooting ranges of the plurality of cameras on a drawing image of the store so that the shooting ranges of the plurality of cameras do not overlap; setting a region for calculating the complexity of the store and a region for measuring the contamination level of the store, respectively, within the shooting ranges captured by the plurality of cameras according to the type of store management service mapped to each of the plurality of cameras; and generating a de-identified image by masking an image including the region of interest captured by the plurality of cameras, and providing the store management service based on the de-identified image. Claim 2 A method of operation according to claim 1, wherein the step of masking the shooting range comprises: a step of identifying reference points set in the drawing image; a step of extracting a plurality of points corresponding to the reference points from the initial images; and a step of deriving a perspective transformation matrix so that the initial images are projected onto the drawing image by mapping the plurality of points to the reference points. Claim 3 In paragraph 2, the step of masking the shooting range comprises: a step of masking the shooting area captured by the plurality of cameras on the drawing image; and a step of setting one of the plurality of cameras to capture the overlapping shooting areas when a portion of the shooting areas captured by the plurality of cameras overlap. Claim 4 A method of operation according to claim 1, wherein the step of setting the area of ​​interest further includes the step of setting the area of ​​an object including a dish return rack or a plurality of tables installed in the store as the area of ​​interest for measuring the contamination level. Claim 5 A method of operation according to claim 1, wherein the step of providing the store management service comprises: inputting an image captured by the plurality of cameras into a de-identification module trained to extract a face region from an image to extract a face region of the store customer or store employee from the image; and de-identifying the face region using the de-identification module to generate the de-identified image. Claim 6 A method of operation according to claim 1, wherein the step of providing the store management service comprises: extracting the location of a store customer and the location of a plurality of seats placed in the store from the de-identified image; mapping the location of the customer to the seat closest to the location of the customer among the plurality of seats; and providing information on a non-preempted seat among the plurality of seats in the store that has not been preempted by the store customer to a management terminal installed in the store. Claim 7 A method of operation according to claim 1, wherein the step of providing the store management service comprises: a step of extracting customers who have visited the store from the image and confirming the number of customers who have visited the store; and a step of calculating the congestion level of the store based on the number of customers and the number of maximum personnel previously set for the store, and providing congestion level information corresponding to the calculated congestion level. Claim 8 A method of operation according to claim 1, wherein the step of providing the store management service includes the step of obtaining information on the gender, age group, and time of visit of customers who visited the store in the image as store operation data. Claim 9 In claim 4, the step of providing the store management service comprises: a step of extracting and dividing the object from an initial image of a region of interest set to measure the contamination level of the store and storing it as a reference dividing region; a step of extracting and dividing the object from images of the regions of interest, and calculating the sum of the number of labels and the pixel area values ​​of the labels based on the reference dividing region; a step of calculating the complexity of each of the regions of interest based on the number of labels and the sum of the pixel area values; and a step of transmitting the contamination level of the region of interest to a management terminal within the store if the calculated complexity is higher than a preset threshold. Claim 10 A store management system comprising an interface linked with a plurality of cameras already installed in a store and a processor, wherein the processor receives initial images from a plurality of cameras installed in a store, masks the shooting ranges of the plurality of cameras on a floor image of the store based on the initial images so that the shooting ranges of the plurality of cameras do not overlap, sets a region of interest in the shooting ranges captured by the plurality of cameras according to the type of store management service mapped to each of the plurality of cameras, receives images captured by the plurality of cameras, generates a de-identified image by masking the images, and provides the store management service based on the de-identified image. Claim 11 A store management system according to claim 10, wherein the processor identifies reference points set in the drawing image, extracts a plurality of points corresponding to the reference points from the initial images, and then marks the plurality of points on the reference points to derive a perspective transformation matrix. Claim 12 A store management system according to claim 11, wherein the processor masks the shooting area captured by the plurality of cameras on the drawing image, and if a portion of the shooting areas captured by the plurality of cameras overlaps, one of the plurality of cameras is configured to capture the overlapping shooting areas. Claim 13 A store management system according to claim 12, wherein the processor sets an area for calculating the complexity of the store as a region of interest, and sets an area of ​​an object including a dish return rack or a plurality of tables installed in the store as a region of interest to measure the contamination level of the store. Claim 14 A store management system according to claim 13, wherein the processor inputs images captured by the plurality of cameras into a de-identification module trained to extract face regions from an image, extracts face regions of the store customers or store employees from the image, and de-identifies the face regions using the de-identification module to generate the de-identified image. Claim 15 A store management system according to claim 14, wherein the processor extracts the location of a store customer and the locations of a plurality of seats placed in the store from the de-identified image, maps the location of the customer to the seat closest to the location of the customer among the plurality of seats, and provides information on non-preempted seats among the plurality of seats in the store that are not preempted by the store customer to a management terminal installed in the store. Claim 16 In claim 15, the processor extracts customers who have visited the store from the image, identifies the number of customers who have visited the store, calculates the congestion level of the store based on the number of customers and the number of maximum personnel set for the store, and provides congestion level information corresponding to the calculated congestion level, in a store management system. Claim 17 In claim 16, the processor extracts and divides the object from an initial image taken of a region of interest set to measure the contamination level of the store and stores it as a reference division area; extracts and divides the object from images taken of the regions of interest; calculates the number of labels that divide the object based on the reference division area and the sum of the pixel area values ​​of the labels; calculates the complexity of each of the regions of interest based on the number of labels and the sum of the pixel area values; and if the calculated complexity is higher than a preset threshold, transmits the contamination level for the region of interest to a management terminal within the store.

Images