Visitor group grasping system

Abstract

To realize a visitor group grasping system which, without roughly limiting a place for use in classification of visitors, can rapidly classify the visitors into a single visitor and a group of visitors and can rapidly estimate the number of the group.SOLUTION: A visitor group grasping system 10 according to the present invention has at least one camera 12, 13 for capturing a predetermined place, and a visitor group grasping apparatus 20 for analyzing an image captured by the at least one camera 12, 13 to specify at least one visitor, estimating a distance between the specified visitor and other visitors based on the captured image, classifying the specified visitor into any one of a single visitor and a group visitor, and estimating the number of the group to which the specified visitor belongs if the specified visitor is a group visitor.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a visitor group identification system that includes at least one camera that takes images of a specified location, analyzes the images taken by the camera to identify at least one visitor, classifies the identified visitor as either a solo visitor or a group visitor, and estimates the number of people in the group in the case of a group visitor. [Background technology]

[0002] There is an increasing trend to utilize images captured by cameras, such as surveillance cameras, for purposes other than crime prevention, such as analyzing people flow and measuring congestion, for facility operations and services. For example, regarding the use of commercial facilities with multiple stores, such as department stores and shopping centers, in marketing surveys, it is sometimes important to know whether visitors to the facility are "solo visitors" or "group visitors, such as couples or families." Marketing strategies change depending on whether there are more solo visitors or more group visitors, so a means of quickly obtaining such an analysis is desirable.

[0003] In addition, if it is possible to estimate the number of people in a group from the same camera image, it will enable more effective marketing, so it is also desirable to use it as a means of understanding the number of people in a group.

[0004] Patent Document 1 describes an information management device that classifies customers who visit a store into "individual customers" if there is only one person in the "individually seated table" area from video data from a surveillance camera, and into "group customers" if there are multiple people. The information management device also describes that in the area of ​​adjacent tables, the classification of individual customers and group customers is performed based on the direction of their faces. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-96695 Summary of the Invention [Problem to be solved by the invention]

[0006] The system described in Patent Document 1 classifies customers into individual customers and group customers based on camera image data, but the classification is limited to the specific state in which customers are seated in chairs paired with tables. For this reason, customer classification can only be performed in very limited locations, such as restaurants, where images of customers seated in chairs can be captured.

[0007] To provide a visitor group grasping system that can quickly classify visitors as either solo visitors or group visitors without greatly limiting the places where visitors are classified, and can also quickly estimate the number of people in a group if the visitors are in a group. [Means for solving the problem]

[0008] One example of an embodiment of the visitor group identification system is a visitor group identification system that includes at least one camera that photographs a predetermined location, and a visitor group identification device that analyzes images photographed by the at least one camera to identify at least one visitor, estimates the distance between the identified visitor and other visitors from the photographed image, classifies the identified visitor as either a solo visitor or a group visitor from the estimated distance, and, in the case of a group visitor, estimates the number of people in the group. [Effects of the Invention]

[0009] According to the visitor group identification system of the present invention, visitors can be quickly classified as either solo visitors or group visitors, and the number of people in a group can be quickly estimated, without greatly limiting the places where visitors are classified. [Brief explanation of the drawings]

[0010] [Figure 1]1 is a block diagram showing the configuration of a visitor group identification system according to an embodiment; [Figure 2] FIG. 1 is a diagram showing an example of a photographed image of a plurality of visitors who have come to a plaza in a shopping center, which is a predetermined location. [Figure 3] FIG. 3 is an enlarged view of part A in FIG. 2, showing an example of a method for measuring the distance between a plurality of visitors. [Figure 4] FIG. 10 is a diagram showing another example of a method for measuring the distance between multiple attendees. [Figure 5] FIG. 10 is a diagram showing a table of the measurement results of the distances between a plurality of visitors measured using the frames of one captured image. [Figure 6] 10 is a flowchart showing a method for determining the group of a specific attendee using the attendee group identification system of the embodiment. [Figure 7] 10 is a flowchart showing a method for determining the group of a specific attendee using a visitor group identification system according to another embodiment. [Figure 8] This figure shows an example of changes in the estimated positions of visitors and robots on a floor plan used to estimate the distance between multiple visitors by analyzing images taken by a camera mounted on a moving robot using a visitor group understanding system in another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, with reference to the drawings, a detailed description will be given of an embodiment of a visitor group tracking system according to the present invention. Each embodiment described below is merely an example, and the present invention is not limited to the following embodiments. Furthermore, the present invention also includes forms that are formed by selectively combining multiple embodiments and modified examples described below.

[0012] FIG. 1 is a block diagram showing the configuration of a visitor group identification system 10, which is an example of an embodiment. The visitor group identification system 10 includes a camera 12 and a visitor group identification device 20. The visitor group identification system 10 analyzes images of visitors who have visited a plaza in a shopping center, which is a commercial facility, to identify at least one visitor, estimate whether the identified visitor is visiting alone or in a group, and, if a visitor is in a group, estimate the number of people in the group. The plaza is often used for meetings and as an event venue, and corresponds to a predetermined location.

[0013] The place where visitors are classified is not limited to indoor facilities of a shopping center, but may be an outdoor facility, or may be an outdoor facility such as inside a department store or on a rooftop. The visitor group identification system may also classify visitors who have visited a single store.

[0014] Fig. 2 is a diagram showing an example of a photographed image of a plurality of visitors 1 who have visited a shopping center plaza 100. The photographed image in Fig. 2 is an image with a timestamp in which the photographed date and time are stored in association with the photographed image.

[0015] As shown in Figure 2, various visitors 1 visit the plaza 100, and if the visitor group identification system 10 of the embodiment can determine whether the majority of the multiple visitors 1 are visiting alone or in groups, the shopping center can use the results to create an appropriate marketing strategy, making it easier to provide more efficient operations and services to the visitors 1.

[0016] The following provides a detailed description of the configuration of the visitor group tracking system 10. As described above, the visitor group tracking system 10 includes the camera 12 and the visitor group tracking device 20. The visitor group tracking system 10 further includes a display device 30, a mobile information terminal 31, and an identification information acquisition device 32.

[0017] The camera 12 is attached to a predetermined position from which it can photograph the plaza 100, for example, a position on a building wall, pillar, ceiling, or the like from which it can photograph the visitor 1 from diagonally above, and photographs the plaza 100 and the visitor 1 who has entered the plaza 100. The camera 12 is used as a surveillance camera, and the images photographed by the camera 12 may be transmitted to a display device 30 (described later) and displayed on the display device 30. This may enable a monitor near the display device 30 to monitor the presence or absence of suspicious persons in the plaza 100. The camera 12 photographs continuously, at predetermined time intervals, or at predetermined timings during the shopping center's business hours, and transmits the photographed images to the visitor group understanding device 20 via wired or wireless communication. Below, we will explain the case where camera 12 photographs plaza 100 and visitors 1 who have arrived at plaza 100, and classifies visitors 1 who have arrived at plaza 100.However, the specified location for classifying visitors 1 is not limited to plaza 100, and can be various locations such as other waiting areas, corridors outside the store, inside the store, etc.

[0018] The camera 12 may be any camera capable of capturing images of predetermined body parts, such as the heads, of multiple visitors 1 and acquiring the two-dimensional positions of the heads in the captured images. The camera 12 may also be a stereo camera capable of three-dimensional measurement or a TOF (Time-of-Flight) camera. Using a camera 12 capable of three-dimensional measurement makes it possible to acquire the positions of predetermined body parts, such as the heads, of visitors 1 in three mutually perpendicular directions: X, Y, and Z. For example, the X axis may be the axis along the width of the plaza 100 (the left-right direction in FIG. 2), the Y axis may be the axis along the depth of the plaza 100, and the Z axis may be the axis along the height. The visitor group tracking system 10 may have multiple cameras 12 and 13 positioned at different locations, and the X, Y, and Z positions of predetermined body parts of the visitors 1 may be acquired from images captured by the multiple cameras 12 and 13 using the principle of triangulation. The images captured by the cameras 12 and 13 may be used to acquire the positions of multiple specific body parts of the visitor 1, such as the neck, groin, shoulders, waist, and joints. The positions of these multiple body parts can be used to estimate the visitor 1's skeleton.

[0019] The visitor group identification device 20 is configured, for example, by a personal computer (PC). The visitor group identification device 20 includes, for example, a processing unit such as a CPU; a storage unit 21 such as ROM, RAM, or external memory; an input unit consisting of a keyboard, mouse, touch panel, or other pointing device; and an output unit 22 having a communication control unit for outputting data externally via wired or wireless connection, an I / F, etc. The storage unit 21 stores a program for executing the visitor classification process, and data such as numerical values ​​obtained by executing the program. The visitor group identification device 20 may be configured on a virtual server of a cloud service via a communication line.

[0020] The visitor group identification device 20 also has a visitor identification unit 23, a distance estimation unit 24, and a visitor group identification unit 25. The visitor identification unit 23 analyzes images captured by one camera 12 or multiple cameras 12, 13 to identify at least one visitor 1. In this case, the visitor identification unit 23 identifies the visitor 1 using one or a combination of face recognition technology, skeletal detection technology, and human detection technology using bounding boxes.

[0021] For example, face recognition technology determines whether a facial feature image exists in a captured image. A facial feature image is, for example, an image showing feature points such as eyes, nose, mouth, and eyebrows. In face recognition technology, if it is determined that the recognition rate of the facial feature image in the captured image is equal to or greater than a predetermined value, it determines that a facial image exists. In face recognition technology, if a face is turned backwards and only the head on the opposite side of the face is visible, it may be determined that a facial image exists based on the feature points of the head.

[0022] The skeletal detection technology determines whether or not a captured image contains feature images that indicate feature points of a plurality of predetermined body parts, including the head, neck, base of limbs, shoulders, waist, and joints. The skeletal detection technology determines that a human image is present when it is determined that the recognition rate of the feature images of the predetermined body parts is equal to or greater than a predetermined value.

[0023] In bounding box-based human detection technology, an object in a captured image is surrounded by a rectangular bounding box that roughly touches the outer edge, and an evaluation score (score) is calculated for the human model of the object surrounded by the bounding box. If the calculated evaluation score is equal to or greater than a predetermined value, the object is detected as a human.

[0024] The distance estimation unit 24 estimates the distance between the at least one identified visitor 1 and other visitors 1 by calculation from the images captured by the cameras 12 and 13.

[0025] For example, in FIG. 2, facial recognition technology determines that the image of visitor 1's face or head is present in the area surrounded by multiple rectangular frames, and identification numbers (IDs) P01 to P11 are assigned near some of the rectangular frames. The following describes the case where the identified visitor 1 is the visitor 1 assigned P01. The visitor 1 may be identified arbitrarily by the visitor group identification device 20, or arbitrarily by the user using the input unit. After classifying one identified visitor 1 as a solo visitor or a group visitor, all visitors 1 included in the image may be identified one by one, and the identified visitors 1 may be classified in order.

[0026] Fig. 3 is an enlarged view of part A in Fig. 2, showing an example of a method for measuring the distances between multiple visitors 1. In the example shown in Fig. 3, for visitors 1 identified by P01, P02, and P03, the distance estimation unit calculates and estimates the distances between rectangular frames in the captured image: distance L1 between P01 and P02, distance L2 between P01 and P03, and distance L3 between P02 and P03.

[0027] The distance between multiple visitors 1 may also be measured using another method shown in FIG. 4. In FIG. 4, the distances between visitors 1 identified by P01, P02, and P03 are calculated as distances on a plan view from directly above. In this case, the two-dimensional positions of the visitors 1 identified by P01, P02, and P03 on the plan view are estimated using multiple cameras 12 and 13, or using a camera 12 capable of three-dimensional measurement. In FIG. 4, the left-right direction corresponds to the width direction of the plaza 100, and the up-down direction corresponds to the depth direction of the plaza 100. After estimating the two-dimensional positions of each visitor 1 in this manner, the distance estimation unit 24 calculates the distance L1a between P01 and P02, the distance L2a between P01 and P03, and the distance L3a between P02 and P03 as the distances between the visitors 1.

[0028] The above describes the case where the distances between P01 and P02, between P01 and P03, and between P02 and P03 are estimated, but the distance estimation unit 24 estimates from the captured image at least all of the distances between the visitor 1 of P01, who is at least one identified visitor 1, and multiple other visitors 1. In this case, calculation of the distance between the identified visitor 1 and other visitors 1 who are at a predetermined distance or more from the identified visitor 1 and who are clearly not in the same group may be omitted.

[0029] The visitor group identification unit 25 classifies the identified visitor 1 as either a solo visitor or a group visitor based on the distance estimated by the distance estimation unit 24, and if the identified visitor 1 is classified as a group visitor, estimates the number of people in the group to which the identified visitor 1 belongs. At this time, if the average value of the distance between the identified visitor 1 and other visitors 1 in multiple image frames is equal to or less than a predetermined value, it is estimated that the identified visitor 1 belongs to the same group. Specific examples of the predetermined value will be described later.

[0030] FIG. 5 shows a table of distance measurements between multiple visitors 1 measured using frames from a single captured image. Table 40 in FIG. 5 shows the estimated distance between two corresponding visitors 1, represented by the numerical values ​​at the intersections of the vertical columns P01, P02, . . . P11 and the horizontal columns P01, P02, . . . P11. The distance values ​​in FIG. 5 are in meters. For example, the distance between visitor 1 P01 and visitor 1 P02 is 1.0 m, and the distance between visitor 1 P01 and visitor 1 P11 is 2.5 m. The distance between visitor 1 P04 and visitor 1 P05 is 0.5 m, and the distance between visitor 1 P04 and visitor 1 P11 is 4.9 m. The data in table 40 in FIG. 5 is stored in the memory unit 21 of the visitor group understanding device 20.

[0031] At this time, as shown in Fig. 5, the visitor group understanding unit 25 calculates the ranking of the proximity between visitor 1 of P01 and other visitors 1 from the data stored in table 40, and also stores the ranking results in the memory unit 21. For example, in the example of Fig. 5, the visitor 1 closest to visitor 1 of P01 is P07, and the distance is 0.9 m, and the visitor 1 next closest to visitor 1 of P01 is P02, and the distance is 1.0 m.

[0032] Alternatively, the visitor group understanding device 20 may acquire N frames of consecutively captured images with timestamps, create a table similar to table 40 shown in Fig. 5 for each frame, and calculate the average value of the distance between visitor 1 P01 and other visitors 1 P02, P04, ... P11. In this case, the visitor group understanding unit 25 may calculate a ranking of the average value of the closeness of the distance between P01 and the other visitors 1 by analyzing the N frames of the captured images.

[0033] Furthermore, a predetermined value that is an upper limit of the distance required for people to be considered as belonging to the same group may be preset in the visitor group grasping unit 25. This predetermined value may be arbitrarily set by the user using the input unit.

[0034] In this case, the predetermined value is a distance threshold (parameter) defined to determine whether a particular visitor 1 (e.g., P01) and another visitor 1 (e.g., P07 or P02) belong to the same group. If the distance (or average distance) between visitors 1 is within the threshold, the visitors are determined to be in the same group. For example, if the average distance between P01 and P07 is 0.9 m, then if the predetermined value = 0.8 m, P01 and P07 are not in the same group, but if the predetermined value = 0.95 m, P01 and P07 are in the same group.

[0035] The predetermined value will be explained using a specific example. For example, in the example of FIG. 5, if the predetermined value, which is the upper limit of the distance closeness for estimating that visitors 1 are in the same group, is 0.8 m, table 40 in FIG. 5 shows that visitor 1 P01 does not have any other visitors 1 who are closer than that distance. Therefore, visitor 1 P01 is a solo visitor. On the other hand, if the predetermined value, which is the upper limit of the distance closeness for estimating that visitors are in the same group, is 0.95 m, table 40 in FIG. 5 shows that visitor 1 P01 and another visitor 1 who is closer than that distance are visitor 1 P07, who is 0.9 m away. In this case, visitor 1 P01 and visitor 1 P07 belong to the same group, and it can be seen that visitor 1 P01 is a group visitor. In this case, the number of people in the group to which visitor 1 P01 belongs is two.

[0036] Furthermore, if the above-mentioned predetermined value is set to 1.1 m, in table 40 of Figure 5, the other visitors 1 who are closer to visitor 1 P01 but are less than the distance are visitor 1 P07 and visitor 1 P02, who is 1.0 m away. In this case, it can be seen that visitor 1 P01 and visitors 1 P07 and P02 belong to the same group, and visitor 1 P01 is a group visitor. In this case, the number of people in the group to which visitor 1 P01 belongs is three.

[0037] Furthermore, if the above-mentioned predetermined value is set to 1.25 m, in table 40 of Figure 5, the other visitors 1 who are closer to visitor 1 P01 but are less than this distance are visitor 1 P07, visitor 1 P02, and visitor 1 P08, who is 1.2 m away. In this case, it can be seen that visitor 1 P01 belongs to the same group as visitors 1 P07, P02, and P08, and that visitor 1 P01 is a group visitor. In this case, the number of people in the group to which visitor 1 P01 belongs is four.

[0038] The display device 30 is connected to the visitor group identification device 20 and includes a liquid crystal display or the like. It displays the classification results for the identified visitor 1, such as P01, classified by the visitor group identification unit 25, or for all visitors 1 captured in the N frames of the captured image, i.e., whether they are individual visitors or group visitors. The display device 30 may display individual visitors 1 and group visitors 1 separately, and may further display visitors 1 belonging to one or more groups together. The display device 30 also displays the number of people in the group to which the identified visitor 1, such as P01, belongs. The display device 30 may also display the number of people in the group to which each visitor 1 belongs, for all visitors 1 captured in the N frames of the captured image. The display device 30 may be a large-scale digital signage device or a display monitor.

[0039] The connection between the display device 30 and the visitor group understanding device 20 is not limited to a wired connection, but may be a wireless connection such as Wi-Fi, or a connection via an external communication line such as LTE or 5G.

[0040] Like the display device 30, the mobile information terminal 31 is a terminal that displays information related to a group obtained from the visitor group understanding device 20. The mobile information terminal 31 has a display unit such as an LCD display unit, and can communicate with the visitor group understanding device 20 via a mobile phone line such as a mobile communication system, or a wireless communication line such as a wireless LAN, and may be configured, for example, as a mobile phone or tablet such as a smartphone owned by the user, or a PDA without a calling function.

[0041] The identification information acquisition device 32 is attached near a security gate installed at the entrance to the target location to be classified, such as the plaza 100 or a store. The security gate determines whether or not the visitor 1 is permitted to enter the target location based on the visitor 1's possessions, such as an ID tag, an ID card, a ticket with a two-dimensional code such as a QR code (registered trademark) or a barcode printed on it, or a mobile information terminal capable of displaying the two-dimensional code, and opens the gate if entry is permitted, or unlocks a locked door.

[0042] The identification information acquisition device 32 acquires the identification information of the visitor 1 by receiving the identification information on the visitor's belongings via short-range communication or by reading it with a laser scanner or the like. At this time, the visitor identification unit 23 of the visitor group understanding device 20 may be configured to identify the visitor 1 associated with the identification information acquired by the identification information acquisition device 32, and estimate the distance to other visitors 1 using only captured images that include the identified visitor 1 associated with the identification information. Examples of the identification information acquisition device 32 that may be used include a wireless tag system and a contactless IC card system.

[0043] In this case, the visitor group understanding device 20 only needs to analyze images of the visitor 1 associated with the identification information acquired by the identification information acquisition device 32, for example, visitor 1 P01, to estimate the distance to and classify the visitor 1. This reduces the amount of calculation required by the computer. Also, the analysis of the visitor 1 can begin as soon as the identification information is acquired by the identification information acquisition device 32.

[0044] 6 is a flowchart showing a method for performing group determination of a specific visitor 1 using the visitor group identification system 10 of the embodiment. In FIG. 6, the specific visitor 1 is designated as P01, and the flow of group determination for P01 will be described as an example. The processing shown in steps S10 to S18 in FIG. 6 is realized by the calculation processing unit of the visitor group identification device 20 executing a program stored in the storage unit 21.

[0045] First, in step S10, the visitor identification unit 23 of the visitor group understanding device 20 acquires N frames of photographed images with timestamps. N is a natural number equal to or greater than 1. Next, in step S11, 1 is assigned to a preset variable M. The variable M is used to repeat the process thereafter.

[0046] Next, in step S12, the visitor identification unit 23 detects people in the captured image of the Mth frame and assigns each detected person an identification number (ID) such as P01 as visitor 1. The same person is detected in the captured images of all N frames and assigned the same identification number (ID). At this time, person detection (detection of the same person) is performed using one or a combination of the above-mentioned face recognition technology, skeletal detection technology, and person detection technology using bounding boxes. At this time, people may be detected using a trained image recognition model that has previously learned human features.

[0047] Next, in step S13, the distance estimation unit 24 determines that a specific visitor 1 in the captured image of the Mth frame is the visitor 1 with identification number P01, and estimates the distance between the visitor 1 with identification number P01 and the other visitors 1, i.e., between P01 and P02, between P01 and P03, between P01 and P04, etc. At this time, the position of each visitor 1 on the plan view viewed from directly above shown in Figure 4 may be estimated, and the estimated positions may be used to measure the distance between P01 and the other visitors 1.

[0048] Next, the visitor group understanding device 20 determines whether the variable M is equal to the total number N of frames of the acquired photographed images (step S14). If the determination result of step S14 is affirmative (YES), the process proceeds to step S16. If the determination result of step S14 is negative (NO), the process proceeds to step S15. In step S15, M+1 is substituted for the variable M, the process returns to step S12, and the processes of steps S12 to S14 are repeated.

[0049] On the other hand, in step S16, since the distance between P1 and the other visitors 1 has already been measured for the total number N of frames acquired, the visitor group understanding unit 25 calculates, from the analysis of each frame, the average value of the closeness of the distance between P01 and each of the other visitors 1 and the ranking of that average value, as described in Figure 5.

[0050] Next, in step S17, the visitor group understanding unit 25 performs group determination based on whether the average value of the distance between P01 and each of the other visitors 1 is equal to or less than a predetermined value. That is, in step S17, if the average value of the distance between P01 and another visitor 1 is equal to or less than a predetermined value, it is determined for all of the other visitors 1 whether or not the two visitors 1 are estimated to belong to the same group. The predetermined value is determined in advance as a parameter.

[0051] Next, in step S18, the visitor group identification unit 25 classifies P01 as a solo visitor or a group visitor. Specifically, if it is estimated that P01 and at least one other visitor 1 belong to the same group, P01 is classified as a group visitor; otherwise, P01 is classified as a solo visitor. If P01 is classified as a group visitor, the number of people in the group to which P01 belongs is calculated. The classification results and calculation results are sent to and displayed on the display device 30 and mobile information terminal 31, and the process ends.

[0052] In the flowchart in Figure 6, P01 can be classified as a solo visitor or a group visitor, and it can be determined how many people are in the group P01 is in. If this analysis is performed on all visitors 1 who have been assigned identification numbers, such as P01, P02, P03, etc., it will be possible to estimate the group composition for all visitors 1 appearing in the image, and the logic of this flowchart can also be used in this way.

[0053] The visitor group identification system 10 described above estimates the distance between visitors 1 from the captured image without using the number of people in the table area or the orientation of the faces of the seated people, and can use this distance to classify the visitors 1 as either solo visitors or group visitors. This allows the visitors 1 to be quickly classified as solo visitors or group visitors without greatly limiting the locations where the visitors 1 are classified.

[0054] In addition, in the embodiment, if the identified visitor 1 is a group visitor, the number of people in the group to which the identified visitor 1 belongs is calculated. This makes it easier to provide more appropriate services to the visitor according to the tendency of the number of people in the visitor 1's group.

[0055] In the above embodiment, the visitor group identification device 20 can also estimate the height of the visitor 1 from the captured image. Furthermore, the visitor group identification device 20 can use face recognition technology to classify the captured image into gender (male / female) and age groups (for example, infants, elementary school students, junior high school students, high school students, university students, 20s, 30s, 40s, 50s, 60s and older).

[0056] Classification of men and women may be performed based on clothing, hairstyle, etc., or a trained image recognition model that has previously learned the characteristics of men and women may be used.

[0057] The visitor group identification device 20 can also use the above-mentioned gender classification and age group classification to classify the attributes of the group to which the visitor 1 belongs, such as whether the group is a family with children, a young couple or an elderly couple, or a group of friends of a certain age group. For example, it can distinguish between a young couple with one child and a group of three high school girls.

[0058] In addition, by using skeleton detection and size detection of objects recognized as people, it is possible to estimate the ratio of adults to children when a group of two or more people is estimated, for example, two adults and one child, or one adult and two children. Furthermore, when skeleton detection estimates that two people are holding hands and size detection of objects recognized as people estimates the sizes of the two people, it is possible to estimate that the group is a family of an adult and a child or younger, or that an adult is holding the hand of an elderly person.

[0059] Furthermore, if the visitor group grasping device 20 classifies the identified visitor 1 as a group visitor and determines from image analysis of the captured image that the identified visitor 1 and other visitors 1 are wearing the same type of clothing, for example, the uniforms of the same team, it can infer that the visitors 1 wearing the same type of clothing belong to the same group. This can further increase the accuracy of the classification of the visitors 1.

[0060] The visitor group tracking device 20 can also obtain the estimated head position of the visitor 1 and the estimated skeletal position below the estimated head position from the captured image. For example, using the above-described skeletal detection technology, it is possible to obtain the estimated head position and the position of at least a portion of the skeleton below the head, such as the skeletal positions of the upper body, such as the neck, shoulders, elbows, and waist. At this time, the visitor group tracking device 20 can estimate the foot position of the visitor 1 from the obtained estimated skeletal position, and can also estimate the distance between the identified visitor 1 and other visitors 1 from the estimated foot positions.

[0061] For example, it is possible to obtain at least a part of the skeleton, and further estimate the foot positions of visitor 1 using the size of the skeleton and a trained image recognition model that has been trained in advance.

[0062] As a result, even if the foot position of a visitor 1 is not captured in the captured image because the lower half of the visitor 1 is hidden by another visitor 1, the foot position of the visitor 1 can be estimated, and the distance between the feet of the visitors 1 can be estimated, so the distance between the visitors 1 can be estimated with greater accuracy. For example, even if there is a large height difference between the visitors 1, such as a parent and child, the distance between the visitors 1 can be estimated with greater accuracy.

[0063] 7 is a flowchart showing a method for performing group determination of a specific visitor 1 using the visitor group identification system 10 according to another embodiment. In this example, the visitor group identification device 20 estimates that a specific visitor 1 belongs to the same group if the distance between the identified visitor 1 and other visitors 1 is equal to or greater than a predetermined number of frames of continuously captured images and is equal to or less than a predetermined value.

[0064] The processing from step S20 to step S25 in FIG. 7 is the same as the processing from step S10 to step S15 in FIG.

[0065] In this example, if the determination in step S24 is affirmative (YES), the process proceeds to step S26. In step S26, a group determination is made based on whether the distance between P01, a specific visitor 1, and each of the other multiple visitors 1 is equal to or less than a predetermined value for a predetermined number of consecutive frames or more, based on an analysis of consecutive frames of images captured by at least one camera 12, 13. That is, in step S26, a determination is made for each of the other visitors 1 as to whether the two visitors 1 are estimated to belong to the same group based on the distance between P01 and the other visitors 1 being equal to or less than a predetermined value for a predetermined number of consecutive frames or more.

[0066] Next, in step S27, P01 is classified as a solo visitor or a group visitor. Specifically, if it is estimated that P01 and at least one other visitor 1 belong to the same group, P01 is classified as a group visitor; otherwise, P01 is classified as a solo visitor. If P01 is classified as a group visitor, the number of people in the group to which P01 belongs is calculated, and the classification and calculation results are sent to and displayed on the display device 30 and the mobile information terminal 31, after which the process ends.

[0067] In another example of the configuration configured as above and performing the classification process as above, if it is estimated that P01 and another visitor 1 are at a distance of a predetermined value or less for a predetermined period of time or more corresponding to a predetermined number of consecutive frames, it is estimated that the two belong to the same group, thereby making it possible to improve the accuracy of classification estimation. In this example, the other configurations and operations are the same as those of Figures 1 to 6.

[0068] In each of the above examples, we have described a case where the visitor group tracking system has both a display device 30 and a mobile information terminal 31 as destinations for transmission from the visitor group tracking device 20, but the visitor group tracking system may also be configured to have only one of the display device 30 and the mobile information terminal 31 as destinations for transmission from the visitor group tracking device 20.

[0069] Furthermore, although the above description has been given of the case where the cameras 12 and 13 are fixed to a building, the cameras may also be mounted on a moving vehicle. Figure 8 shows an example of changes in the estimated positions of the visitors 1 and the robot 42 on a floor plan used to estimate the distances between multiple visitors 1 by analyzing images captured by a camera (not shown) mounted on a moving robot 42 using the visitor group tracking system 10 according to another embodiment.

[0070] In this example, the visitor group tracking system 10 uses a camera mounted on a mobile robot 42. The robot 42 is a so-called service robot, which can move in any direction on the floor using a program pre-stored in the robot 42 or by remote communication with a main server, and provides services such as providing information using a display and audio output, security, and automatic cleaning.

[0071] For example, the robot 42 has rollers for turning and rollers for running on the bottom, and is capable of running autonomously or remotely on the floor surface and rotating 360 degrees or more around the vertical direction. Furthermore, the robot 42 has a camera on the top, which photographs a predetermined location such as the plaza 100 and visitors 1 who have come to the predetermined location. For example, the camera may be mounted on the top of the robot 42, and the lens may be configured to constantly rotate in one direction around the vertical direction to photograph the surroundings.

[0072] The robot 42 also estimates its current position from a distance sensor or images captured by a camera mounted on the robot 42. The distance sensor, for example, transmits light such as a laser beam from the robot 42 to multiple fixed positions, such as a wall, and estimates the distance from the time difference between the received light. A lidar, which detects the distance from the robot 42 to a fixed position using a laser beam, may be used as the distance sensor. The current position of the robot 42 may be estimated by analyzing images captured by a camera facing the same direction at multiple positions rotated at predetermined angles relative to the vertical, removing noise such as people, and comparing the images with images captured in a state where no people are moving. The current position and the camera images acquired by the robot 42 are transmitted from the robot 42 to the visitor group assessment device 20. The camera images may also include a timestamp and the position of the robot 42 and the camera orientation at the time of capture, associated with the captured image.

[0073] The visitor group grasping device 20 estimates the distance between a specific visitor 1 and other visitors 1 when the specific visitor 1 and other visitors 1 are captured in the captured image from the image captured by the camera of the robot 42 and the position and camera direction of the robot 42 associated with the captured image.

[0074] In this case, by using a camera capable of three-dimensional measurement or multiple cameras, the positions of the robot 42 and multiple visitors 1 on a virtual plan view viewed from directly above may be estimated, as shown in FIG. 8. FIG. 8 shows a case where the image (b) is captured after the image (a) is captured. As shown in FIG. 8, since the robot 42 moves, even if the multiple visitors 1 are stationary, the positions of the multiple visitors 1 in the captured image may move. Even in this case, the visitor group understanding device 20 can estimate the distances between the multiple visitors 1 by estimating the positions of the multiple visitors 1 on the plan view at each capture time, as described above. In this example, other configurations and operations are the same as those shown in FIGS. 1 to 6 or FIG. 7.

[0075] In the configuration of this example, the visitor group identification system 10 may be configured to use both a mobile camera mounted on the robot 42 and a fixed camera attached to the wall of a building or the like, and estimate the distance between multiple visitors 1 using images captured by both cameras. In this case, the visitor group identification system 10 may be configured to classify the visitors 1 using the average distance between the distance between two visitors 1 obtained from images captured by the mobile camera and the distance between two visitors 1 identical to the above two visitors 1 obtained from images captured by the fixed camera.

[0076] The present invention is not limited to the above-described embodiment and its modifications, and it goes without saying that various changes and modifications can be made within the scope of the claims of this application.

[0077] The present invention is further illustrated by the following embodiments. Configuration 1: at least one camera for capturing an image of a predetermined location; a visitor group grasping device that analyzes an image captured by the at least one camera to identify at least one visitor, estimates a distance between the identified visitor and other visitors from the captured image, classifies the identified visitor into either a solo visitor or a group visitor from the estimated distance, and, if the identified visitor is a group visitor, estimates the number of people in the group to which the identified visitor belongs. Visitor group identification system. Configuration 2: the visitor group grasping device estimates that the identified visitor and another visitor belong to the same group when an average value of the distance between the identified visitor and another visitor in a plurality of image frames is equal to or less than a predetermined value; 3. The visitor group identification system according to configuration 1. Configuration 3: the visitor group grasping device estimates that the identified visitor and another visitor belong to the same group when the distance between the identified visitor and another visitor is equal to or greater than a predetermined value for a predetermined number of frames of the continuously captured images; 3. The visitor group identification system according to configuration 1. Configuration 4: the visitor group grasping device estimates attributes of a group to which the identified visitor belongs if the identified visitor is a group visitor; A visitor group identification system according to any one of configurations 1 to 3. Configuration 5: the visitor group grasping device infers that the identified visitor and another visitor belong to the same group if the identified visitor and another visitor are wearing the same type of clothing; A visitor group identification system according to any one of configurations 1 to 4. Configuration 6: the visitor group grasping device identifies the visitor associated with the identification information acquired by the identification information acquiring device, and estimates the distance to other visitors using only the image including the identified visitor associated with the identification information; A visitor group identification system according to any one of configurations 1 to 5. Configuration 7: The visitor group grasping device acquires an estimated head position of the visitor and an estimated skeleton position below the estimated head position from the captured image, estimates the foot positions of the visitor from the acquired estimated skeleton positions, and estimates the distance between the identified visitor and other visitors from the estimated foot positions. A visitor group identification system according to any one of configurations 1 to 6. [Explanation of symbols]

[0078] 1 Visitor, 10 Visitor group identification system, 12 Camera, 20 Visitor group identification device, 21 Memory unit, 22 Output unit, 23 Visitor identification unit, 24 Distance estimation unit, 25 Visitor group identification unit, 30 Display device, 31 Mobile information terminal, 32 Identification information acquisition device, 40 Table, 42 Robot, 100 Plaza.

Claims

1. At least one camera for capturing an image of a predetermined location; a visitor group grasping device that analyzes an image captured by the at least one camera to identify at least one visitor, estimates a distance between the identified visitor and other visitors from the captured image, classifies the identified visitor into either a solo visitor or a group visitor from the estimated distance, and, if the identified visitor is a group visitor, estimates the number of people in the group to which the identified visitor belongs. Visitor group identification system.

2. the visitor group grasping device estimates that the identified visitor and another visitor belong to the same group when an average value of the distance between the identified visitor and another visitor in a plurality of image frames is equal to or less than a predetermined value; The visitor group identification system according to claim 1 .

3. the visitor group grasping device estimates that the identified visitor and another visitor belong to the same group when the distance between the identified visitor and another visitor is equal to or greater than a predetermined value for a predetermined number of frames of the continuously captured images; The visitor group identification system according to claim 1 .

4. the visitor group grasping device estimates attributes of a group to which the identified visitor belongs if the identified visitor is a group visitor; The visitor group identification system according to claim 1 .

5. the visitor group grasping device infers that the identified visitor and another visitor belong to the same group if the identified visitor and another visitor are wearing the same type of clothing; The visitor group identification system according to claim 1 .

6. the visitor group grasping device identifies the visitor associated with the identification information acquired by the identification information acquiring device, and estimates the distance to other visitors using only the image including the identified visitor associated with the identification information; The visitor group identification system according to claim 1 .

7. The visitor group grasping device acquires an estimated head position of the visitor and an estimated skeleton position below the estimated head position from the captured image, estimates the foot positions of the visitor from the acquired estimated skeleton positions, and estimates the distance between the identified visitor and other visitors from the estimated foot positions. The visitor group identification system according to claim 1 .

Images