Analysis program, analysis method, and information processing apparatus

Abstract

To improve accuracy of a position of a person on a map.SOLUTION: An information processing device analyzes a video obtained by capturing the inside of a facility to specify a plurality of parts of a person included in an image constituting the video. The information processing device converts the coordinate position on the image of each of the plurality of parts of the specified person into the coordinate position on the map in the facility. The information processing device extracts any target part of the plurality of parts on the basis of a positional relationship between the converted coordinate positions on the map. The information processing apparatus sets the coordinate position of the extracted target portion on the map as the position where the person is present.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an analysis program and the like. [Background technology]

[0002] In retail businesses, for example, based on video footage of each customer from the time they enter a store until the time they leave, they identify the number of people who entered the store, their attributes, and the movement patterns that indicate their routes, and they use this information to implement measures to increase sales at the store.

[0003] Here, floor mapping is used to identify the movement line of the person. FIG. 19 is a diagram for explaining conventional floor mapping. A camera image 10 is an image (video) taken while a person 5 is walking between shelves. For example, the area (bounding box) of the person 5 at time t is defined as area b. t The area of ​​person 5 that can be placed at time t+1 is area b t+1 The area of ​​person 5 that can be placed at time t+2 is area b t+2 Let's say.

[0004] In the prior art, the center of the bottom edge of the area that is in contact with the floor plane is assumed to be the feet of person 5. t The coordinates of the feet of person 5 in camera image 10 obtained from (x t ,y t ) in area b t+1 The coordinates of the feet of person 5 in camera image 10 obtained from (x t+1 ,y t+1 ) in area b t+2 The coordinates of the feet of person 5 in camera image 10 obtained from (x t+2 ,y t+2 )

[0005] In the prior art, the coordinates (x, y) of the feet of a person assumed in the camera image 10 are converted into coordinates (X, Y) of the map 15 using a homography matrix H. For example, the coordinates (x, y) of the feet of the person 5 in the camera image 10 are converted into coordinates (X, Y) of the map 15. t ,y t ) is the coordinate of map 15 (Xt ,Y t ) is converted into the coordinates (x t+1 ,y t+1 ) is the coordinate of map 15 (X t+1 ,Y t+1 ) is converted into the coordinates (x t+2 ,y t+2 ) is the coordinate of map 15 (X t+2 ,Y t+2 )

[0006] Map 15 coordinates (X t ,Y t ), coordinates (X t+1 ,Y t+1 ), coordinates (X t+2 ,Y t+2 ) is identified as the line of movement (movement trajectory) of the person 5. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent Publication No. 2021-33395 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-177757 [Patent Document 3] International Publication No. 2020 / 188746 Summary of the Invention [Problem to be solved by the invention]

[0008] However, the above-described conventional technology has a problem in that when the feet of a person are occluded, the accuracy of the position of the person on the map decreases.

[0009] For example, if a person's feet are occluded, the coordinates of the person's feet in the camera image will be incorrect. If the homography matrix H is applied to the incorrect coordinates of the feet in the camera image, the resulting map coordinates will differ from the actual coordinates of the person.

[0010] In one aspect, the present invention aims to provide an analysis program, an analysis method, and an information processing device that can improve the accuracy of the position of a person on a map. [Means for solving the problem]

[0011] In the first proposal, a computer executes the following process. The computer analyzes video footage taken inside a facility to identify multiple body parts of a person included in the images that make up the video. The computer converts the coordinate positions on the image of each of the identified multiple body parts of the person into respective coordinate positions on a map of the facility. The computer extracts one of the multiple target body parts based on the positional relationship of each of the converted coordinate positions on the map. The computer sets the coordinate position on the map of the extracted target body part as the location where the person is present. [Effects of the Invention]

[0012] It can improve the accuracy of a person's location on the map. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram for explaining the homography matrix H. As shown in FIG. [Figure 2] FIG. 2 is a diagram for explaining multi-layer homography. [Figure 3] FIG. 3 is a diagram for explaining the process in which the information processing device selects an optimal homography matrix. [Figure 4] FIG. 4 is a diagram (1) for explaining the process of determining whether or not the feet of a person are occluded. [Figure 5] FIG. 5 is a diagram (2) for explaining the process of determining whether or not the feet of a person are occluded. [Figure 6] FIG. 6 is a diagram (3) for explaining the process of determining whether the feet of a person are occluded or not. [Figure 7]FIG. 7 is a diagram for explaining mapping onto a floor map. [Figure 8] FIG. 8 is a diagram showing an example of a person's movement trajectory depicted on a floor map. [Figure 9] FIG. 9 is a diagram illustrating an information processing system according to this embodiment. [Figure 10] FIG. 10 is a functional block diagram showing the configuration of an information processing apparatus according to this embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of the data structure of the homography matrix table. [Figure 12] FIG. 12 is a diagram illustrating an example of skeleton information. [Figure 13] FIG. 13 is a diagram (1) for explaining the processing of the estimation unit. [Figure 14] FIG. 14 is a diagram (2) for explaining the processing of the estimation unit. [Figure 15] FIG. 15 is a flowchart illustrating a processing procedure of the information processing device according to the present embodiment. [Figure 16] FIG. 16 is a flowchart showing the procedure of the estimation process. [Figure 17] FIG. 17 is a flowchart showing the processing procedure for the foot concealment determination process. [Figure 18] FIG. 18 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing device according to this embodiment. [Figure 19] FIG. 19 is a diagram for explaining conventional floor mapping. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, an analysis program, an analysis method, and an information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments. [Example]

[0015] Before describing this embodiment, we will explain the homography matrix H. For example, the homography matrix H is defined by equation (1).

[0016]

number

[0017] In equation (1), (x, y) indicates the coordinates of the floor in the camera image. (X, Y) indicates the coordinates on the map. The homography matrix H is a matrix including h1 to h8 and 1.

[0018] 1 is a diagram illustrating the homography matrix H. For example, from the correspondence of four or more points between the camera image 10 and the map 15, the homography matrix H is calculated to convert the coordinates (x, y) of the camera image 10 into the coordinates (X, Y) of the map 15.

[0019] In the example shown in Figure 1, points p1-1, p1-2, p1-3, and p1-4 on shelf 3 in camera image 10 are associated with points p2-1, p2-2, p2-3, and p2-4 on shelf 3 in map 15, respectively, and a homography matrix H is calculated from these point correspondences.

[0020] For example, in the conventional technology described in Fig. 19, the coordinates of the camera image of person 5 are converted into map coordinates using one homography matrix H. As described above, such conventional technology has a problem in that the accuracy of the person's position on the map decreases when the person's feet are occluded.

[0021] In order to solve the problems of the conventional technology, in this embodiment, multiple homography matrices are used to solve the problem of reduced accuracy in determining the position of a person on a map even when the person's feet are occluded. In the following description, an apparatus that executes processing according to this embodiment will be referred to as an "information processing apparatus 100."

[0022] For example, the information processing device 100 calculates a homography matrix H0 and a homography matrix Hi A multi-layer homography consisting of the above is used. Figure 2 is a diagram for explaining the multi-layer homography.

[0023] The homography matrix H0 is a homography matrix for converting the coordinates of a point in the camera image 20 into a floor position (coordinate) on the map 25. The information processing device 100 uses the homography matrix H0 when converting the coordinates of the feet p0 of the person 5 into coordinates on the map 25.

[0024] Homography matrix H i is a homography matrix for converting the coordinates of points in the camera image 20 into coordinates on a plane having a certain height from the floor of the map 25. The homography matrix H i As an example, the homography matrices H1, H2, and H3 are shown below. The height of the plane 25-1 is 60 cm from the floor of the map 25. The height of the plane 25-2 is 130 cm from the floor of the map 25. The height of the plane 25-2 is 150 cm from the floor of the map 25.

[0025] The information processing device 100 uses a homography matrix H1 when transforming the coordinates of the waist p1 of the person 5 into coordinates on the plane 25-1. The information processing device 100 uses a homography matrix H2 when transforming the coordinates of the shoulder p2 of the person 5 into coordinates on the plane 25-1. The information processing device 100 uses a homography matrix H3 when transforming the coordinates of the head p3 of the person 5 into coordinates on the plane 25-3. It is assumed that the height from the feet p0 of the person 5 to the waist p1 is 60 cm. The height from the feet p0 to the shoulder p2 of the person 5 is 130 cm. The height from the feet p0 to the head p3 of the person 5 is 150 cm.

[0026] By using the multi-layer homography shown in Figure 2, when the feet p0 of person 5 are hidden, it is possible to calculate the coordinates (two-dimensional) of map 25 using the coordinates of the waist p1 and head p3 observable from camera image 20 and the corresponding homography matrices H1 and H3.

[0027] Here, if the height of the waist p1 of person 5 is 60 cm, the coordinates of map 25 can be accurately calculated using the homography matrix H1. On the other hand, if the height of the waist p1 of person 5 is not 60 cm, the coordinates of map 25 cannot be accurately calculated using the homography matrix H1. The same applies to the shoulders p2 and head p3 of person 5.

[0028] Therefore, when the feet p0 of the person 5 are hidden, the information processing device 100 calculates the coordinates of the waist p1, shoulders p2, and head p3 of the person 5 in the camera image 20 and the corresponding homography matrix H i The coordinates of the map 25 are calculated comprehensively using the above.

[0029] As mentioned above, the conversion from camera image coordinates to floor coordinates using the homography matrix H requires that the height of the body part of the person being used be known. In other words, to accurately convert the camera image coordinates of a certain body part of a person into map coordinates, the homography matrix H corresponding to the height of the certain body part must be known. i If a transformation is performed using a homography matrix H that is different from the height of a certain part, an error will occur in the floor coordinates.

[0030] The height of a person detected from a camera image and the height of each part of the body from the floor vary depending on the person and cannot be known in advance. Therefore, it is important to select an appropriate homography matrix H.

[0031] Regarding the above point, the information processing device 100 performs the following process to convert the coordinates of a person in a camera image into map coordinates. The information processing device 100 identifies multiple body parts of the person in the camera image and determines whether the person's feet are occluded. If the person's feet are not occluded, the information processing device 100 converts the person's coordinates into map coordinates using the homography matrix H0.

[0032] On the other hand, when the person's feet are occluded, the information processing device 100 converts the person's coordinates into map coordinates using an appropriate homography matrix H corresponding to parts of the person other than the feet (head, waist, etc.).

[0033] First, a description will be given of an example of a process for determining whether or not the feet of a person are occluded by the information processing device 100. Note that, as a preliminary preparation for determining whether or not the feet of a person are occluded, the information processing device 100 executes a process for selecting an optimal homography matrix H that matches the height of the body part of the person.

[0034] For example, the information processing device 100 prepares a plurality of homography matrices H for each body part of a person, and selects the optimal homography matrix H for each body part by performing the process described with reference to FIG.

[0035] FIG. 3 is a diagram illustrating the process of selecting an optimal homography matrix. For convenience of explanation, FIG. 3 will be described assuming that the body parts of a person are the "waist" and the "head." For example, information processing device 100 prepares homography matrices H for the "waist" body part at 5-cm intervals ranging from 40 cm to 70 cm in height. This results in six types of homography matrices H corresponding to the "waist" body part. Note that the homography matrices H may be set at intervals of 10 cm or 2 cm, instead of 5 cm.

[0036] Information processing device 100 prepares homography matrices H for the body part "head" in 5 cm increments for heights from 120 cm to 190 cm. This results in 14 types of homography matrices H corresponding to the body part "head." Note that the homography matrices H may be set in increments of 10 cm or 2 cm, instead of 5 cm.

[0037] The information processing device 100 evaluates which combination of heights of the homography matrix H can be used to accurately calculate the coordinates of the person on the floor for the person detected from the camera image, and performs processing to identify the optimal combination of homography matrices H.

[0038] The information processing device 100 calculates the homography matrix H 11 The position (coordinates) of the map 25 obtained by transforming the coordinates of the waist p1 of the person 5 using the homography matrix H 31 The position (coordinates) of the map 25 obtained by converting the coordinates of the head p3 of the person 5 using the coordinate system is defined as P31.

[0039] For example, the homography matrix H 11 is the homography matrix H corresponding to a waist height of 50 cm. 31 is the homography matrix H corresponding to a head height of 120 cm.

[0040] The information processing device 100 calculates the homography matrix H 12 The position (coordinates) of the map 25 obtained by transforming the coordinates of the waist p1 of the person 5 using the homography matrix H 32 The position (coordinates) of the map 25 obtained by converting the coordinates of the head p3 of the person 5 using the coordinate system is set to P32.

[0041] For example, the homography matrix H 12 is the homography matrix H corresponding to a waist height of 60 cm. 32 is the homography matrix H corresponding to a head height of 160 cm.

[0042] The information processing device 100 calculates the homography matrix H based on the distribution of P11 and P31 in the map 25. 11 and the homography matrix H 31 The information processing device 100 evaluates the pair of the homography matrix H 11 and the homography matrix H 31 A larger score is given to pairs with

[0043] The information processing device 100 calculates the homography matrix H based on the distribution of P12 and P32 in the map 25. 12 and the homography matrix H32 The information processing device 100 evaluates the pair of the homography matrix H 12 and the homography matrix H 32 A larger score is given to pairs with

[0044] The information processing device 100 repeatedly performs the above process for other combinations of the homography matrix H for the body part "waist" and the homography matrix H for the body part "head," and evaluates each combination of homography matrices H. The information processing device 100 selects the combination of homography matrices H that results in the maximum score as the optimal homography matrix H for each body part. Of the multiple body parts, the homography matrix H corresponding to the feet is fixed to homography matrix H0.

[0045] Note that, when there are three or more target body parts, the information processing device 100 may determine the score based on the distance between the two most distant coordinates among the coordinates of the map mapped by the homography matrix H. For example, when mapping the coordinates of the head, shoulders, and waist, if the distance between the coordinates of the head and shoulders is the most distant among the combinations of head and waist, head and shoulders, and shoulders and waist, the information processing device 100 calculates the score based on the distance between the coordinates of the head and shoulders.

[0046] Next, a process will be described in which the information processing device 100 determines whether or not the feet of a person are occluded by using the optimal homography matrix H for each selected part. Figures 4, 5, and 6 are diagrams for explaining the process of determining whether or not the feet of a person are occluded.

[0047] First, FIG. 4 will be described. In the camera image 21 shown in FIG. 4, an obstacle 6 exists between the person 5 and the camera. From the camera image 21, the coordinates (incorrect coordinates) of the feet of the person 5 (x a0 ,y a0 ) and the coordinates of Person 5's waist (x a1 ,y a1 ) and the coordinates of Person 5's waist (x a3 ,y a3) is detected.

[0048] For convenience of explanation, the optimal combination of the homography matrix H of the waist of person 5 and the homography matrix H of the head of person 5 is referred to as the homography matrix H 1op and the homography matrix H 3op The homography matrix H corresponding to the feet is the homography matrix H0.

[0049] The information processing device 100 calculates the coordinates (x a0 ,y a0 ) is converted to map 26 and the coordinates are (X a0 ,Y a0 ) The information processing device 100 calculates the homography matrix H 1op Using this, the coordinate of person 5's waist (x a1 ,y a1 ) is converted to map 26 and the coordinates are (X a1 ,Y a1 ) The information processing device 100 calculates the homography matrix H 3op Using this, the coordinates of the head of person 5 (x a3 ,y a3 ) is converted to map 26 and the coordinates are (X a3 ,Y a3 )

[0050] As shown in FIG. 4, when the feet of person 5 are occluded, (X a1 ,Y a1 ) and (X a3 ,Y a3 ) compared to the densely packed locations of coordinates (X a0 ,Y a0 ) are separated.

[0051] Next, a description will be given of Fig. 5. In the example shown in Fig. 5, the coordinates (x b0 ,y b0 ) and the coordinates of Person 5's waist (x a1 ,y a1 ) and the coordinates of Person 5's waist (x a3 ,y a3) is detected.

[0052] For convenience of explanation, the optimal combination of the homography matrix H of the waist of person 5 and the homography matrix H of the head of person 5 is referred to as the homography matrix H 1op and the homography matrix H 3op The homography matrix H corresponding to the feet is the homography matrix H0.

[0053] The information processing device 100 calculates the coordinates (x b0 ,y b0 ) is converted to map 27 and the coordinates are (X b0 ,Y b0 ) The information processing device 100 calculates the homography matrix H 1op Using this, the coordinate of person 5's waist (x a1 ,y a1 ) is converted to map 27 and the coordinates are (X a1 ,Y a1 ) The information processing device 100 calculates the homography matrix H 3op Using this, the coordinates of the head of person 5 (x a3 ,y a3 ) is converted to map 27 and the coordinates are (X a3 ,Y a3 )

[0054] As shown in FIG. 5, when the feet of person 5 are not occluded, (X a1 ,Y a1 ), (X a3 ,Y a3 ), (X b0 ,Y b0 ) are densely packed.

[0055] That is, the information processing device 100 determines whether the person's feet are occluded by executing the following process: The information processing device 100 converts the coordinates of the person's feet in the camera image into map coordinates using the homography matrix H0. The information processing device 100 converts the coordinates of each part of the person's body other than the feet (for example, the head, waist, etc.) into map coordinates using the optimal homography matrix H, and calculates the average coordinate of each coordinate.

[0056] The information processing device 100 determines that the person's feet are occluded when the distance between the map coordinates calculated from the coordinates of the person's feet and the average coordinates is equal to or greater than the threshold r. On the other hand, the information processing device 100 determines that the person's feet are not occluded when the distance between the map coordinates calculated from the coordinates of the person's feet and the average coordinates is less than the threshold r.

[0057] Let us now move on to the explanation of Figure 6. For example, among the multiple body parts of a person, each body part (for example, head, waist, etc.) other than the feet is converted into map coordinates using the optimal combination of homography matrix H, and the average coordinate obtained as a result is (X av1 ,Y av1 ) and the coordinates of the person's feet are converted to map coordinates using the homography matrix H0. a0 ,Y a0 In this case, the information processing device 100 calculates the average coordinate (X av1 ,Y av1 ) and coordinates (X a0 ,Y a0 ) is greater than or equal to the threshold r, it is determined that the person's feet are occluded.

[0058] For example, among the multiple body parts of a person, each body part other than the feet (for example, the head, waist, etc.) is converted into map coordinates using the optimal combination of homography matrices H, and the average coordinates obtained as a result are expressed as (X av2 ,Y av2 ) and the coordinates of the person's feet are converted to map coordinates using the homography matrix H0. b0 ,Y b0 In this case, the information processing device 100 calculates the average coordinate (Xav2 ,Y av2 ) and coordinates (X b0 ,Y b0 ) is greater than or equal to the threshold r, it is determined that the person's feet are not occluded.

[0059] An example of the process in which the information processing device 100 determines whether the feet of a person are concealed or not has been described above.

[0060] If the information processing device 100 determines that the person's feet are not occluded, it converts the coordinates of the person's feet in the camera image into map coordinates using the homography matrix H0, and maps the map coordinates onto a floor map. This floor map is information that includes the above-mentioned map information as well as product layout information for the store.

[0061] On the other hand, when it is determined that the feet of the person are occluded, the information processing device 100 converts the coordinates of the person's head in the camera image into map coordinates using the homography matrix H optimal for the head, and maps the head to the coordinates on the floor map corresponding to the map coordinates. Note that the homography matrix H optimal for the body part is determined in advance by the process described in FIG. 3.

[0062] 7 is a diagram for explaining mapping onto a floor map. When it is determined that the feet of a person are occluded, the information processing device 100 calculates the coordinates (x a3 ,y a3 ) is converted into map coordinates, and the coordinates (X a3 ,Y a3 )

[0063] On the other hand, when it is determined that the feet of the person are not occluded, the information processing device 100 calculates the coordinates (x b0 ,y b0) is converted into map coordinates, and the coordinates (X b0 ,Y b0 )

[0064] The information processing device 100 repeatedly executes the above process on time-series camera images to draw a movement trajectory of a person on a floor map 28. FIG. 8 is a diagram showing an example of a movement trajectory of a person drawn on a floor map. For example, the floor map 28 shown in FIG. 8 is displayed on a display device or the like. In the example shown in FIG. 8, a plurality of shelves 28a, a cash register 28b, and doors 28c and 28d are set in advance on the floor map 28. The information processing device 100 causes the display device to display a display screen on which the information of FIG. 8 is set. For example, the floor map 28 is a two-dimensional map of the inside of a store made up of shelf areas where products are arranged and aisle areas where people move.

[0065] Furthermore, the information processing device 100 performs the above-described processing on time-series camera images of person A, thereby mapping the movement trajectory LA of person A on the floor map 28. From the movement trajectory LA, it can be seen that person A entered the store through door 28c, looked at several shelves 28a, paid at cash register 28b, and left the store through door 28d.

[0066] Similarly, the information processing device 100 performs the above process on time-series camera images of person B, thereby mapping person B's movement trajectory LB on the floor map 28. From the movement trajectory LB, it can be seen that person B entered the store through door 28d, looked at several shelves 28a, and then left through door 28d without purchasing anything. In other words, the information processing device 100 generates a trajectory of the person from the entrance to the exit within the facility on the floor map.

[0067] Furthermore, the information processing device 100 identifies the entrances and exits that the person has passed through among multiple entrances and exits within the store, and plots the person's trajectory in the passage area from the identified entrance to the exit on the floor map. For example, the information processing device 100 identifies the entrances and exits by tracking the person's movement trajectory within the store. Note that the expressions "entrance" within a store and "exit" within a facility include the areas set aside for entering and exiting each floor in a one-story or multi-story facility.

[0068] As described above, the information processing device 100 identifies multiple body parts of a person included in a camera image, and converts the coordinates of each of the identified multiple body parts on the camera image into coordinates on a map using the homography matrix H. The information processing device 100 extracts a target body part from among the multiple body parts based on the positional relationship of each of the coordinates on the map, and sets the coordinates on the map of the extracted target body part as the position where the person is located.

[0069] For example, if the distance between the coordinates corresponding to the person's feet on the map and the coordinates (average coordinates) corresponding to other parts of the map (face, waist, shoulders) is less than a threshold r, the information processing device 100 determines that the person's feet are not occluded. In this case, the information processing device 100 sets the coordinates of the person's feet on the map as the position where the person is located. Therefore, if the person's feet are not occluded, the person's position can be set using the highly reliable coordinates of the person's feet.

[0070] On the other hand, when the distance between the coordinates corresponding to the person's feet on the map and the coordinates (average coordinates) corresponding to other parts of the map (face, waist, shoulders) is equal to or greater than the threshold r, the information processing device 100 determines that the person's feet are occluded. In this case, the information processing device 100 sets the coordinates on the map of other parts (for example, head) instead of the feet as the position where the person is located. Therefore, when the person's feet are occluded, the position of the person can be set using the coordinates of other parts of the map that are more reliable.

[0071] Here, for each combination of multiple homography matrices corresponding to each body part excluding the feet, the information processing device 100 calculates the coordinates of each body part projected by the homography matrix and evaluates the combination of homography matrices based on the distribution of the calculated coordinates. Based on the evaluation results, the information processing device 100 identifies an optimal homography matrix. This allows the accurate calculation of the person's position on the map from body parts excluding the feet using the optimal homography matrix, even if the person's height and the height of each body part from the floor are not known in advance.

[0072] The information processing device 100 can notify the user of information on the person's movement trajectory by repeatedly executing the process of mapping the map coordinates of the person obtained by the above process onto the floor map for time-series camera images. Note that in this embodiment, the case where coordinates are mapped onto the floor map from the map coordinates has been described, but if there is a one-to-one correspondence between the map coordinates and the floor map coordinates, the position of the person may be mapped onto the floor map directly from the coordinates of the camera image.

[0073] Next, an example of an information processing system including an information processing device 100 that executes the above-described processing will be described. Fig. 9 is a diagram showing an information processing system according to this embodiment. As shown in Fig. 9, an information processing system 30 has camera devices 31-1, 31-2, 31-3, ..., 31-n and an information processing device 100. The camera devices 31-1 to 31-n (n is an arbitrary natural number) and the information processing device 100 are connected to each other via a network 32.

[0074] The network 32 may be any of a variety of communication networks, whether wired or wireless, such as an intranet used within a retail store. The network 32 may not be a single network, but may instead be configured, for example, by connecting an intranet and the Internet via a network device such as a gateway or other device (not shown). The expression "inside" a retail store is not limited to indoors, but may also include outdoors within the premises of the retail store.

[0075] Camera devices 31-1 to 31-n are surveillance cameras installed in each sales floor or cash register area in a store such as a retail store. In the following description, camera devices 31-1 to 31-n will be collectively referred to as camera device 31. Video data captured by camera device 31 is transmitted to information processing device 100 via network 32. The video data includes time-series camera images. The inside of the facility is, for example, the inside of a store such as a retail store.

[0076] The information processing device 100 receives video data from the camera device 31, and executes the above-described processing on the time-series camera images included in the received video data.

[0077] Next, a configuration example of the information processing device 100 will be described. Fig. 10 is a functional block diagram showing the configuration of the information processing device according to this embodiment. As shown in Fig. 10, the information processing device 100 includes a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0078] The communication unit 110 executes data communication with the camera device 31 and the like via the network 32. For example, the communication unit 110 receives an image of a person from the camera device 31.

[0079] The input unit 120 is an input device that inputs various types of information to the control unit 150 of the information processing device 100. For example, the input unit 120 corresponds to a keyboard, a mouse, a touch panel, or the like.

[0080] The display unit 130 is a display device that displays information output from the control unit 150.

[0081] The storage unit 140 includes an image DB 141 and a homography matrix table 142. The storage unit 140 is a memory or the like.

[0082] The video DB 141 stores data of time-series camera images captured by the camera device 31. The video DB 141 can store position information such as coordinates for identifying individual products or sales area areas, or extracted people from each camera image, by linking them to each camera image. The video DB 141 may also store skeletal information of people detected and identified from the camera images. The generation of skeletal information will be described later.

[0083] Homography matrix table 142 is a table that holds multiple homography matrices corresponding to each body part. Fig. 11 is a diagram showing an example of the data structure of the homography matrix table. As shown in Fig. 11, homography matrix table 142 associates body parts with homography matrices.

[0084] 11, the body parts are the head, shoulders, waist, and feet. For the body part "head," multiple homography matrices corresponding to each head height are set. For example, a homography matrix is ​​set for each head height in 5 cm increments from 120 cm to 190 cm.

[0085] For the "shoulder" body part, multiple homography matrices are set corresponding to different shoulder heights. For example, a homography matrix is ​​set for each shoulder height in 5 cm increments between 100 cm and 170 cm.

[0086] For the body part "waist," multiple homography matrices are set corresponding to different waist heights. For example, a homography matrix is ​​set for each waist height in 5 cm increments between 40 cm and 70 cm.

[0087] For the part "feet", one homography matrix H0 for the feet is set.

[0088] Next, we move on to explaining the control unit 150. The control unit 150 has an acquisition unit 151, a detection unit 152, an estimation unit 153, a determination unit 154, and a mapping unit 155. The control unit 150 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0089] The acquisition unit 151 acquires video data (time-series camera image data) from the camera device 31. The acquisition unit 151 stores the acquired video data in the video DB 141.

[0090] The detection unit 152 detects a person's area from the camera image stored in the video DB 141, and generates skeletal information of the person from the detected person's area. The detection unit 152 identifies the coordinates (coordinates of the camera image) of each part (face, shoulders, waist, feet) from the skeletal information, and stores the coordinates of each part in the video DB 141 in association with the camera image.

[0091] First, an example of a process in which the detection unit 152 detects a person's area from a camera image will be described. The detection unit 152 extracts a person from a camera image using an existing detection algorithm such as YOLO (YOU Only Look Once), SSD (Single Shot Multibox Detector), or RCNN (Region Based Convolutional Neural Networks). The extracted person is indicated by a bounding box (BBOX) that surrounds the area in a rectangular shape on the image.

[0092] Next, a description will be given of an example of a process in which the detection unit 152 generates skeletal information from a person's area. The detection unit 152 acquires skeletal information by inputting the extracted person's area (image of BBOX) into a trained machine learning model constructed using an existing algorithm such as DeepPose or OpenPose.

[0093] FIG. 12 is a diagram showing an example of skeletal information. The skeletal information can use 18 pieces of definition information (numbered from 0 to 17) in which each joint identified by a known skeletal model is numbered. For example, the right shoulder joint (SHOULDER_RIGHT) is assigned number 7, the left elbow joint (ELBOW_LEFT) is assigned number 5, the left knee joint (KNEE_LEFT) is assigned number 11, and the right hip joint (HIP_RIGHT) is assigned number 14. Therefore, coordinate information for the 18 skeletal joints shown in FIG. 12 can be acquired from image data; for example, "X coordinate = X7, Y coordinate = Y7" is acquired as the position of the right shoulder joint number 7.

[0094] For example, the detection unit 152 acquires the coordinates of the body part (HEAD) identified as No. 3 in the skeletal information as the coordinates of the body part "head." The detection unit 152 acquires the coordinates of the right shoulder joint (SHOULDER_RIGHT) identified as No. 7 in the skeletal information as the coordinates of the body part "shoulder." The detection unit 152 may also acquire the coordinates of the left shoulder joint (SHOULDER_LEFT) identified as No. 4 in the skeletal information as the coordinates of the body part "shoulder."

[0095] The detection unit 152 acquires the coordinates of the part numbered 0 (SPLINE_BASE) in the skeletal information as the coordinates of the part "waist." The detection unit 152 acquires the average coordinates of the coordinates of the part numbered 13 (FOOT_LEFT) and the coordinates of the part numbered 17 (FOOT_RIGHT) in the skeletal information as the coordinates of the part "feet." Note that the detection unit 152 may acquire the coordinates of the center of the bottom edge of the person's area (BBOX) as the coordinates of the part "feet."

[0096] Estimation unit 153 selects (estimates) from homography matrix table 142 an optimal combination of homography matrices corresponding to each body part registered in association with a camera image in video DB 141. For example, for each combination of multiple homography matrices corresponding to each body part excluding the feet, estimation unit 153 calculates the coordinates of each body part projected onto a map by the homography matrix, and evaluates the combination of homography matrices from the distribution of the calculated coordinates. Estimation unit 153 estimates the optimal combination of homography matrices based on the evaluation results.

[0097] The processing of the estimation unit 153 basically corresponds to the processing described in FIG. 3, but as described in FIG. 13, by narrowing down the waist height relative to the head height position, the number of combinations of homography matrices to be evaluated can be reduced.

[0098] 13 is a diagram (1) for explaining the processing of the estimation unit. Here, the description will be given using the head and the waist as body parts. For example, the estimation unit 153 narrows down the homography matrices of the waist p1, which are to be combined with the homography matrices for the head p3 having a height of 1.5 m, to those of the waist that fall between 40% (0.6 m) of 1.5 m and 60% (0.9 m) of 1.5 m, and performs evaluation.

[0099] The estimation unit 153 narrows down the homography matrices of the waist p1 to be combined with the homography matrices for the head p3 height of 1.6 m to those that fall between 40% (0.64 m) of 1.6 m and 60% (0.96 m) of 1.6 m, and performs evaluation.

[0100] The estimation unit 153 narrows down the homography matrices of the waist p1 to be combined with the homography matrices for the head p3 height of 1.7 m to those that fall between 40% (0.68 m) of 1.6 m and 60% (0.1.02 m) of 1.6 m, and performs evaluation.

[0101] 14 is a diagram (2) for explaining the processing of the estimation unit. For example, the estimation unit 153 uses a homography matrix of the height of the head p3, "1.5 m", to calculate the coordinates of the head of the person in the camera image as the coordinates (X a3 ,Y a3 The estimation unit 153 converts the coordinates of the waist of the person in the camera image into coordinates (X a1 ,Y a1 )

[0102] The estimation unit 153 uses the homography matrix of the height of the head p3, "1.6 m", to calculate the coordinates of the head of the person in the camera image as the coordinates (X b3 ,Y b3 The estimation unit 153 converts the coordinates of the waist of the person in the camera image into coordinates (X b1 ,Y b1 )

[0103] The estimation unit 153 uses the homography matrix of the height of the head p3, "1.7 m", to calculate the coordinates of the head of the person in the camera image as the coordinates (X c3 ,Y c3 The estimation unit 153 converts the coordinates of the waist of the person in the camera image into coordinates (X c1 ,Y c1 )

[0104] The estimation unit 153 evaluates the distances of the coordinates of the maps 28-1 to 28-3, and finds that the coordinates of the map 28-2 (X b3 ,Y b3 ) and (X b1 ,Y b1 ) is the smallest, and the score is also the largest. The estimation unit 153 estimates that the combination of the homography matrix of the head p3 height of "1.6 m" and the homography matrix of the waist p1 height of "0.6 m", which results in the smallest distance, is the optimal combination of homography matrices. Note that this also makes it possible to estimate that the height of the person in the camera image is "1.6 m" and the height of the person's waist is "0.6 m".

[0105] The estimation unit 153 outputs the estimated optimal homography matrix for each part to the determination unit 154 and the mapping unit.

[0106] The determination unit 154 determines whether the feet of a person in a camera image are occluded. For example, the determination unit 154 converts each of the multiple body parts of a person other than the feet (head, shoulders, waist, etc.) into map coordinates using an optimal combination of homography matrices, and calculates the average coordinates.

[0107] The determination unit 154 determines that the person's feet are occluded when the distance between the average coordinate and the coordinates of the person's feet on the map is equal to or greater than the threshold r. On the other hand, the determination unit 154 determines that the person's feet are not occluded when the distance between the average coordinate and the coordinates of the person's feet on the map is less than the threshold r.

[0108] Determination section 154 outputs the determination result to mapping section 155. Other explanations regarding determination section 154 are the same as those explained with reference to Figs.

[0109] Mapping unit 155 maps the coordinates of the person on the map using the determination result of determination unit 154. If the feet of the person are not occluded, mapping unit 155 uses homography matrix H0 to convert the coordinates of the person in the camera image into coordinates on the map.

[0110] When the feet of a person are occluded, mapping unit 155 converts the coordinates of the person in the camera image into coordinates on the map using the homography matrix H that is optimal for the head height. For example, when the head height of the person is estimated to be "1.6 m" by the processing described in Fig. 14, mapping unit 155 converts the coordinates of the person in the camera image into coordinates on the map using the homography matrix H for the head height of "1.6 m".

[0111] The mapping unit 155 maps the position of the person to coordinates on the floor map that correspond to the coordinates of the person on the map. The mapping unit 155 generates a floor map by repeatedly executing the above process on time-series camera images, and outputs the generated floor map to the display unit 130 for display. For example, the floor map that the mapping unit 155 displays on the display unit 130 corresponds to the floor map 28 shown in FIG. 8.

[0112] Other details regarding the mapping unit 155 are the same as those described with reference to FIGS.

[0113] Next, an example of a processing procedure of the information processing device 100 according to this embodiment will be described. Fig. 15 is a flowchart showing the processing procedure of the information processing device according to this embodiment. As shown in Fig. 15, the acquisition unit 151 of the information processing device 100 acquires video data from the camera device 31 and stores it in the video DB 141 (step S101).

[0114] The detection unit 152 of the information processing device 100 acquires a camera image from the video DB 141, detects a person area from the camera image, and generates skeleton information from the person area (step S102). The estimation unit 153 of the information processing device 100 executes estimation processing (step S103).

[0115] The determination unit 154 of the information processing device 100 executes a feet occlusion determination process (step S104). If the mapping unit 155 of the information processing device 100 determines that the feet are not occluded (step S105, No), the process proceeds to step S106. The information processing device 100 converts the coordinates of the feet in the camera image into map coordinates using the feet homography matrix H0, maps them on the floor map (step S106), and proceeds to step S108.

[0116] On the other hand, if it is determined that the feet are occluded (step S105, Yes), the mapping unit 155 proceeds to step S107. The information processing device 100 converts the coordinates of the head in the camera image into map coordinates using the homography matrix H corresponding to the estimated head height, and maps them on the floor map (step S107).

[0117] If the information processing device 100 continues the process (step S108, Yes), the process proceeds to step S102. On the other hand, if the information processing device 100 does not continue the process (step S108, No), the process ends.

[0118] Next, an example of the processing procedure of the estimation processing described in step S103 of Fig. 15 will be described. Fig. 16 is a flowchart showing the processing procedure of the estimation processing. As shown in Fig. 16, when there is an unobtained combination of parts (head, shoulders, waist, etc.) (step S201, Yes), the estimation unit 153 of the information processing device 100 proceeds to step S202.

[0119] The estimation unit 153 acquires one set of homography matrices for a combination that has not yet been acquired from the homography matrix table 142 (step S202). The estimation unit 153 uses the acquired set of homography matrices to convert the coordinates of the camera image of each part into coordinates of the map (step S203).

[0120] The estimation unit 153 calculates and stores the coordinates of the map of each part (step S204), and proceeds to step S201.

[0121] On the other hand, if there is no combination of parts (head, shoulders, waist, etc.) that has not yet been acquired (No at step S201), the estimation unit 153 proceeds to step S205.

[0122] The estimation unit 153 selects a pair of homography matrices that minimizes the distance between the coordinates of the maps of each body part (step S205). The estimation unit 153 estimates the height of each body part corresponding to the selected pair of homography matrices as the height of each body part of the person (step S206).

[0123] Next, an example of the processing procedure for the foot occlusion determination processing described in step S104 of Fig. 15 will be described. Fig. 17 is a flowchart showing the processing procedure for the foot occlusion determination processing. As shown in Fig. 17, the determination unit 154 of the information processing device 100 converts the coordinates of the camera image of each part into map coordinates using a homography matrix corresponding to the estimated height of each part, and calculates the average coordinates (step S301).

[0124] The determination unit 154 converts the coordinates of the camera image of the feet into map coordinates and performs mapping using the homography matrix H0 of the feet (step S302).

[0125] If the distance between the average coordinates and the coordinates of the feet is less than the threshold r (step S303, Yes), the determination unit 154 determines that the feet are not occluded (step S304). On the other hand, if the distance between the average coordinates and the coordinates of the feet is not less than the threshold r (step S303, No), the determination unit 154 determines that the feet are occluded (step S305).

[0126] The processing procedure of the information processing device 100 according to this embodiment has been described above.

[0127] Next, the effects of the information processing device 100 according to this embodiment will be described. The information processing device 100 identifies multiple body parts of a person included in a camera image, and converts the coordinates of each of the identified multiple body parts on the camera image into coordinates on a map using a homography matrix. The information processing device 100 extracts a target body part from among the multiple body parts based on the positional relationship of each of the coordinates on the map, and sets the coordinates on the map of the extracted target body part as the location where the person is located. This can improve the accuracy of the person's position on the map.

[0128] If the distance between the coordinates corresponding to the person's feet on the map and the coordinates (average coordinates) corresponding to other parts of the map (face, waist, shoulders) is less than threshold r, the information processing device 100 determines that the person's feet are not occluded. In this case, the information processing device 100 sets the coordinates of the person's feet on the map as the position where the person is located. Therefore, if the person's feet are not occluded, the person's position can be set using the highly reliable coordinates of the person's feet.

[0129] The information processing device 100 determines that the person's feet are occluded when the distance between the coordinates corresponding to the person's feet on the map and the coordinates (average coordinates) corresponding to other parts of the map (face, waist, shoulders) is equal to or greater than a threshold r. In this case, the information processing device 100 sets the coordinates on the map of other parts (for example, head) instead of the feet as the position where the person is located. Therefore, when the person's feet are occluded, the position of the person can be set using the coordinates of other parts of the map that are more reliable.

[0130] For each combination of multiple homography matrices corresponding to each body part excluding the feet, the information processing device 100 calculates the coordinates of each body part projected by the homography matrix and evaluates the combination of homography matrices based on the distribution of the calculated coordinates. Based on the evaluation results, the information processing device 100 identifies an optimal homography matrix. This allows the accurate calculation of a person's position on a map from body parts excluding the feet using the optimal homography matrix, even if the person's height and the height of each body part from the floor are not known in advance.

[0131] The information processing device 100 can notify the user of information about the person's movement trajectory by repeatedly performing the process of mapping the map coordinates of the person obtained by the above process onto a floor map for time-series camera images.

[0132] It has been described that the information processing device 100 identifies the position of a person on a map by using the person's feet as a preset body part. However, the information processing device 100 may identify the position of a person on a map by using a body part other than the feet as a preset body part. For example, the information processing device 100 may use the head.

[0133] The information processing device 100 identifies multiple body parts of a person in a camera image and determines whether the person's head is occluded. If the person's head is not occluded, the information processing device 100 converts the person's coordinates into map coordinates using a homography matrix H0 (homography matrix corresponding to the head). On the other hand, if the person's head is occluded, the information processing device 100 converts the person's coordinates into map coordinates using an appropriate homography matrix H corresponding to parts other than the person's head (feet, waist, etc.).

[0134] More specifically, for example, the determination unit 154 converts the coordinates of the camera image of the head into coordinates of the map using the head homography matrix H0. Then, if the distance between the average coordinates and the head coordinates is less than the threshold r, the determination unit 154 determines that the head is not occluded. On the other hand, if the distance between the average coordinates and the head coordinates is not less than the threshold r, the determination unit 154 determines that the head is occluded.

[0135] Next, for example, if it is determined that the head is not occluded, mapping unit 155 converts the coordinates of the head in the camera image into map coordinates using homography matrix H0 and maps them onto the floor map. On the other hand, if it is determined that the head is occluded, mapping unit 155 converts the coordinates of the feet in the camera image into map coordinates using homography matrix H and maps them onto the floor map.

[0136] This allows the information processing device 100 to improve the accuracy of the position of a person on a map even when a predetermined body part is hidden.

[0137] Next, a description will be given of an example of a hardware configuration of a computer that realizes the same functions as the above-described information processing device 100. Fig. 18 is a diagram showing an example of a hardware configuration of a computer that realizes the same functions as the information processing device according to this embodiment.

[0138] 18, the computer 200 has a CPU 201 that executes various types of arithmetic processing, an input device 202 that accepts data input from a user, and a display 203. The computer 200 also has a communication device 204 that exchanges data with the camera device 31 and the like via a wired or wireless network, and an interface device 405. The computer 200 also has a RAM 206 that temporarily stores various types of information, and a hard disk device 207. The devices 201 to 207 are connected to a bus 408.

[0139] The hard disk drive 207 stores an acquisition program 207a, a detection program 207b, an estimation program 207c, a determination program 207d, and a mapping program 207e. The CPU 201 reads out each of the programs 207a to 207e and loads them into the RAM 206.

[0140] The acquisition program 207a functions as an acquisition process 206a. The detection program 207b functions as a detection process 206b. The estimation program 207c functions as an estimation process 206c. The determination program 207d functions as a determination process 206d. The mapping program 207e functions as a mapping process 206e.

[0141] The processing of the acquisition process 206a corresponds to the processing of the acquisition unit 151. The processing of the detection process 206b corresponds to the processing of the detection unit 252. The processing of the estimation process 206c corresponds to the processing of the estimation unit 153. The processing of the determination process 206d corresponds to the processing of the determination unit 154. The processing of the mapping process 206e corresponds to the processing of the mapping unit 155.

[0142] It should be noted that each of the programs 207a to 207e does not necessarily have to be stored in the hard disk drive 207 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207e.

[0143] The following supplementary notes are further disclosed regarding the embodiments including the above examples.

[0144] (Appendix 1) By analyzing the video taken inside the facility, multiple body parts of a person included in the images constituting the video are identified, converting the coordinate positions on the image of each of the plurality of body parts of the identified person into respective coordinate positions on a map within the facility; extracting a target region from among the plurality of regions based on the positional relationship of each of the converted coordinate positions on the map; The coordinate position of the extracted target part on the map is set as a position where a person is present. An analysis program that causes a computer to execute processing.

[0145] (Appendix 2) The analysis program described in Appendix 1 is characterized in that the process of extracting the target area comprises, based on the comparison result between the coordinate position of a first area on the map and the distribution of coordinate positions on the map of multiple other areas other than the first area among the multiple areas, if the first area is located in a position different from the distribution range of the multiple other areas, extracting a second area included in the multiple other areas as the target area.

[0146] (Appendix 3) The analysis program according to appendix 1, further comprising the computer executing the following process: acquiring multiple images including the person from a camera; performing image analysis on each of the multiple images to detect the coordinate positions of multiple body parts including the person's head and feet; projecting the detected multiple coordinate positions onto the map using a homography matrix; and, if the difference between the coordinate position of the foot projected onto the map and the coordinate position of a representative body part other than the foot is equal to or greater than a threshold, setting the coordinate position of the person's head as the position on the map where the person was located.

[0147] (Appendix 4) The analysis program according to appendix 3, further comprising causing a computer to execute a process of selecting the first homography matrix or the second homography matrix based on a distribution of coordinate positions of the plurality of body parts obtained by projecting the coordinate positions of each of the plurality of body parts, including the head and feet of the person, onto the map using a first homography matrix, and a distribution of coordinate positions of the plurality of body parts obtained by projecting the coordinate positions of the plurality of body parts onto the map using a second homography matrix different from the first homography matrix.

[0148] (Appendix 5) The analysis program according to appendix 4, characterized in that the setting process uses the first homography matrix or the second homography matrix selected by the selecting process to project onto the map the coordinate positions of each of a plurality of parts including the person's head and feet to obtain a representative coordinate position of a part other than the feet, and sets the coordinate position of the person's head as the position on the map where the person was located if the difference between the coordinate position of the feet projected onto the map is equal to or greater than a threshold.

[0149] (Appendix 6) The analysis program described in Appendix 1, further comprising causing a computer to execute a process of generating skeletal information of a person included in the video, and identifying the coordinate positions on the image of each of a plurality of parts of the person based on the joint positions of the generated skeletal information.

[0150] (Appendix 7) A floor map showing the layout of the store on a two-dimensional plane, including shelf areas where products are placed and aisle areas where people move, is obtained; Based on a time-series change in the position where the person is present, a trajectory of the person in the passage area from an entrance through which the person passed to an exit among a plurality of entrances in the store is drawn on the floor map; The analysis program according to claim 1, further comprising causing a computer to execute a process of displaying on a display device a screen on which the trajectory of the person is drawn on the floor map.

[0151] (Supplementary Note 8) When it is determined that a difference between a coordinate position of the first body part and a coordinate position indicating a set of the other plurality of body parts on the map satisfies a first condition, a first trajectory of the person is generated using the coordinate position of the first body part; generating a second trajectory of the person using the coordinate positions of the second body part when it is determined that a difference between the coordinate position of the first body part and the coordinate positions indicating a set of the other plurality of body parts on the map satisfies a second condition; The analysis program described in Appendix 2 further causes the computer to execute a process of generating a trajectory of a person on a floor map based on a third trajectory that is an integration of the first trajectory and the second trajectory.

[0152] (Appendix 9) By analyzing the video taken inside the facility, multiple body parts of a person included in the images constituting the video are identified, converting the coordinate positions on the image of each of the plurality of body parts of the identified person into respective coordinate positions on a map within the facility; extracting a target region from among the plurality of regions based on the positional relationship of each of the converted coordinate positions on the map; The coordinate position of the extracted target part on the map is set as a position where a person is present. An analysis method characterized in that the processing is executed by a computer.

[0153] (Appendix 10) The analysis method described in Appendix 9, characterized in that the process of extracting the target area comprises, based on a comparison result between the coordinate position of a first area on a map and the distribution of coordinate positions on a map of multiple other areas other than the first area among the multiple areas, if the first area is located in a position different from the distribution range of the multiple other areas, extracting a second area included in the multiple other areas as the target area.

[0154] (Appendix 11) The analysis method according to appendix 9, further characterized in that the computer executes the following process: acquiring a plurality of images including the person from a camera; performing image analysis on each of the plurality of images to detect the coordinate positions of each of a plurality of body parts including the person's head and feet; projecting the detected plurality of coordinate positions onto the map using a homography matrix; and, if the difference between the coordinate position of the foot projected onto the map and the coordinate position of a representative body part other than the foot is equal to or greater than a threshold, setting the coordinate position of the person's head as the position on the map where the person was located.

[0155] (Appendix 12) The analysis method according to appendix 11, further characterized in that the computer executes a process of selecting the first homography matrix or the second homography matrix based on a distribution of coordinate positions of the plurality of body parts obtained by projecting the coordinate positions of each of the plurality of body parts, including the head and feet of the person, onto the map using a first homography matrix, and a distribution of coordinate positions of the plurality of body parts obtained by projecting the coordinate positions of the plurality of body parts onto the map using a second homography matrix different from the first homography matrix.

[0156] (Appendix 13) The analysis method according to appendix 12, characterized in that the setting process uses the first homography matrix or the second homography matrix selected by the selecting process to project onto the map the coordinate positions of each of a plurality of parts including the person's head and feet to obtain a representative coordinate position of a part other than the feet, and sets the coordinate position of the person's head as the position on the map where the person was located when the difference between the coordinate position of the feet projected onto the map is equal to or greater than a threshold.

[0157] (Appendix 14) The analysis method described in Appendix 9, characterized in that the computer further executes a process of generating skeletal information of a person included in the video, and identifying the coordinate positions on the image of each of multiple parts of the person based on the joint positions of the generated skeletal information.

[0158] (Appendix 15) A floor map showing the configuration of the store on a two-dimensional plane having shelf areas where products are placed and aisle areas where people move is obtained, Based on a time-series change in the position where the person is present, a trajectory of the person in the passage area from an entrance through which the person passed to an exit among a plurality of entrances in the store is drawn on the floor map; The analysis method according to claim 9, further characterized in that the computer executes a process of displaying on a display device a screen on which the trajectory of the person is drawn on the floor map.

[0159] (Supplementary Note 16) When it is determined that a difference between a coordinate position of the first body part and a coordinate position indicating a set of the other plurality of body parts on the map satisfies a first condition, a first trajectory of the person is generated using the coordinate position of the first body part; generating a second trajectory of the person using the coordinate positions of the second body part when it is determined that a difference between the coordinate position of the first body part and the coordinate positions indicating a set of the other plurality of body parts on the map satisfies a second condition; The analysis method described in Appendix 10, characterized in that the computer further executes a process of generating a trajectory of a person on a floor map based on a third trajectory that is an integration of the first trajectory and the second trajectory.

[0160] (Appendix 17) By analyzing the video taken inside the facility, multiple body parts of a person included in the images constituting the video are identified, converting the coordinate positions on the image of each of the plurality of body parts of the identified person into respective coordinate positions on a map within the facility; extracting a target region from among the plurality of regions based on the positional relationship of each of the converted coordinate positions on the map; The coordinate position of the extracted target part on the map is set as a position where a person is present. An information processing device having a control unit that executes processing.

[0161] (Appendix 18) The information processing device described in Appendix 17 is characterized in that the process of extracting the target part extracts a second part included in the other multiple parts as the target part based on the comparison result between the coordinate position of the first part on the map and the distribution of coordinate positions on the map of other multiple parts other than the first part among the multiple parts, if the first part is located in a position different from the distribution range of the other multiple parts.

[0162] (Appendix 19) The information processing device described in Appendix 17, characterized in that the control unit further executes a process of acquiring multiple images including the person from the camera, performing image analysis on each of the multiple images to detect the coordinate positions of each of multiple body parts including the person's head and feet, projecting the detected multiple coordinate positions onto the map using a homography matrix, and setting the coordinate position of the person's head as the position where the person was located on the map if the difference between the coordinate position of the foot projected onto the map and the coordinate position of a representative body part other than the foot is equal to or greater than a threshold.

[0163] (Appendix 20) The information processing device described in Appendix 19, characterized in that the control unit further executes a process of selecting the first homography matrix or the second homography matrix based on a distribution of coordinate positions of the multiple parts obtained as a result of projecting the coordinate positions of each of the multiple parts, including the person's head and feet, on the map using a first homography matrix, and a distribution of coordinate positions of the multiple parts obtained as a result of projecting the coordinate positions of the multiple parts on the map using a second homography matrix different from the first homography matrix.

[0164] (Appendix 21) The information processing device described in Appendix 20, characterized in that the setting process uses the first homography matrix or the second homography matrix selected by the selecting process to project the coordinate positions of each of the multiple parts including the person's head and feet onto the map to obtain a representative coordinate position of a part other than the feet, and sets the coordinate position of the person's head as the position where the person was located on the map when the difference between the coordinate position of the feet projected onto the map is equal to or greater than a threshold.

[0165] (Appendix 22) The information processing device described in Appendix 17, characterized in that the control unit further performs a process of generating skeletal information of a person included in the video and identifying the coordinate positions on the image of each of multiple parts of the person based on the joint positions of the generated skeletal information.

[0166] (Supplementary Note 23) The control unit acquires a floor map showing the layout of the store, including shelf areas where products are placed and aisle areas where people move, drawing a trajectory of the person in the passage area of ​​the floor map on the floor map based on a time series change in the position where the person is present, The information processing device according to claim 17, further comprising a process of displaying on a display device a screen on which the trajectory of the person is drawn on the floor map.

[0167] (Supplementary Note 24) When it is determined that a difference between a coordinate position of the first body part and a coordinate position indicating a set of the other plurality of body parts on the map satisfies a first condition, the control unit generates a first trajectory of the person using the coordinate position of the first body part; generating a second trajectory of the person using the coordinate positions of the second body part when it is determined that a difference between the coordinate position of the first body part and the coordinate positions indicating a set of the other plurality of body parts on the map satisfies a second condition; The information processing device described in Appendix 18, further comprising a process of generating a trajectory of a person on a floor map based on a third trajectory that is an integration of the first trajectory and the second trajectory. [Explanation of symbols]

[0168] 100 Information processing device 110 Communications Department 120 Input section 130 Display section 140 Storage section 141 Video DB 142 Homography Matrix Table 150 control section 151 Acquisition Department 152 Detection unit 153 Estimation Department 154 Judgment section

Claims

1. By analyzing the video taken inside the facility, a plurality of body parts of a person included in the images constituting the video are identified, converting the coordinate positions on the image of each of the plurality of body parts of the identified person into respective coordinate positions on a map within the facility; extracting a target region from among the plurality of regions based on the positional relationship of each of the converted coordinate positions on the map; The coordinate position of the extracted target part on the map is set as a position where a person is present. An analysis program that causes a computer to execute processing.

2. The analysis program of claim 1, characterized in that the process of extracting the target part extracts a second part included in the other multiple parts as the target part based on a comparison result between the coordinate position of a first part on the map and the distribution of coordinate positions on the map of other multiple parts other than the first part among the multiple parts, if the first part is located in a position different from the distribution range of the other multiple parts.

3. The analysis program according to claim 1, further comprising the steps of: acquiring a plurality of images including the person from a camera; performing image analysis on each of the plurality of images to detect the coordinate positions of each of a plurality of body parts including the person's head and feet; projecting the detected plurality of coordinate positions onto the map using a homography matrix; and, if a difference between the coordinate position of the foot projected onto the map and a representative coordinate position of a body part other than the foot is equal to or greater than a threshold, setting the coordinate position of the person's head as the position on the map where the person was located.

4. 4. The analysis program according to claim 3, further causing a computer to execute a process of selecting the first homography matrix or the second homography matrix based on a distribution of coordinate positions of the plurality of body parts obtained as a result of projecting the coordinate positions of each of the plurality of body parts, including the person's head and feet, on the map using a first homography matrix, and a distribution of coordinate positions of the plurality of body parts obtained as a result of projecting the coordinate positions of the plurality of body parts on the map using a second homography matrix different from the first homography matrix.

5. 5. The analysis program according to claim 4, wherein the setting process uses the first homography matrix or the second homography matrix selected by the selecting process to project coordinate positions of a plurality of body parts including the person's head and feet onto the map, and if a difference between a representative coordinate position of a body part other than the feet obtained by projecting and transforming the coordinate positions of each of the body parts including the person's head and feet onto the map and the coordinate position of the feet projected onto the map is equal to or greater than a threshold value, sets the coordinate position of the person's head as the position on the map where the person was located.

6. The analysis program according to claim 1, further comprising causing a computer to execute a process of generating skeletal information of a person included in the video, and identifying the coordinate positions on the image of each of a plurality of parts of the person based on the joint positions of the generated skeletal information.

7. A floor map showing the layout of the store on a two-dimensional plane having shelf areas where products are placed and aisle areas where people move is obtained; Based on a time-series change in the position where the person is present, a trajectory of the person in the passage area from an entrance through which the person passed to an exit among a plurality of entrances in the store is drawn on the floor map; 2. The analysis program according to claim 1, further causing a computer to execute a process of displaying, on a display device, a screen on which the trajectory of the person is drawn on the floor map.

8. generating a first trajectory of the person using the coordinate position of the first body part when it is determined that a difference between a coordinate position of the first body part and a coordinate position indicating a set of the other plurality of body parts on the map satisfies a first condition; generating a second trajectory of the person using the coordinate positions of the second body part when it is determined that a difference between the coordinate position of the first body part and the coordinate positions indicating a set of the other plurality of body parts on the map satisfies a second condition; The analysis program according to claim 2, further comprising causing the computer to execute a process of generating a trajectory of a person on a floor map based on a third trajectory that is an integration of the first trajectory and the second trajectory.

9. By analyzing the video taken inside the facility, a plurality of body parts of a person included in the images constituting the video are identified, converting the coordinate positions on the image of each of the plurality of body parts of the identified person into respective coordinate positions on a map within the facility; extracting a target region from among the plurality of regions based on the positional relationship of each of the converted coordinate positions on the map; The coordinate position of the extracted target part on the map is set as a position where a person is present. An analysis method characterized in that the processing is executed by a computer.

10. By analyzing the video taken inside the facility, a plurality of body parts of a person included in the images constituting the video are identified, converting the coordinate positions on the image of each of the plurality of body parts of the identified person into respective coordinate positions on a map within the facility; extracting a target region from among the plurality of regions based on the positional relationship of each of the converted coordinate positions on the map; The coordinate position of the extracted target part on the map is set as a position where a person is present. An information processing device having a control unit that executes processing.

Images