Data collection system

Abstract

To provide a data collection system capable of appropriately collecting data related to exercise.SOLUTION: A system 1 comprises a camera 300, estimation means for estimating activity amounts of users based on image data, identification means for identifying users captured by the camera, and presentation means for presenting exercise data indicating the activity amounts to the users. The identification means extracts a person region including a person from the image data, and when the person regions of two or more users overlap, calculates an ID vector indicating a probability of each ID of target users for the overlapping person regions as a branching scene with the two or more users as the target users. The identification means estimates joint coordinates of users during exercise based on the image data. The identification means extracts characteristic data indicating characteristics of exercise patterns of the users from time-series data to identify the users.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a data collection system.

Background Art

[0002] Patent Document 1 discloses a system for detecting a person represented in an image. In this system, a person area is detected from an image, and when two or more person areas overlap, they are integrated. The system identifies the skeleton of the person closest to the front among the people included in the integrated person area. Then, based on the skeleton, the system masks the area representing the person in the front and identifies the skeleton of the person behind.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, devices have been developed that measure a user's activity level and present exercise data to the user. For example, a camera can be used to image a user, and the image of the camera can be used to estimate the user's activity level. However, in the image of the camera, when two or more people overlap, it becomes difficult to appropriately estimate the user's activity level. For example, when occlusion occurs, it becomes difficult to identify the user.

[0005] The present disclosure has been made in view of such problems, and provides a data collection system capable of appropriately collecting data related to exercise.

Means for Solving the Problems

[0006] The data collection system according to one aspect of the present disclosure is installed in a sports facility where a user exercises, and includes a camera that images the user, an estimation means for estimating the activity amount of the user based on the image data of the camera, an identification means for identifying the user imaged by the camera, and a presentation means for presenting exercise data indicating the activity amount of the user to the identified user. The identification means extracts a person area including a person from the image data of the camera, and when the person areas of two or more users overlap, as a branching scene in which the two or more users are target users, an ID vector indicating the probability for each ID of the target user is calculated for the overlapping person areas. Based on the image data of the camera, the joint coordinates of the user during exercise are estimated, feature data indicating the characteristics of the user's exercise pattern is extracted from the time-series data of the joint coordinates, and the user is identified based on the feature data.

Advantages of the Invention

[0007] According to the present disclosure, it is possible to provide a data collection system that can appropriately collect data related to exercise.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0009] Hereinafter, specific embodiments to which the present disclosure is applied will be described in detail with reference to the drawings. However, the present disclosure is not limited to the following embodiments. Also, for clarity of explanation, the following description and drawings are appropriately simplified.

[0010] Hereinafter, the system 1 according to the embodiment will be described with reference to the drawings. FIG. 1 is a schematic diagram showing the overall configuration of the system. The system 1 collects data related to the exercise of the user U who exercises in the exercise facility 10. Specifically, the system 1 includes a server 100, a user terminal 200, a camera 300, and the like.

[0011] The exercise facility 10 is a fitness gym, a stadium, or the like. Note that the exercise facility 10 is not limited to indoor facilities as long as the user U can exercise, and may be outdoor. In addition, the exercise facility 10 is provided with exercise equipment 500 and an authentication device 510.

[0012] The exercise equipment 500 includes a running machine, a treadmill, a walking machine, a rowing machine, a cross-trainer, a fitness bike, a vibration machine, and the like. In addition, the exercise equipment 500 may be a training machine using weights. The exercise equipment 500 may be a stretching machine, a massage chair, or the like. The user U can perform a desired exercise using the exercise equipment 500. Note that the exercise equipment 500 is not limited to those provided with an operation mechanism for performing exercise, and may be a yoga mat, a fitness mat, or the like. A unique device ID may be set for the exercise equipment 500. The exercise performed by the user U may not use equipment such as squats, sit-ups, and push-ups.

[0013] The authentication device 510 authenticates users who use the sports facility 10. The authentication device 510 is provided at the entrance 11 of the sports facility 10 and the like. That is, the entrance 11 serves as a gate with an authentication function. By the authentication device 510 performing authentication, the user U entering the room is identified. As a result, the pre-registered user U can enter the sports facility 10. An ID is registered for each user U. The authentication device 510 may be a device that uses biometric authentication such as face authentication, fingerprint authentication, or voice authentication. In this case, the authentication device 510 is equipped with a sensor according to the authentication method.

[0014] Also, the authentication device 510 may be a device that uses a QR code (registered trademark). Alternatively, the authentication device 510 may perform authentication using a password or PIN. In this case, the authentication device 510 is equipped with an input device such as a touch panel for the user U to input a password or PIN. When the authentication device 510 performs authentication using face authentication or a QR code, it is equipped with a camera for capturing the face or QR code. When performing voice authentication, the authentication device 510 is equipped with a microphone or the like.

[0015] The user U authenticated by the authentication device 510 is also referred to as the target user. That is, the target user is the user U who has been authenticated by the authentication device 510 and entered the sports facility 500. In other words, the authentication device 510 extracts the entered target user from the registered users who are facility users. Then, the authentication device 510 sends the authentication result to the server 100.

[0016] The camera 300 is provided in the sports facility 10 and captures the user U. For example, a plurality of cameras 300 are installed on the walls and ceilings of the sports facility 10. A unique camera ID may be set for the plurality of cameras 300.

[0017] In the exercise machine 500, the camera 300 captures an image of the user U who has entered the exercise facility 10. Also, in the exercise facility 10, it is preferable to install the camera 300 so that all the exercise machines 500 can be imaged. Each camera 300 captures an image of the user U who is exercising on the exercise machine 500. It is desirable that the camera 300 be installed so that there are no blind spots within the exercise facility 10. Also, based on the image of the camera 300 and the authentication result of the authentication device 510, the user within the exercise facility 10 can be detected.

[0018] The server 100 is connected to the user terminal 200, the camera 300, and the authentication device 510 via a network. The server 100 receives various data from the camera 300, the authentication device 510, and the user terminal 200. The server 100 transmits various data to the camera 300, the authentication device 510, and the user terminal 200. The server 100 may be installed within the exercise facility 10 or may be installed at a remote location away from the exercise facility 10.

[0019] The server 100 is an information processing device equipped with a processor, a memory, etc., and performs predetermined processing on data. The server 100 performs processing for presenting exercise data indicating the activity level of the user U to the user U. The server 100 transmits the exercise data of the user U to the user terminal 200. The processing in the server 100 will be described later. At least a part of the processing in the server 100 may be implemented by the camera 300, the user terminal 200, etc. For example, when the camera 300 has an arithmetic processing function, the camera 300 may perform processing such as the posture vacancy described later.

[0020] In addition, the user terminal 200 is a smartphone, tablet terminal, personal computer, etc. held by the user U. The user terminal 200 includes a processor, a memory, a display, a touch panel, a microphone, a speaker, etc. The user terminal 200 may be a wearable device such as a smartwatch or smart glasses. For example, each user U uses one or more user terminals 200. The user terminal 200 is associated according to the user ID information of the user U. When the user terminal 200 receives motion data from the server 100, it displays it on the display screen. The user U can check the exercise time, exercise content, energy consumption, etc. In this way, the system 1 can present motion data to the user U.

[0021] The processing in the server 100 will be described with reference to FIGS. 2 and 3. FIG. 2 is a control block diagram showing the configuration of the server 100. FIG. 3 is a schematic diagram showing an example of an image captured by the camera 300. In FIG. 3, two exercise devices 500 are identified as exercise devices 500A and 500B. In FIG. 3, the exercise device 500A is a stationary bicycle, and the exercise device 500B is a treadmill, but the number and type of the exercise device 500 are not particularly limited. Also, in FIG. 3, two users U are identified as users U1 and U2. User U1 is exercising on the exercise device 500A, and user U2 is walking towards the exercise device 500B.

[0022] As shown in FIG. 2, the server 100 includes a specifying unit 110, an estimating unit 120, a presenting unit 130, a communication unit 140, and a database 150. The specifying unit 110 includes a person detection unit 111, an ID vector calculation unit 112, a joint coordinate estimation unit 113, and a user specifying unit 114. The estimating unit 120 includes a posture estimation unit 121, an ROI detection unit 122, a motion counter 123, and a motion time acquisition unit 124.

[0023] The database 150 stores user data for each user U. For example, the user data may include data regarding the age, gender, height, weight, etc. of user U. Further, the database 150 may store exercise data regarding the user's exercise. The exercise data includes feature data indicating the characteristics of the exercise pattern when the user exercises. The feature data is data corresponding to the time-series data of joint coordinates. For example, the feature data becomes data indicating the trajectory of the coordinates of the toes when running. The characteristics of the exercise pattern may be, for example, the stroke of the joint position in the reciprocating motion.

[0024] The feature data and exercise data are stored in the database 150 for each user. The feature data and exercise data stored in the database 150 are used as reference data. The physique, etc. of each user is different. Therefore, even when users perform the same exercise, the feature data varies according to the user. The feature data may be data obtained from an image captured in the past, or may be data estimated from physical characteristics such as the user's height and weight.

[0025] Further, the database 150 stores data regarding the exercise facility 10. For example, the database 150 stores equipment data regarding the exercise equipment 500 installed in the exercise facility 10. The equipment data includes data regarding the type, installation location, and number of the exercise equipment 500, etc. Further, the installation location and imaging direction of the camera 300 are associated with the exercise equipment. For example, within the exercise facility 10, the positions and orientations of the exercise equipment 500 and the camera 300 are known. Therefore, the camera 300 that images each exercise equipment 500 can be specified. Also, within the image of the camera 300, the position of the exercise equipment 500 can be specified.

[0026] Therefore, the server 100 can identify the exercise area in the image captured by the camera 300 where the user exercises using the exercise equipment 500. In FIG. 3, the exercise area RA for the exercise equipment 500A and the exercise area RB for the exercise equipment 500B are shown. The exercise areas RA and RB are identified as a predetermined range in the image captured by the camera 300. The exercise areas RA and RB may be identified by the camera ID of the camera 300 and the xy coordinates of the camera 300. An exercise area is associated with each exercise equipment 500.

[0027] The identification unit 110 identifies a person based on the image data of the camera 300. For example, the camera 300 captures an image of the entrance 11 and its surroundings. The identification unit 110 includes a person detection unit 111, an ID vector calculation unit 112, a joint coordinate estimation unit 113, and a user identification unit 114. The processing in the ID vector calculation unit 112, the joint coordinate estimation unit 113, and the user identification unit 114 will be described later.

[0028] The person detection unit 111 detects and tracks a person who has entered the room based on the image data around the entrance 11. For example, the identification unit 110 uses deep learning techniques such as SSD (Single Shot MultiBox Detector) or YOLO (You Only Look Once) to detect a person. The person detection unit 111 uses a deep learning-based technique such as Deepsort or a general method such as a Kalman filter to track a person. Further, the authentication device 510 authenticates the user who has entered from the entrance 11. Therefore, the identification unit 110 can identify the ID of the user U who has entered from the entrance 11. In FIG. 3, the person area B1 of the user U1 and the person area B2 of the user U2 are shown. The person areas B1 and B2 are frames that define the area where a person (user) exists in the image. For example, the person area B1 is a bounding box that surrounds the user U1.

[0029] The estimation unit 120 estimates the user's activity level based on the image data of the camera 300. The estimation unit 120 includes a posture estimation unit 121, an ROI detection unit 122, a motion counter 123, and a motion time acquisition unit 124.

[0030] The camera 300 is imaging the exercise equipment 500 and its surroundings. The posture estimation unit 121 estimates the posture of the user U using the exercise equipment 500 based on the image. That is, it detects the posture of the user U moving in the exercise area. In FIG. 3, the estimated postures of the users U1 and U2 are shown respectively. The posture estimation unit 121 estimates the joint coordinates for the area where people are located using posture estimation techniques such as Openpose and Transpose.

[0031] The posture estimation unit 121 identifies the xy coordinates (also referred to as joint coordinates) indicating the joint positions in each frame of the image of the camera 300. Then, the posture estimation unit 121 estimates the user's posture (skeleton) by connecting the respective joint coordinates. The posture estimation unit 121 can obtain time-series data of the user's joint coordinates by estimating the posture at predetermined time intervals.

[0032] The ROI (Region Of Interest) detection unit 122 detects the ROI (region of interest) from the image based on the estimated posture. For example, it detects as the ROI the range including joints such as the knee joint, ankle joint, elbow joint, and wrist joint of the user U in the exercise area.

[0033] The motion counter 123 counts the number of joint movements in the ROI. For example, the motion counter 123 counts the number of motion movements based on the time-series data of the xy coordinates of the joint positions. The motion counter 123 calculates the peak value of the crown coordinates from the time-series data of the joint coordinates. More specifically, when the user exercises on a treadmill or a running machine, the motion counter 123 counts the motion based on the coordinates of the toes. The motion counter 123 obtains the peak value of the time-series data indicating the variation of the toe coordinates and counts the number of steps at the timing of the peak value. Thereby, the motion counter 123 can count the number of motion movements such as the number of steps (stride count).

[0034] When the user is in the motion area, the motion counter 123 increments the number of steps at the timing when the left and right toe coordinates reach the peak value. Of course, the motion counter 123 can also count the number of repetitions based on the time-series data of the joint positions for walking, pedaling (ankle movement), etc. The motion counter 123 counts the number of times the joint coordinates reciprocate in the image as the number of motion movements. Also, the motion counter 123 may calculate the motion period, the number of motion movements, etc. by performing a short-time Fourier transform on the time-series data.

[0035] Also, it is preferable that joints serving as the ROI are set according to the camera 300, the exercise equipment 500, and the motion area. For example, the joints that move significantly vary depending on the exercise. Therefore, the joints that move significantly are set as the ROI. In a bike machine, etc., the left and right toes, knee joints, and ankle joints are set as the ROI. Also, in a tracking machine, a running machine, a walking machine, etc., the left and right toes, wrist joints, elbow joints, etc. are set as the ROI. Furthermore, depending on the installation position and imaging direction of the camera 300, the joints serving as the ROI may be set. For example, the joints that become blind spots of the camera 300 may be set so as not to be the ROI.

[0036] The exercise time acquisition unit 124 acquires the exercise time of each user based on the images of the camera 300. For example, the exercise time acquisition unit 124 sets the time when the user enters the exercise area of the exercise device 500 as the exercise start time, and the time when the user exits the exercise area of the exercise device 500 as the exercise end time. The exercise time acquisition unit 124 calculates the time from the exercise start time to the exercise end time as the exercise time. Alternatively, the exercise time may be calculated based on the time of the actions counted by the motion counter 123 or the time series data of the joint coordinates. Alternatively, the exercise time acquisition unit 124 may acquire the exercise time from the exercise device 500. For example, when the exercise device 500 is a treadmill or a bike machine with a timer or clock function, the server 100 can acquire the exercise time from the exercise device 500.

[0037] The presentation unit 130 presents the exercise data indicating the activity level of the user to the user. For example, the presentation unit 130 generates exercise data indicating the exercise content, exercise type, exercise time, exercise frequency, degree of joint bending, load intensity, energy consumption, etc. for each user. The exercise data is not limited to the above data as long as it indicates the activity level. The communication unit 140 transmits the exercise data indicating the activity level of the user to the user terminal 200 of the user. The exercise data may be in a general format such as the csv format.

[0038] In addition, in order for the server 100 to perform the above processing, the communication unit 140 receives the image data of the camera 300. The communication unit 140 receives the authentication result of the authentication device 510. The data communication by the communication unit 140 etc. may be wireless communication such as WiFi (registered trademark), or may be wired communication. When the user terminal 200 receives the exercise data from the server 100, it displays it on the display. Thereby, the user U can confirm the exercise data regarding his own exercise. Therefore, the system 1 can support the user U to live a healthier life.

[0039] It is possible to identify the exercising user U without installing a camera 300, a personal authentication device, or the like on each exercise device 500. In addition, since it is only necessary to provide the camera 300 serving as a sensor in the exercise facility 10, exercise data can be presented at low cost.

[0040] In addition, as shown in FIG. 4, in the image, there may be a case where two or more users U cross each other. That is, when an occlusion occurs where the users U cross each other, there is a possibility that the system 1 cannot appropriately track the users. Since the system 1 may misidentify the users, there is a possibility that the exercise data may be presented to the wrong user U. Therefore, in the first embodiment, a process for preventing the replacement of user IDs is performed.

[0041] When the human regions of two or more users overlap, the ID vector calculation unit 112 sets a branch scene in which two or more users are target users, and calculates an ID vector indicating the probability for each ID of the target users with respect to the overlapping human regions. For example, in a branch scene where occlusion occurs, the ID vector indicates the probability that a user of each ID exists in one human region. That is, the ID vector calculation unit 112 obtains a probability distribution in which a plurality of users U exist for each human region. When there are n (n is an integer of 2 or more) users U in the exercise facility 10, the ID vector x is represented by the following equation (1). x = [p1, p2, …, pi, .., pn] ··· (1)

[0042] p1 is the existence probability of the first user U1, and pn is the existence probability of the nth user Un. pi is the existence probability of the ith (i is an arbitrary integer of 1 or more and n or less) user Ui. Here, the system 1 can detect the users in the exercise facility 10 and the number n thereof based on the authentication result at the authentication device 510 and the image. The sum of p1 to pn may be set to 1. The ID vector x is set for each human region.

[0043] Next, the process of presenting exercise data to the user using the ID vector x will be described with reference to FIGS. 2 and 5. FIG. 5 is a flowchart showing the processing in the system 1.

[0044] First, the authentication device 510 authenticates the user who has entered the room (S101). As a result, the user ID of the user who has entered the room can be obtained. Next, the human detection unit 111 detects and tracks the user based on the image from the camera 300 (S102). That is, human regions B1 and B2 as shown in FIGS. 3 and 4 are set for each user in the exercise facility 10.

[0045] At this time, the ID vector calculation unit 112 calculates the probability of the presence of a user in the exercise facility 10. For example, when the user U1 enters the room, in the human region around the entrance 11, a single user U1 is identified based on the authentication result of the authentication device 510. Therefore, an initial ID vector x is set such that the probability of presence p1 of the identified single user U1 is 1, and the probabilities of presence p2 to pn of the other users U2 to Un are 0, respectively. Also, when a user exits, the probability of presence of the exiting user is set to 0. In this way, when the authentication device 510 detects entry or exit, the ID vector calculation unit 112 may update the ID vector x. Also, when the authentication device 510 detects entry or exit, the dimensionality of the ID vector x may change. And through user tracking, the value of the ID vector is maintained until the branching scene.

[0046] The ID vector calculation unit 112 updates the ID vector x in the branching scene (S103). When the overlapping ratio or overlapping area of the two human regions becomes a certain value or more, the ID vector calculation unit 112 determines that occlusion has occurred. When occlusion occurs, it becomes a branching scene in which the ID vector is updated. The ID vector calculation unit 112 updates the ID vector x for each of the two overlapping human regions. That is, the ID vector calculation unit 112 changes the probability of presence of the target user with the users in the overlapping human regions as the target users.

[0047] As shown in FIG. 4, when the human region B1 and the human region B2 overlap, the ID vector calculation unit 112 changes the existence probability of the target users with the users U1 and U2 as the target users. The ID vector calculation unit 112 decreases the existence probability p1 and increases the existence probability p2 in the ID vector x1 of the human region B1. For example, the ID vector calculation unit 112 decreases the existence probability p1 by multiplying the existence probability p1 by a coefficient less than 1. Further, the ID vector calculation unit 112 may increase the existence probability p2 by the amount by which the existence probability p1 is decreased.

[0048] For example, when the ID vector x = [1, 0, 0, ….., 0] before the overlap, after the overlap, the ID vector x = [0.5, 0.5, 0, ….., 0]. Similarly, when the human region B1 and the human region B2 overlap, in the ID vector x2 of the human region B2, the existence probability p2 is decreased and the existence probability p1 is increased. The timing at which a plurality of human regions overlap becomes a branch scene, and the ID vector calculation unit 112 updates the existence probability of the target user in the ID vector x. The target user can be the user with the highest existence probability.

[0049] When occlusion occurs multiple times for one human region, each time the ID vector calculation unit 112 updates the ID vector x. Of course, when three or more human regions overlap, a coefficient corresponding to the number of people may be multiplied by the existence probability. For example, when three human regions overlap, multiply 1 / 3 by the existence probability. Also, the ID vector calculation unit 112 may change the rate at which the existence probability is decreased according to the size of the person, the orientation of the person, and the overlap ratio. The ID vector calculation unit 112 may update the ID vector by a network obtained by machine learning.

[0050] The joint coordinate estimation unit 113 estimates the joint coordinates of the user (S104). Here, the joint coordinate estimation unit 113 can estimate the joint coordinates using the estimation result of the posture estimation unit 121. The joint coordinate estimation unit 113 acquires time-series data of the joint coordinates.

[0051] When the joint coordinates enter the motion area, the user identification unit 114 extracts the feature data of the motion pattern and calculates the similarity probability of the motion pattern (S105). Since the user U has entered the motion area and is performing a motion, the user identification unit 114 can extract the feature data of the motion pattern. As described above, the feature data of the motion pattern is preset as reference data for each user. Also, since the time-series data of the joint positions is acquired from the image of the camera 300, the user identification unit 114 can extract the feature pattern.

[0052] The user identification unit 114 compares the reference pattern with the extracted feature data and calculates the similarity probability of the pattern. Then, the more the extracted feature data resembles the reference pattern, the higher the similarity probability. The similarity probability is calculated for each user. The similarity probability vector y indicating the similarity probabilities for n users is represented by the following equation (2). y = [q1, q2, …, qi, .., qn] ··· (2)

[0053] Here, q1 is the similarity probability for the first user U1, qn is the similarity probability for the nth user Un, and qi is the similarity probability for the ith user Ui (where i is an arbitrary integer from 1 to n). It may be set such that the sum of q1 to qn is 1.

[0054] The user identification unit 114 identifies the specific user performing the motion based on the ID vector x and the similarity probability vector y (S106). For example, the user identification unit 114 calculates the coincidence probability vector P using the ID vector x and the similarity probability vector y. The coincidence probability vector P can be obtained by the following equation (3). P = [p1*q1, p2*q2, …, pi*qi, …, pn*qn] ··· (3)

[0055] The coincidence probability vector P, the ID vector x, and the similarity probability vector y have the same number of dimensions. The product of the existence probability p1 and the similarity probability q1 is the coincidence probability. The user identification unit 114 calculates the coincidence probability (pi * qi) for each user. The user identification unit 114 identifies the user with the ID for which the product of the existence probability and the similarity probability is maximized as the identified user. That is, the user identification unit 114 identifies the ID of the user performing the movement by the following formula (4). ID = argmax Pi ··· (4)

[0056] When the user identification unit 114 identifies the identified user, the ID vector calculation unit 112 sets the existence probability of the identified user in the ID vector in other people's areas to 0 (S107). For example, when the user identification unit 114 identifies user U1 as the identified user in the ID vector x of the person area B1 which is a person area. Then, the ID vector calculation unit 112 sets the existence probability p1 of user U1 in the ID vectors x of the person areas B2 to Bn to 0. In other words, since the user U1 identified in one person area B1 cannot exist in the other person areas B2 to Bn, the ID vector calculation unit 112 can reduce the existence probability p1 of user U1 in the person areas B2 to Bn.

[0057] In the person areas B2 to Bn, the user identification unit 114 may increase the existence probabilities p2 to pn of the other users U2 to Un by the amount by which the existence probability p1 is reduced. By doing so, the sum of the existence probabilities in the ID vector x can be made a constant value. According to the identification result of the user identification unit 114, the ID vector calculation unit 112 can dynamically update the values of the plurality of ID vectors x. That is, since the system 1 can appropriately obtain the ID vectors x in the plurality of person areas, it is possible to prevent misidentifying the user. Therefore, robust processing against occlusion becomes possible.

[0058] When user U finishes exercising, the estimation unit 120 estimates the user's activity level (S108). For example, when the user moves outside the preset exercise area in the image, the server 100 detects the end of the exercise. Also, the motion counter 123 counts the number of exercises. Furthermore, the exercise time acquisition unit 124 acquires the exercise time from the exercise start time, exercise end time, etc. Therefore, the presentation unit 130 can estimate the activity level such as the number of steps, running distance, and energy consumption based on these values. Also, when the user performs two or more types of exercises, the estimation unit 120 may estimate the respective activity levels.

[0059] The presentation unit 130 presents exercise data indicating the activity level to the user (S109). That is, the communication unit 140 transmits the exercise data of the specific user to the user terminal 200 of the user. The exercise data may be indicated by values for each type of exercise, or may be indicated by a value summarizing a plurality of exercises. The exercise data may include the start time and end time of the exercise.

[0060] The user can check the exercise data regarding the exercises performed by the user himself / herself. Therefore, the user can improve the motivation for exercise and can lead a healthier life. Since the system 1 can automatically collect the exercise data, it can appropriately present the exercise data to the user.

[0061] Note that the user identification unit 114 may calculate a similarity probability based on body data indicating the physical characteristics of the user. For example, depending on the user, the shoulder width, arm length, leg length, torso length, etc. are different. Therefore, values such as shoulder width, arm length, leg length, and torso length from the joint positions can be used as body data. The joint coordinate estimation unit 113 or the posture estimation unit 121 extracts the body data shown from the image of the camera 300. Then, the user identification unit 114 may calculate the similarity probability by comparing the body data with the user data. For example, the closer the leg or arm length is, the higher the similarity probability. The user identification unit 114 may integrate the similarity probability of the characteristics of the body data and the similarity probability of the characteristics of the motion pattern to calculate a similarity probability vector y.

[0062] Note that the ID vector calculation unit 112 may calculate the ID vector x using a Kalman filter. For example, as shown in FIG. 6, the ID vector calculation unit 112 predicts a prior error distribution D2 from the posterior distribution D1 at the previous time (t - 1). The prior error distribution D2 can be predicted from the moving speed of the user U and the moving distance in one frame. Note that the ID vector calculation unit 112 can obtain the moving distance and moving speed of the user by comparing a plurality of frame images. The ID vector calculation unit 112 obtains an observation distribution D3 based on the image of the camera 300. The ID vector calculation unit 112 updates the posterior distribution D4 based on the prior error distribution D2 and the observation distribution D3. The posterior distribution D4 indicates the distribution at the current time t.

[0063] The ID vector calculation unit 112 obtains the ID vector using the central probability of the posterior distribution D4. In this way, the ID vector calculation unit 112 can accurately track the user by using a Kalman filter. Therefore, the ID vector calculation unit 112 can appropriately obtain the ID vector. Of course, the calculation of the ID vector is not limited to the process using a Kalman filter, and techniques based on deep learning can be used.

[0064] Note that the present invention is not limited to the above-described embodiments, and can be appropriately modified without departing from the gist thereof. Further, the present disclosure can be realized by causing a processor such as a CPU (Central Processing Unit) to execute a computer program for part or all of the control processing in the system 1. For example, the server 100, the user terminal 200, etc. can be implemented as a device capable of executing a program such as a central processing unit of a computer. And various functions can also be realized by a program.

Explanation of Signs

[0065] 1 System 100 Server 200 User Terminal 300 Camera 500 Exercise Equipment U User

Claims

1. It is installed in a sports facility where the user exercises, and includes a camera that images the user, Estimation means for estimating the activity amount of the user based on the image data of the camera, Identification means for identifying the user imaged by the camera, Presentation means for presenting exercise data indicating the activity amount of the user to the identified user, and The identification means Extracts a human area containing a person from the image data of the camera, When the human areas of two or more users overlap, as a branching scene with the two or more users as target users, for the overlapping human areas, calculates an ID vector indicating the probability for each ID of the target users, Estimates the joint coordinates of the user during exercise based on the image data of the camera, Extracts feature data indicating the characteristics of the user's exercise pattern from the time-series data of the joint coordinates, and based on the feature data, a data collection system that identifies the user.

2. Based on the feature data, calculates a similarity probability vector indicating the similarity probability of features for each target user, and The data collection system according to claim 1, wherein the user is identified based on the similarity probability vector and the ID vector.

3. When the user included in one human area is identified as a specific user, In the ID vector corresponding to the other human area, sets the probability of the specific user to 0. The data collection system according to claim 1 or 2.

4. The identification means Extracts body data indicating the physical characteristics of the user from the image data of the camera, and The data collection system according to any one of claims 1 to 3, wherein the user is identified based on the body data.

5. When the load of the exercise changes during the user's exercise, The data collection system according to any one of claims 1 to 4, wherein feature data is extracted based on the time-series data when the load is high.

Images