Data collection system

Abstract

To provide a data collection system capable of appropriately collecting data related to exercise.SOLUTION: A system 1 comprises a camera 300, estimation means for estimating activity amounts of users based on image data, identification means for identifying users captured by the camera, and presentation means for presenting exercise data indicating the activity amounts to the users. The identification means extracts a person region including a person from the image data, and when the person regions of two or more users overlap, calculates an ID vector indicating a probability of each ID of target users for the overlapping person regions as a branching scene with the two or more users as the target users. The identification means estimates joint coordinates of users during exercise based on the image data. The identification means extracts characteristic data indicating characteristics of exercise patterns of the users from time-series data to identify the users.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] This disclosure relates to a data collection system.

Background Art

[0002] Patent Document 1 discloses a system for detecting a person represented in an image. In this system, a person region is detected from the image, and when two or more person regions overlap, they are integrated. The system identifies the skeleton of the person closest to the front among the people included in the integrated person region. Then, based on the skeleton, the system masks the region representing the person in the front and identifies the skeleton of the person behind that person.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, devices have been developed that measure a user's activity level and present exercise data to the user. For example, a camera can be used to image a user, and the image of the camera can be used to estimate the user's activity level. However, in the image of the camera, when two or more people overlap, it becomes difficult to appropriately estimate the user's activity level. For example, when occlusion occurs, it becomes difficult to identify the user.

[0005] This disclosure has been made in view of such problems, and provides a data collection system that can appropriately collect data related to exercise.

Means for Solving the Problems

[0006] A data collection system in one aspect of the present disclosure is installed in an exercise facility where a user exercises and comprises a camera that images the user, estimation means for estimating the user's activity level based on the image data from the camera, identification means for identifying the user captured by the camera, and presentation means for presenting exercise data indicating the user's activity level to the identified user. The identification means extracts human regions containing people from the image data from the camera, and if the human regions of two or more users overlap, calculates an ID vector indicating the probability for each ID of the target user for the overlapping human regions as a branching scene targeting the two or more users, estimates the joint coordinates of the user during exercise based on the image data from the camera, extracts feature data indicating the characteristics of the user's exercise pattern from the time-series data of the joint coordinates, and identifies the user based on the feature data. [Effects of the Invention]

[0007] According to this disclosure, a data collection system can be provided that can appropriately collect data related to exercise. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram schematically shows the overall system configuration. [Figure 2] This is a block of the system's control system. [Figure 3] This is a schematic diagram showing an image captured by a camera. [Figure 4] This is a schematic diagram showing an image captured by a camera. [Figure 5] This is a flowchart showing how to identify a user. [Figure 6] This diagram illustrates an example using a Kalman filter. [Modes for carrying out the invention]

[0009] The following describes specific embodiments applying this disclosure with reference to the drawings. However, this disclosure is not limited to the following embodiments. Also, for clarity, the following descriptions and drawings have been simplified as appropriate.

[0010] The following describes System 1 according to an embodiment with reference to the drawings. Figure 1 is a schematic diagram showing the overall configuration of the system. System 1 collects data on the exercise of user U who exercises at the exercise facility 10. Specifically, System 1 includes a server 100, a user terminal 200, a camera 300, etc.

[0011] The exercise facility 10 is a fitness gym, a gymnasium, etc. The exercise facility 10 is not limited to indoors; it can be outdoors as long as it is a facility where user U can exercise. The exercise facility 10 is also equipped with exercise equipment 500 and an authentication device 510.

[0012] The exercise equipment 500 may include a running machine, treadmill, walking machine, rowing machine, elliptical trainer, fitness bike, vibration machine, etc. The exercise equipment 500 may also be a weight training machine. The exercise equipment 500 may also be a stretching machine, massage chair, etc. User U can perform desired exercises using the exercise equipment 500. Note that the exercise equipment 500 is not limited to equipment with a motion mechanism for exercise; it may also be a yoga mat or fitness mat, etc. The exercise equipment 500 may be assigned a unique equipment ID. The exercises performed by User U may include squats, sit-ups, push-ups, etc., which do not require the use of equipment.

[0013] The authentication device 510 authenticates users of the sports facility 10. The authentication device 510 is installed at the entrance 11 of the sports facility 10, etc. In other words, the entrance 11 is a gate with an authentication function. The authentication device 510 identifies the user U entering the facility by performing authentication. This allows pre-registered users U to enter the sports facility 10. An ID is registered for each user U. The authentication device 510 may also be a device that uses biometric authentication such as facial recognition, fingerprint recognition, or voice recognition. In this case, the authentication device 510 is equipped with sensors according to the authentication method.

[0014] Furthermore, the authentication device 510 may be a device that uses QR codes (registered trademark). Alternatively, the authentication device 510 may perform authentication using a password or PIN. In this case, it is equipped with an input device such as a touch panel for user U to enter a password or PIN. When performing authentication using facial recognition or QR codes, the authentication device 510 is equipped with a camera to capture images of the face or QR code. When performing voice authentication, the authentication device 510 is equipped with a microphone or the like.

[0015] User U, authenticated by the authentication device 510, is also called the target user. In other words, the target user is User U, who has been authenticated by the authentication device 510 and entered the sports facility 500. To put it another way, the authentication device 510 extracts the target user who has entered the facility from the registered users who are users of the facility. The authentication device 510 then sends the authentication result to the server 100.

[0016] Camera 300 is installed in the sports facility 10 and captures images of user U. For example, multiple cameras 300 are installed on the walls and ceiling of the sports facility 10. Each of the multiple cameras 300 may be assigned a unique camera ID.

[0017] In the exercise machine 500, the camera 300 captures an image of the user U who has entered the exercise facility 10. Also, in the exercise facility 10, it is preferable to install the camera 300 so that all the exercise machines 500 can be imaged. Each camera 300 captures an image of the user U exercising on the exercise machine 500. It is desirable that the camera 300 be installed so that there are no blind spots within the exercise facility 10. Also, based on the image of the camera 300 and the authentication result of the authentication device 510, the user within the exercise facility 10 can be detected.

[0018] The server 100 is connected to the user terminal 200, the camera 300, and the authentication device 510 via a network. The server 100 receives various data from the camera 300, the authentication device 510, and the user terminal 200. The server 100 transmits various data to the camera 300, the authentication device 510, and the user terminal 200. The server 100 may be installed in the exercise facility 10 or at a remote location away from the exercise facility 10.

[0019] The server 100 is an information processing device equipped with a processor, a memory, etc., and performs predetermined processing on data. The server 100 performs processing for presenting exercise data indicating the activity level of the user U to the user U. The server 100 transmits the exercise data of the user U to the user terminal 200. The processing in the server 100 will be described later. At least a part of the processing in the server

[0020] 100 may be implemented by the camera 300, the user terminal 200, etc. For example, if the camera 300 has an arithmetic processing function, the camera 300 may perform processing such as the posture space described later.

[0020] In addition, the user terminal 200 is a smartphone, tablet terminal, personal computer, etc. held by the user U. The user terminal 200 includes a processor, a memory, a display, a touch panel, a microphone, a speaker, etc. The user terminal 200 may also be a wearable device such as a smartwatch or smart glasses. For example, each user U uses one or more user terminals 200. The user terminal 200 is associated according to the user ID information of the user U. When the user terminal 200 receives motion data from the server 100, it displays it on the display screen. The user U can check the exercise time, exercise content, energy consumption, etc. In this way, the system 1 can present motion data to the user U.

[0021] The processing in the server 100 will be described with reference to FIGS. 2 and 3. FIG. 2 is a control block diagram showing the configuration of the server 100. FIG. 3 is a schematic diagram showing an example of an image captured by the camera 300. In FIG. 3, two exercise devices 500 are identified as exercise devices 500A and 500B. In FIG. 3, the exercise device 500A is a stationary bike, and the exercise device 500B is a treadmill, but the number and types of the exercise devices 500 are not particularly limited. Also, in FIG. 3, two users U are identified as users U1 and U2. User U1 is exercising on the exercise device 500A, and user U2 is walking towards the exercise device 500B.

[0022] As shown in FIG. 2, the server 100 includes a specifying unit 110, an estimating unit 120, a presenting unit 130, a communication unit 140, and a database 150. The specifying unit 110 includes a person detection unit 111, an ID vector calculation unit 112, a joint coordinate estimation unit 113, and a user specifying unit 114. The estimating unit 120 includes a posture estimation unit 121, an ROI detection unit 122, a motion counter 123, and a motion time acquisition unit 124.

[0023] Database 150 stores user data for each user U. For example, the user data may include data such as user U's age, gender, height, and weight. Furthermore, database 150 may also store exercise data related to the user's movements. The exercise data includes feature data that shows the characteristics of the movement pattern when the user exercises. The feature data corresponds to time-series data of joint coordinates. For example, the feature data would show the trajectory of the foot coordinates when running. The characteristics of the movement pattern may be the stroke of joint positions during a back-and-forth motion.

[0024] Feature data and movement data are stored in database 150 for each user. The feature data and movement data stored in database 150 are used as reference data. Each user has different physiques, etc. Therefore, even if users perform the same exercise, the feature data will differ depending on the user. Feature data may be data obtained from images taken in the past, or it may be data estimated from the user's physical characteristics such as height and weight.

[0025] Furthermore, the database 150 stores data related to the sports facility 10. For example, the database 150 stores equipment data related to the exercise equipment 500 installed in the sports facility 10. The equipment data includes data such as the type, installation location, and number of exercise equipment 500. In addition, the installation location and imaging direction of the cameras 300 are associated with the exercise equipment. For example, the location and orientation of the exercise equipment 500 and the cameras 300 within the sports facility 10 are known. Therefore, it is possible to identify which camera 300 is imaging each piece of exercise equipment 500. Also, it is possible to identify the location of the exercise equipment 500 within the image captured by the cameras 300.

[0026] Therefore, the server 100 can identify the exercise area in the image from the camera 300 where the user performs exercises using the exercise equipment 500. Figure 3 shows the exercise area RA for exercise equipment 500A and the exercise area RB for exercise equipment 500B. The exercise areas RA and RB are identified as predetermined ranges in the image from the camera 300. The exercise areas RA and RB may also be identified by the camera ID of the camera 300 and the x and y coordinates of the camera 300. An exercise area is associated with each exercise equipment 500.

[0027] The identification unit 110 identifies a person based on the image data from the camera 300. For example, the camera 300 captures images of the entrance 11 and its surroundings. The identification unit 110 includes a person detection unit 111, an ID vector calculation unit 112, a joint coordinate estimation unit 113, and a user identification unit 114. The processing in the ID vector calculation unit 112, the joint coordinate estimation unit 113, and the user identification unit 114 will be described later.

[0028] The person detection unit 111 detects and tracks people who enter the room based on image data around the entrance 11. For example, the identification unit 110 detects people using deep learning technologies such as SSD (Single Shot MultiBox Detector) or YOLO (You Only Look Once). The person detection unit 111 tracks people using deep learning-based technologies such as Deepsort or common methods such as Kalman filters. Furthermore, the authentication device 510 authenticates users who enter through the entrance 11. Therefore, the identification unit 110 can identify the ID of user U who entered through the entrance 11. Figure 3 shows the person region B1 of user U1 and the person region B2 of user U2. Person regions B1 and B2 are frames that define the area in the image where a person (user) exists. For example, person region B1 is a bounding box surrounding user U1.

[0029] The estimation unit 120 estimates the user's activity level based on image data from the camera 300. The estimation unit 120 includes a posture estimation unit 121, an ROI detection unit 122, a motion counter 123, and an exercise time acquisition unit 124.

[0030] Camera 300 is capturing images of the exercise equipment 500 and its surroundings. The posture estimation unit 121 estimates the posture of user U using the exercise equipment 500 based on the images. In other words, it detects the posture of user U who is exercising in the exercise area. Figure 3 shows the estimated postures for user U1 and user U2, respectively. The posture estimation unit 121 estimates joint coordinates for the area where a person is located using posture estimation techniques such as Openpose and Transpose.

[0031] The posture estimation unit 121 identifies x and y coordinates (also called joint coordinates) indicating joint positions in each frame of the image from the camera 300. The posture estimation unit 121 then estimates the user's posture (skeleton) by connecting these joint coordinates. By estimating the posture at predetermined time intervals, the posture estimation unit 121 can acquire time-series data of the user's joint coordinates.

[0032] The ROI (Region of Interest) detection unit 122 detects ROIs (Regions of Interest) from the image based on the estimated posture. For example, it detects the area including joints such as the knee, ankle, elbow, and wrist joints of user U who is in the movement area as an ROI.

[0033] The motion counter 123 counts the number of joint movements in the ROI. For example, the motion counter 123 counts the number of movements based on time-series data of the x and y coordinates of the joint position. The motion counter 123 calculates the peak value of the coronary artery coordinate from the time-series data of the joint coordinates. More specifically, when a user exercises on a treadmill or running machine, the motion counter 123 counts the motion based on the coordinates of the toes. The motion counter 123 finds the peak value of the time-series data showing the change in the toe coordinates and counts the number of steps at the timing of the peak value. This allows the motion counter 123 to count the number of movements such as the number of steps.

[0034] When the user is in the movement area, the motion counter 123 increments the step count when the coordinates of the left and right toes reach their peak values. Of course, the motion counter 123 can also count the number of repetitions for walking, pedaling (leg-pedaling motion), etc., based on time-series data of joint positions. The motion counter 123 counts the number of times the joint coordinates move back and forth in the image as the number of movement actions. The motion counter 123 may also calculate the movement cycle and number of actions by performing a short-time Fourier transform on the time-series data.

[0035] Furthermore, it is preferable that the joints that become ROIs are set according to the camera 300, the exercise equipment 500, and the range of motion. For example, different joints move significantly depending on the movement. Therefore, the joints that move significantly are set to become ROIs. For example, on a cycling machine, the left and right toes, knee joints, and ankle joints are set to become ROIs. Similarly, on trekking machines, running machines, and walking machines, the left and right toes, wrist joints, elbow joints, etc., are set to become ROIs. In addition, the joints that become ROIs may be set according to the installation position and imaging direction of the camera 300. For example, joints that are in the blind spot of the camera 300 may be set not to become ROIs.

[0036] The exercise time acquisition unit 124 acquires each user's exercise time based on the image from the camera 300. For example, the exercise time acquisition unit 124 defines the time when the user enters the exercise area of ​​the exercise equipment 500 as the exercise start time and the time when the user exits the exercise area as the exercise end time. The exercise time acquisition unit 124 calculates the time from the exercise start time to the exercise end time as the exercise time. Alternatively, the exercise time may be calculated based on the time of movement counted by the motion counter 123 or time-series data of joint coordinates. Alternatively, the exercise time acquisition unit 124 may acquire the exercise time from the exercise equipment 500. For example, if the exercise equipment 500 is a treadmill or exercise bike with a timer or clock function, the server 100 can acquire the exercise time from the exercise equipment 500.

[0037] The display unit 130 presents the user with exercise data indicating the user's activity level. For example, the display unit 130 generates exercise data for each user that includes exercise content, type of exercise, exercise time, number of repetitions, degree of joint flexion, load intensity, energy expenditure, etc. The exercise data is not limited to the above data, as long as it indicates the amount of activity. The communication unit 140 transmits the exercise data indicating the user's activity level to the user terminal 200. The exercise data may be in a general-purpose format such as CSV format.

[0038] Furthermore, in order for the server 100 to perform the above processing, the communication unit 140 receives image data from the camera 300. The communication unit 140 also receives the authentication result from the authentication device 510. Data communication by the communication unit 140, etc., may be wireless communication such as Wi-Fi (registered trademark) or wired communication. When the user terminal 200 receives exercise data from the server 100, it displays it on the display. This allows user U to check their own exercise data. Thus, system 1 can support user U in leading a healthier life.

[0039] It is possible to identify the user U exercising without installing cameras 300 or personal authentication devices on each piece of exercise equipment 500. Furthermore, since it is only necessary to install the camera 300, which acts as a sensor, in the exercise facility 10, exercise data can be presented at a low cost.

[0040] Furthermore, as shown in Figure 4, there are cases where two or more users U intersect in the image. In other words, if occlusion occurs where users U intersect, there is a risk that system 1 may not be able to properly track the users. Because system 1 may incorrectly identify a user, there is a risk that motion data will be presented to the wrong user U. Therefore, in this embodiment 1, processing is performed to prevent the swapping of user IDs.

[0041] When the user domains of two or more users overlap, the ID vector calculation unit 112 determines a branching scene with two or more users as target users and calculates an ID vector for each target user ID in the overlapping user domains. For example, in a branching scene where occlusion occurs, the ID vector indicates the probability that a user of each ID exists in a single user domain. In other words, the ID vector calculation unit 112 finds the probability distribution of the existence of multiple users U in each user domain. If there are n (n is an integer of 2 or more) users U in the sports facility 10, the ID vector x is given by the following equation (1). x= [p1, p2, …, pi, .., pn] ··· (1)

[0042] p1 is the probability of the first user U1 being present, and pn is the probability of the nth user Un being present. pi is the probability of the i-th user Ui being present (i is any integer between 1 and n). Here, system 1 can detect users in the sports facility 10, and their number n, based on the authentication results and images from the authentication device 510. The sum of p1 to pn may be set to 1. An ID vector x is set for each person area.

[0043] The process of presenting motion data to the user using the ID vector x will be explained below with reference to Figures 2 and 5. Figure 5 is a flowchart of the process in System 1.

[0044] First, the authentication device 510 authenticates the user who has entered the room (S101). This allows the user ID of the user who has entered the room to be obtained. Next, the person detection unit 111 detects and tracks the user based on the image from the camera 300 (S102). In other words, person areas B1 and B2, as shown in Figures 3 and 4, are set for each user in the exercise facility 10.

[0045] At this time, the ID vector calculation unit 112 calculates the probability of a user being present in the sports facility 10. For example, when user U1 enters the room, in the human area around the entrance 11, a single user U1 is identified by the authentication result of the authentication device 510. Therefore, an initial ID vector x is set such that the probability p1 of the identified user U1 being present is 1, and the probabilities p2 to pn of the other users U2 to Un being present are each 0. Also, when a user leaves, the probability of the departing user being present is set to 0. In this way, when the authentication device 510 detects entry or exit, the ID vector calculation unit 112 may update the ID vector x. Also, when the authentication device 510 detects entry or exit, the dimensionality of the ID vector x may change. Then, the value of the ID vector is maintained until the branching scene through user tracking.

[0046] The ID vector calculation unit 112 updates the ID vector x in the branching scene (S103). If the overlap ratio or overlap area of ​​the two person regions exceeds a certain value, the ID vector calculation unit 112 determines that occlusion has occurred. When occlusion occurs, it becomes a branching scene in which the ID vector is updated. The ID vector calculation unit 112 updates the ID vector x for each of the two overlapping person regions. In other words, the ID vector calculation unit 112 changes the probability of existence of the target user, treating the user in the overlapping person region as the target user.

[0047] As shown in Figure 4, if human domains B1 and B2 overlap, the ID vector calculation unit 112 changes the existence probability of users U1 and U2, which are the target users. The ID vector calculation unit 112 decreases the existence probability p1 and increases the existence probability p2 in the ID vector x1 of human domain B1. For example, the ID vector calculation unit 112 decreases the existence probability p1 by multiplying it by a coefficient less than 1. Furthermore, the ID vector calculation unit 112 may increase the existence probability p2 by the amount obtained by subtracting the existence probability p1.

[0048] For example, if the ID vector x before duplication is [1, 0, 0, …, 0], then after duplication, the ID vector x becomes [0.5, 0.5, 0, …, 0]. Similarly, if person domain B1 and person domain B2 overlap, the existence probability p2 is decreased and the existence probability p1 is increased in the ID vector x2 of person domain B2. The timing of overlapping multiple person domains becomes a branching scene, and the ID vector calculation unit 112 updates the existence probability of the target user in the ID vector x. The target user can be the user with the highest existence probability.

[0049] If occlusion occurs multiple times for a single person's domain, the ID vector calculation unit 112 updates the ID vector x each time. Of course, if three or more person domains overlap, the probability of existence should be multiplied by a coefficient corresponding to the number of people. For example, if three person domains overlap, the probability of existence should be multiplied by 1 / 3. The ID vector calculation unit 112 may also change the rate at which the probability of existence is reduced depending on the size of the person, the orientation of the person, and the overlap rate. The ID vector calculation unit 112 may also update the ID vector using a network obtained through machine learning.

[0050] The joint coordinate estimation unit 113 estimates the user's joint coordinates (S104). Here, the joint coordinate estimation unit 113 can estimate the joint coordinates using the estimation results of the posture estimation unit 121. The joint coordinate estimation unit 113 acquires time-series data of the joint coordinates.

[0051] When the joint coordinates enter the movement range, the user identification unit 114 extracts characteristic data of the movement pattern and calculates the similarity probability of the movement pattern (S105). Because user U is in the movement range and performing movement, the user identification unit 114 can extract characteristic data of the movement pattern. As described above, characteristic data of the movement pattern is pre-set as reference data for each user. In addition, since time-series data of joint positions is obtained from the image of camera 300, the user identification unit 114 can extract characteristic patterns.

[0052] The user identification unit 114 compares the reference pattern with the extracted feature data and calculates the similarity probability of the patterns. The more similar the extracted feature data is to the reference pattern, the higher the similarity probability. The similarity probability is calculated for each user. The similarity probability vector y, which shows the similarity probability for n users, is given by the following equation (2). y=[q1, q2, …, qi, .., qn] ··· (2)

[0053] Here, 11 is the similarity probability for the first user U1, and qn is the similarity probability for the nth user Un. qi is the similarity probability for the i-th user Ui (where i is any integer between 1 and n). The sum of q1 to qn may be set to 1.

[0054] The user identification unit 114 identifies a specific user performing the movement based on the ID vector x and the similarity probability vector y (S106). For example, the user identification unit 114 calculates a matching probability vector P using the ID vector x and the similarity probability vector y. The matching probability vector P can be obtained by the following equation (3). P=[p1*q1, p2*q2, …, pi*qi, …, pn*qn] ··· (3)

[0055] The matching probability vector P, the ID vector x, and the similarity probability vector y have the same number of dimensions. The product of the existence probability p1 and the similarity probability q1 is the matching probability. The user identification unit 114 calculates the matching probability (pi*qi) for each user. The user identification unit 114 identifies the user whose ID has the largest product of the existence probability and the similarity probability as the specified user. In other words, the user identification unit 114 identifies the ID of the user performing the exercise using the following equation (4). ID=argmaxPi ··· (4)

[0056] When the user identification unit 114 identifies a specific user, the ID vector calculation unit 112 sets the probability of the specific user's existence in the ID vectors of other user domains to 0 (S107). For example, if the user identification unit 114 identifies user U1 as a specific user in the ID vector x of user domain B1, the ID vector calculation unit 112 sets the probability p1 of user U1's existence in the ID vectors x of user domains B2 to Bn to 0. In other words, since user U1 identified in one user domain B1 cannot exist in other user domains B2 to Bn, the ID vector calculation unit 112 can reduce the probability p1 of user U1's existence in user domains B2 to Bn.

[0057] Furthermore, in human domains B2 to Bn, the user identification unit 114 may increase the existence probabilities p2 to pn of other users U2 to Un by the amount by which existence probability p1 is subtracted. By doing so, the sum of existence probabilities in ID vector x can be kept constant. Depending on the identification result of the user identification unit 114, the ID vector calculation unit 112 can dynamically update the values ​​of multiple ID vectors x. In other words, since system 1 can appropriately determine ID vector x in multiple human domains, it can prevent incorrect user identification. Thus, robust processing against occlusion becomes possible.

[0058] When user U finishes exercising, the estimation unit 120 estimates the user's activity level (S108). For example, if the user moves outside a pre-set exercise area in the image, the server 100 detects the end of the exercise. The motion counter 123 also counts the number of exercises. Furthermore, the exercise time acquisition unit 124 acquires the exercise time from the exercise start time, exercise end time, etc. Therefore, the presentation unit 130 can estimate activity levels such as steps, distance traveled, and energy consumed based on these values. If the user performs two or more types of exercise, the estimation unit 120 may also estimate the activity level for each type of exercise.

[0059] The display unit 130 presents exercise data indicating the amount of activity to the user (S109). In other words, the communication unit 140 transmits the exercise data of a specific user to the user terminal 200 of that user. The exercise data may be shown as a value for each type of exercise, or as a value that combines multiple exercises. The exercise data may include the start time and end time of the exercise.

[0060] Users can view exercise data related to the exercises they perform. Therefore, users can improve their motivation to exercise and lead healthier lives. System 1 can automatically collect exercise data and present it appropriately to the user.

[0061] The user identification unit 114 may also calculate the similarity probability based on body data that indicates the user's physical characteristics. For example, shoulder width, arm length, leg length, and torso length differ depending on the user. Therefore, values ​​such as shoulder width, arm length, leg length, and torso length can be used as body data from joint positions. The joint coordinate estimation unit 113 or the posture estimation unit 121 extracts the body data shown from the image of the camera 300. The user identification unit 114 may then calculate the similarity probability by comparing the body data with user data. For example, the closer the leg and arm lengths are, the higher the similarity probability. The user identification unit 114 may also calculate a similarity probability vector y by integrating the similarity probabilities of the body data features and the similarity probabilities of the movement pattern features.

[0062] The ID vector calculation unit 112 may calculate the ID vector x using a Kalman filter. For example, as shown in Figure 6, the ID vector calculation unit 112 predicts the prior error distribution D2 from the posterior distribution D1 at one time step (t-1). The prior error distribution D2 can be predicted from the user U's movement speed and the distance moved in one frame. The ID vector calculation unit 112 can determine the user's movement distance and movement speed by comparing multiple frame images. The ID vector calculation unit 112 obtains the observed distribution D3 based on the image from the camera 300. The ID vector calculation unit 112 updates the posterior distribution D4 based on the prior error distribution D2 and the observed distribution D3. The posterior distribution D4 shows the distribution at the current time t.

[0063] The ID vector calculation unit 112 uses the central probability of the posterior distribution D4 to determine the ID vector. In this way, the ID vector calculation unit 112 can accurately track the user by using a Kalman filter. Therefore, the ID vector calculation unit 112 can appropriately determine the ID vector. Of course, the calculation of the ID vector is not limited to the process using a Kalman filter; deep learning-based technologies and other methods can also be used.

[0064] It should be noted that the present invention is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention. Furthermore, this disclosure can be implemented by having a processor such as a CPU (Central Processing Unit) execute a computer program to perform some or all of the control processing in System 1. For example, the server 100, user terminal 200, etc., can be implemented as a device capable of executing a program, such as the central processing unit of a computer. Various functions can also be implemented by program. [Explanation of Symbols]

[0065] 1 System 100 servers 200 user terminals 300 Cameras 500 exercise equipment U User

Claims

1. A camera installed in a sports facility where a user exercises, which captures images of the user, An estimation means for estimating the user's activity level based on the image data of the camera, A means for identifying the user captured by the camera, The system includes a presentation means for presenting exercise data indicating the user's activity level to the identified user, The aforementioned specifying means is, From the image data of the aforementioned camera, extract the area containing people. When the user domains of two or more users overlap, a branching scene is created where the two or more users are the target users. This branching scene calculates an ID vector representing the probability for each ID of the target user for the overlapping user domains. Based on the image data from the aforementioned camera, the joint coordinates of the user during movement are estimated. From the time-series data of the joint coordinates, feature data indicating the characteristics of the user's movement pattern is extracted. Based on the aforementioned feature data, a similarity probability vector is calculated for each target user, indicating the similarity probability of similar features. A data collection system that identifies the user based on the similarity probability vector and the ID vector.

2. When the user included in one person domain is identified as a specific user, The data collection system according to claim 1, wherein the probability of the specific user in the ID vector corresponding to the domain of another person is set to 0.

3. The aforementioned specifying means is, From the image data of the aforementioned camera, physical data indicating the user's physical characteristics is extracted. A data collection system according to claim 1 or 2, which identifies the user based on the aforementioned physical data.

4. If the exercise load changes during the user's exercise, A data collection system according to claim 1 or 2, which extracts feature data based on the time-series data when the load is high.

Images