Stationary determination system and computer program

Abstract

To provide a system for determining a user movement and a computer program therefor which can stably determine a user movement.SOLUTION: The present invention is directed to a system for determining whether or not a user body is stationary. The system has a photographing unit, a site specifying unit, an image region setting unit, and a stationary determining unit. The photographing unit acquires photographed data including a plurality of frames by photographing the user movement. The site specifying unit specifies a predetermined site of the user body with respect to the acquired photographed data. The image region setting unit sets an image region corresponding to the specified predetermined site. The stationary determining unit determines whether or not the predetermined site is stationary based on a difference between frames of the image region.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to a stationary determination system and a computer program.

Background Art

[0002] Patent Document 1 discloses a method of acquiring an image of a user performing a motion operation, generating a human motion screening score based on the image, and generating a personalized exercise program based on the score.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In evaluating such user operations, it has been necessary to stably perform stationary determination of the body.

[0005] The present invention has been made in view of such circumstances, and provides a stationary determination system and a computer program that enable stable stationary determination of user operations.

Means for Solving the Problems

[0006] According to the present invention, there is provided a system for determining whether a user's body is stationary, comprising a photographing unit, a part specifying unit, an image area setting unit, and a stationary determination unit. The photographing unit captures the user's actions to obtain photographing data including a plurality of frames. The part specifying unit specifies a predetermined part of the user's body in the obtained photographing data. The image area setting unit sets an image area associated with the specified predetermined part. The stationary determination unit determines whether the predetermined part is stationary based on the difference between frames of the image area.

[0007] With such a configuration, it becomes possible to stably perform a stationary determination for a specific part of the user's body.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Modes for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Various features shown in the embodiments described below can be combined with each other. Further, an invention can be made independently for each feature item.

[0010] <1. First Embodiment> (1-1. Overall Configuration of Determination System 1) As shown in FIG. 1, the determination system 1 according to the first embodiment of the present invention includes a mobile terminal 10. The mobile terminal 10 can be a smartphone, a tablet terminal, or the like that can install a plurality of application programs.

[0011] In the present embodiment, the mobile terminal 10 is provided with a camera 20 and is arranged to photograph the user U from the front side. Note that the configuration shown in FIG. 1 is an example, and the mobile terminal 10 may be arranged at a position other than the front side of the user U.

[0012] (1-2. Configuration of Mobile Terminal 10) As shown in FIG. 2, the housing 12 of the mobile terminal 10 is provided with a touch panel display 14 on its front surface 12A, which is an image display means for displaying an image on the screen. The touch panel display 14 includes, for example, an LCD (Liquid Crystal Display) and a touch sensor. The LCD displays various images, and the touch sensor receives various input operations performed using an indicator such as a finger, a stylus, or a pen.

[0013] Also, on the front surface 12A of the mobile terminal 10, a microphone 16 for inputting sound, a speaker 18 for outputting sound, and a camera 20 for imaging a subject are provided. The camera 20 is provided not only on the front surface 12A of the housing 12 but also on the back surface of the housing 12. Further, on the side surface 12B of the housing 12, operation buttons 22 such as a power button for starting or stopping the mobile terminal 10 and a volume adjustment button for the sound output by the speaker 18 are provided.

[0014] In addition, the housing 12 of the mobile terminal 10 is provided with a slot into which a memory card is inserted, a USB (Universal Serial Bus) terminal, and the like.

[0015] FIG. 3 is a functional block diagram showing the electrical configuration of the mobile terminal 10.

[0016] In addition to the above configuration, the mobile terminal 10 includes a control unit 24, a storage unit 26, and a communication unit 28.

[0017] The control unit 24 is, for example, a CPU (Central Processing Unit), a microprocessor, a DSP (Digital Signal Processor), etc., and controls the overall operation of the mobile terminal 10.

[0018] The storage unit 26 includes, for example, a RAM (Random Access Memory), a DRAM (Dynamic Random Access Memory), etc., and is used as a work area when executing processes based on various programs by the control unit 24.

[0019] The storage unit 26 further includes a non-volatile memory such as a flash memory, and stores various data such as images and programs used for the processing of the control unit 24. The programs stored in the storage unit 26 are, for example, an OS (Operating System) for realizing the basic functions of the mobile terminal 10, drivers for controlling various hardware, application programs for realizing functions such as e-mail and web browsing, and other various functions. In addition, a program for executing the determination method according to the present embodiment, the details of which will be described later, is stored in the storage unit 26 in advance.

[0020] The communication unit 28 is, for example, a NIC (Network Interface Controller) and has a function of connecting to a communication network such as a mobile phone network. Note that the communication unit 28 may have a function of connecting to a wireless LAN (Local Area Network), a function of connecting to a wireless WAN (Wide Area Network), a function enabling short-distance wireless communication such as Bluetooth (registered trademark), and infrared communication, instead of or together with the NIC.

[0021] These control unit 24, storage unit 26, communication unit 28, touch panel display 14, microphone 16, speaker 18, camera 20, and operation buttons 22 are electrically connected to each other via a system bus 30. Therefore, the control unit 24 can access the storage unit 26, display an image on the touch panel display 14, grasp the operation state of the touch panel display 14 and the operation buttons 22 by the user, input sound to the microphone 16, output sound from the speaker 18, control the camera 20, and access various communication networks and other information processing devices via the communication unit 28, etc.

[0022] (1-3. Functional block diagram) FIG. 4 is a functional block diagram regarding the determination function according to the present embodiment.

[0023] As shown in FIG. 4, the control unit 24 includes a photographing unit 31, a part specifying unit 32, an image area setting unit 33, and a stillness determination unit 34.

[0024] The photographing unit 31 photographs the operation of the user U and acquires photographing data including a plurality of frames. The part specifying unit 32 specifies a predetermined part of the user U that is the target of stillness determination for the acquired photographing data.

[0025] The image area setting unit 33 sets an image area corresponding to the specified predetermined part. The stillness determination unit 34 determines whether the specified predetermined part is still based on the difference between frames of the image area.

[0026] Each functional configuration included in the above-described control unit 24 is realized by a computer program stored in the storage unit 26. Further, the computer program may be stored in a computer-readable non-transitory recording medium and read from the recording medium as needed.

[0027] (1-4. Flow of determination process) Referring to FIG. 5, the flow of the stillness determination process using the determination system 1 will be described in detail.

[0028] In step S110, the imaging unit 31 images the operation of the user U. Here, when the imaging is performed at N frames per second for T seconds, the number of frames M included in the imaging data is calculated as N × T. N and T are positive real numbers, and M is a natural number. Each frame includes a still image. N is, for example, 60, and T is, for example, 5. In this case, the imaging data will include 300 (M = 60 × 5 = 300) still images.

[0029] In step S120, the part specifying unit 32 specifies the part of the user U that is the object of the stillness determination for the acquired imaging data. This specification can be performed, for example, based on a pose model by two-dimensional pose estimation for each still image constituting the imaging data.

[0030] In two-dimensional pose estimation, for example, by an algorithm using deep learning, the coordinates of a plurality of feature points of the user U in the still image are estimated, and thus the pose of the user U can be estimated two-dimensionally. Further, by performing two-dimensional pose estimation on a plurality of still images, it is also possible to measure the movement of the skeleton of the user U.

[0031] Figure 6 shows an example of the arrangement of a plurality of feature points. Figure 6 illustrates an image of the user U standing on one foot with the right foot. Here, the feature points 0 to 14 respectively correspond to the head (0), neck (1), right shoulder (2), right elbow (3), right hand (4), left shoulder (5), left elbow (6), left hand (7), waist (8), right hip joint (9), right knee (10), right ankle (11), left hip joint (12), left knee (13), and left ankle (14).

[0032] In this way, when the coordinates of the feature points 0 to 14 of the user U in each still image are obtained by two-dimensional pose estimation, the pose of the user can be estimated based on these coordinates. Note that the number and arrangement of the feature points are not limited to those in this embodiment and can be changed as appropriate.

[0033] In this way, the coordinates of the plurality of feature points obtained by two-dimensional pose estimation for each still image constituting each piece of shooting data can be acquired as measurement data. Here, when the shooting data is composed of M frames, the measurement data corresponding to the shooting data is also composed of M frames.

[0034] The part specifying unit 32 specifies a predetermined part on the body of the user U based on the acquired measurement data (that is, the coordinates of a plurality of feature points). Here, the part specifying unit 32 may change the part to be specified according to the operation of the user U. In this case, the part specifying unit 32 does not need to acquire measurement data for all the feature points, and may acquire measurement data only for the part to be specified for each operation performed by the user U.

[0035] In step S130, the image area setting unit 33 sets an image area D associated with a specified predetermined part. When setting the image area D, feature points corresponding to each operation of the user U are stored in the storage unit 26. As an example, for the operation of standing on one foot with the right foot as shown in FIG. 7, feature points 1 to 7 of the upper body part are associated as a predetermined part. Therefore, the image area setting unit 33 refers to the storage unit 26 and sets an image including feature points 1 to 7 as the image area D. Further, for example, when the part to be specified is the upper body, the image area setting unit 33 may set an image above feature point 8 (waist) as the image area D.

[0036] Also, in another example shown in FIG. 8, the case where the user U extends the left arm is illustrated. In such a case, in order to estimate the position of the left arm from the coordinate positions of feature points 5 to 7, an image including feature points 5 to 7 may be set as the image area D.

[0037] In step S140, the stillness determination unit 34 determines whether or not a specified predetermined part is still based on the difference between frames of the image area. Specifically, the stillness determination unit 34 calculates the difference in gradation values for each pixel included in the image area for three consecutive frames in the acquired imaging data, and determines whether or not a specified predetermined part is still.

[0038] Hereinafter, a specific explanation will be given with reference to FIG. 9. FIG. 9 shows three consecutive images G1 to G3 selected from the acquired imaging data. It is assumed that the imaging data is composed of 300 still images (M = 300) and is composed of a start frame F1 to an end frame F300. The images G1 to G3 respectively correspond to frames Fn to Fn+2 (1 ≤ n ≤ 298).

[0039] The images G1 to G3 respectively include image areas D1 to D3 set by the image area setting unit 33. Here, the image areas D1 to D3 have gradation values from 0 to 255 for each of a plurality of pixels. That is, they can be represented by the following (Equation 1) to (Equation 3).

[0040]

Number

[0041]

Number

[0042]

Number

[0043] Here, D1(i, j) represents each pixel included in the image area D1. The operator "∋" represents the possible values. The stillness determination unit 34 calculates the absolute value of the difference for each continuous image area of frames as the difference images F12 and F23.

[0044]

Number

[0045]

Number

[0046] Next, the stillness determination unit 34 calculates the logical product for the difference images F12 and F23 to calculate the logical product image Fa.

[0047]

Number

[0048] Here, the operator "∩" represents the intersection. Next, the stillness determination unit 34 performs binarization processing by comparing with an arbitrary threshold T to calculate the binarized difference image Ft.

[0049]

Number

[0050] Through the above processing, in the image regions D1 to D3, the portions (pixels) determined to be stationary have a gradation value of 255, and the pixels determined not to be stationary have a gradation value of 0. Note that the threshold value T may be appropriately set according to the accuracy required in the stationary determination.

[0051] Next, the sum F of the pixel values in the difference image Ft is calculated, and the determination value Fv for the stationary determination is calculated.

[0052]

Equation

[0053] As described above, for the frames Fn to Fn+2 (1 ≤ n ≤ 298), a plurality (298 in this embodiment) of determination values Fv are obtained. By performing a predetermined statistical process on this determination value Fv and comparing it with a predetermined threshold value, a stationary determination is made. The statistical process here is, for example, a process of calculating a statistically obtained value such as a simple sum, a weighted sum, or a value based thereon (simple average, weighted average).

[0054] As described above, the determination system 1 according to this embodiment includes a photographing unit 31, a part specifying unit 32, an image region setting unit 33, and a stationary determination unit 34. The photographing unit 31 photographs the operation of the user and acquires photographing data including a plurality of frames. The part specifying unit 32 specifies a predetermined part of the user's body from the acquired photographing data.

[0055] The image region setting unit 33 sets an image region associated with the specified predetermined part. The stationary determination unit 34 determines whether a predetermined part is stationary based on the difference between frames of the image region.

[0056] With such a configuration, it is possible to suppress the influence of blurring at the time of detection and to stably perform a stationary determination for a specific part of the user's body.

[0057] <2. Second Embodiment> Referring to FIG. 10, the determination system 1 according to the second embodiment will be described. The determination system 1 according to the second embodiment is different from the first embodiment in that it includes a master unit 10a and a slave unit 10b as mobile terminals 10. Hereinafter, the description will focus on the differences from the first embodiment.

[0058] As shown in FIG. 10, in this embodiment, the slave unit 10b is arranged on the right side, left side, and rear side of the user U. The slave unit 10b is also composed of a mobile terminal 10, similar to the master unit 10a. The master unit 10a and the slave unit 10b are configured to be communicable through the communication unit 28. The slave unit 10b functions as a photographing unit 31, photographs the actions of the user U from a plurality of directions, and acquires a plurality of pieces of photographing data.

[0059] The photographing data acquired by the slave unit 10b is transmitted to the master unit 10a. The part specifying unit 32 of the master unit 10a specifies a predetermined part for each of the acquired plurality of pieces of photographing data.

[0060] The stillness determination unit 34 sets a weighting based on the photographing direction with respect to the user U for the photographing data acquired from a plurality of directions. The stillness determination unit 34 determines whether a predetermined part of the user is still based on the difference between frames of the image area set by the image area setting unit 33 and the said weighting.

[0061] With such a configuration, it becomes possible to analyze whether the user's body is still based on a plurality of pieces of photographing data, and the stillness determination can be performed more stably.

[0062] Here, preferably, the stillness determination unit 34 may set a weighting based on the photographing direction according to the action of the user U being photographed. The weighting based on the photographing direction is set in association with the specified predetermined part. That is, in the action of the user U, it is conceivable to set a large weighting for the photographing data from the direction in which movement is easily detected for each predetermined part.

[0063] With such a configuration, it becomes possible to change the weighting based on the shooting direction according to the user's operation, and the stillness determination can be performed more stably.

[0064] <3. Other Embodiments> As described above, the embodiments have been explained, but the scope of application of the technical scope of the present application is not limited to the above embodiments.

[0065] For example, the stillness determination unit 34 may be configured to select, as determination target frames, three frames selected based on the determination value for stillness determination from among the consecutive L frames included in the acquired shooting data.

[0066] In this case, the stillness determination unit 34 calculates, for example, the determination values for stillness determination for P sets of consecutive three frames (L = 3) with different starting frames. Here, P is a natural number. The stillness determination unit 34 may select, as determination target frames, frames for which the rank when the P determination values are arranged in ascending order of the small movement of the user U is within the Q-th place. Note that L can also be a value other than 3.

[0067] For example, if P = 90 and Q = 10, ten determination values Fv with small values are selected from among the 90 determination values Fv. Also, if P = 90 and Q = 1, and the determination value Fv of three consecutive frames starting from frame F121 (that is, frames F121 to F123) is the smallest, then frames F121 to F123 become the determination target frames. Note that the determination value Fv may be calculated for three consecutive frames starting from frame F0, but the data immediately after the start of imaging has low reliability. That is, the reliability of the determination may be increased by excluding the first plurality of frames from the measurement data and specifying the determination target frames.

[0068] In addition, when selecting the evaluation target frame in the case of shooting from a plurality of directions as in the second embodiment, the determination value Fv for stillness determination may be calculated from the shooting data from the shooting direction with the largest weighting, and the determination target frame may be selected. In this case, the determination value Fv of the frames in other shooting directions corresponding to the selected determination target frame is calculated, and stillness determination is performed in consideration of the weighting based on the shooting direction for the determination value Fv in each direction.

[0069] In the above embodiment, the determination system 1 is configured using the mobile terminal 10 such as a smartphone or a tablet. However, at least one of the devices constituting the determination system 1 may be any device (e.g., a web camera) that is communicable and has a shooting function.

[0070] Furthermore, the present invention can also be realized as a program for causing a mobile terminal to function to realize the above-described determination system.

[0071] Furthermore, the present invention can also be realized as a computer-readable non-transitory recording medium storing the above-described program.

[0072] <4. Features of the Embodiment> Hereinafter, the features of the embodiments of the present invention will be summarized.

[0073] The determination system 1 includes a shooting unit 31, a part specifying unit 32, an image area setting unit 33, and a stillness determination unit 34. The shooting unit 31 shoots the operation of the user and acquires shooting data including a plurality of frames. The part specifying unit 32 specifies a predetermined part of the user's body from the acquired shooting data. The image area setting unit 33 sets an image area associated with the specified predetermined part. The stillness determination unit 34 determines whether a predetermined part is still based on the difference between the frames of the image area.

[0074] With such a configuration, it is possible to suppress the influence of blurring at the time of detection and to stably perform stillness determination for a specific part of the user's body.

[0075] The site specifying unit 32 included in the determination system 1 may specify a predetermined site on the body of the user U based on the posture model estimated from the photographed data.

[0076] With such a configuration, it is possible to stably identify the sites on the user's body.

[0077] The site specifying unit 32 may specify a predetermined site based on the operation of the user U.

[0078] With such a configuration, the site to be specified for each operation can be set, and the site can be specified more stably.

[0079] A plurality of frames of the photographed data are composed of M frames. The stillness determination unit 34 calculates a determination value Fv for the degree of stillness of a predetermined site based on the difference between frames for P sets of consecutive L frames starting from different frames in the photographed data, and determines whether or not the specified predetermined site is stationary based on the L frames whose rank is within Q when the P determination values Fv are arranged in ascending order of the small movement of the predetermined site.

[0080] With such a configuration, the object of the stillness determination can be narrowed down, and the stillness determination can be performed more stably.

[0081] The photographing unit 31 photographs the operations of the user from a plurality of directions to obtain a plurality of photographed data, and the site specifying unit 32 may specify the predetermined site for each of the obtained plurality of photographed data.

[0082] With such a configuration, a plurality of photographed data can be analyzed, and the stillness determination can be performed more stably.

[0083] The stillness determination unit 34 may set a weighting based on the shooting direction with respect to the user U for a plurality of pieces of shooting data, and determine whether a predetermined part is still based on the difference between frames of the image area and the weighting of the shooting data.

[0084] With such a configuration, weighting in the shooting direction is performed on a plurality of data, and stillness determination can be performed more stably.

[0085] The stillness determination unit 34 may set a weighting based on the shooting direction according to the operation of the user U.

[0086] With such a configuration, the weighting in the shooting direction is changed according to the operation, and stillness determination can be performed more stably.

[0087] Although various embodiments according to the present invention have been described, these are presented as examples and are not intended to limit the scope of the invention. The novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. The embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and the equivalent scope thereof.

Explanation of Reference Numerals

[0088] 1: Determination system 10: Mobile terminal 10a: Master device 10b: Slave device 11: Feature point 12: Housing 12A: Front 12B: Side 13: Feature point 14: Touch panel display 16: Microphone 18: Speaker 20: Camera 22: Operation button 24: Control Unit 26: Memory Unit 28: Communication Unit 30: System Bus 31: Imaging Unit 32: Region Specifying Unit 33: Image Region Setting Unit 34: Still Image Judgment Unit U: User

Claims

**Claim 1** A system comprising a processor, the processor being configured to: obtain capture data including a plurality of frames capturing a user's motion from a plurality of directions; perform weighting of the capture data based on the capture directions; determine stillness of a predetermined part corresponding to a predetermined part specified from among a plurality of parts of the user's body based on a pose model estimated by an algorithm using deep learning, based on a difference between frames of an image region that is part of the capture data and the weighting; a system. **Claim 2** A program causing a processor to obtain capture data including a plurality of frames capturing a user's motion from a plurality of directions, cause the processor to perform weighting of the capture data based on the capture directions, and cause the processor to determine stillness of a predetermined part corresponding to a predetermined part specified from among a plurality of parts of the user's body based on a pose model estimated by an algorithm using deep learning, based on a difference between frames of an image region that is part of the capture data and the weighting. a program. **Claim 3** A method, comprising: obtaining, by a processor, capture data including a plurality of frames capturing a user's motion from a plurality of directions; performing, by the processor, weighting of the capture data based on the capture directions; determining, by the processor, stillness of a predetermined part corresponding to a predetermined part specified from among a plurality of parts of the user's body based on a pose model estimated by an algorithm using deep learning, based on a difference between frames of an image region that is part of the capture data and the weighting. a method.

Images