Posture estimation system, posture estimation device, posture estimation method, learning device, and trained model
Patent Information
- Application Number
- JP2020188818
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-11-12
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2040-11-12
AI Technical Summary
【0017】 本発明によれば、車いす及び乗車者が撮像された画像から当該車いす及び乗車者の姿勢を一体として推定できる。
Smart Images

Figure 0007594778000001 
Figure 0007594778000002 
Figure 0007594778000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a posture estimation system, a posture estimation device, a posture estimation method, a learning device, and a trained model. [Background technology]
[0002] A technique for estimating a posture of a person from a captured image of the person has been disclosed (for example, see Patent Document 1). Also, a technique for estimating a posture of an object from a captured image of the object has been disclosed (for example, see Patent Document 2). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2008-146583 A [Patent Document 2] JP 2008-003794 A Summary of the Invention [Problem to be solved by the invention]
[0004] For example, in the field of sports, in baseball, tennis, soccer, and other sports, the posture of players is detected and used for motion analysis and skill evaluation. Such motion analysis and skill evaluation of players is also desired in parasports such as wheelchair tennis and wheelchair basketball. Since wheelchair tennis and wheelchair basketball are sports in which the rider and horse are one, it is necessary to estimate the posture of the player and wheelchair as a whole.
[0005] However, the technology described in the above-mentioned Patent Document 1 or Patent Document 2 can only estimate the posture of a person from an image, or only the posture of an object from an image, so it is not possible to estimate the posture of the wheelchair and the wheelchair rider (athlete) together. If the postures of the wheelchair and the rider are estimated separately, the amount of calculation increases, requiring more time for processing, and there is a risk that deviations will occur in the estimated coordinates and orientations of the wheelchair and the rider, reducing the estimation accuracy.
[0006] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide a posture estimation system, posture estimation device, posture estimation method, learning device, and trained model that can estimate the posture of a wheelchair and a occupant together from images of the wheelchair and the occupant. [Means for solving the problem]
[0007] The present invention has been made to solve the above-mentioned problems, and one aspect of the present invention is a posture estimation system including an estimation unit that estimates coordinate information of one or more body parts of the wheelchair and the occupant from a second captured image of the wheelchair and the occupant, based on a plurality of first captured images of the wheelchair and the occupant of the wheelchair and coordinate information of one or more body parts of the wheelchair and the occupant captured in each of the plurality of first captured images.
[0008] The posture estimation system further includes a learning unit that generates a trained model for estimating the coordinate information of one or more body parts of the wheelchair and the occupant from a second captured image of the wheelchair and the occupant, by machine learning using a dataset including a plurality of first learning images of the wheelchair and the occupant and coordinate information of one or more body parts of the wheelchair and the occupant captured in each of the plurality of first learning images, and the estimation unit may use the trained model generated by the learning unit to estimate the coordinate information of one or more body parts of the wheelchair and the occupant from the second captured image captured by the imaging unit.
[0009] In addition, the posture estimation system may include an extraction unit that extracts partial images of each of the wheelchairs and the occupants from a second captured image in which the wheelchairs and the occupants are captured, and the estimation unit may estimate coordinate information of one or more parts of the wheelchairs and the occupants from each of the partial images.
[0010] The posture estimation system may further include a posture information generation unit that generates posture information regarding a posture of the wheelchair and the occupant based on the coordinate information estimated by the estimation unit.
[0011] The posture estimation system may also include an evaluation index derivation unit that derives an evaluation index for evaluating a skill of the passenger based on the posture information generated by the posture information generation unit.
[0012] Furthermore, in the above posture estimation system, the estimation unit may estimate coordinate information of at least one or more parts of the object from a second captured image in which the wheelchair, the occupant, and the object are captured, based on a plurality of first captured images in which the wheelchair, the occupant, and an object held or worn by the occupant, and coordinate information of one or more parts of the wheelchair, the occupant, and the object captured in each of the plurality of captured images, and the evaluation index derivation unit may derive the evaluation index based on the coordinate information of at least one or more parts of the object.
[0013] Another aspect of the present invention is a posture estimation device that includes an estimation unit that estimates coordinate information of one or more body parts of the wheelchair and the occupant from a second captured image of the wheelchair and the occupant, based on a plurality of first captured images of the wheelchair and the occupant of the wheelchair and coordinate information of one or more body parts of the wheelchair and the occupant captured in each of the plurality of first captured images.
[0014] Another aspect of the present invention is a posture estimation method in a posture estimation system, the posture estimation method comprising the steps of: an acquisition unit acquiring an image including a wheelchair and an occupant of the wheelchair; and an estimation unit estimating coordinate information of one or more body parts of the wheelchair and the occupant from a second captured image acquired by the acquisition unit using a trained model machine-learned based on a plurality of first learning captured images in which the wheelchair and the occupant are imaged and coordinate information of one or more body parts of the wheelchair and the occupant captured in each of the plurality of learning captured images.
[0015] Another aspect of the present invention is a learning device that includes a learning unit that generates a trained model for estimating coordinate information of one or more body parts of the wheelchair and the occupant from a second captured image of the wheelchair and the occupant by performing machine learning using a dataset that includes a plurality of first captured images for learning in which a wheelchair and an occupant of the wheelchair are captured, and coordinate information of one or more body parts of the wheelchair and the occupant captured in each of the plurality of first captured images for learning.
[0016] Moreover, one aspect of the present invention is a trained model for estimating coordinate information of one or more body parts of a wheelchair and an occupant of the wheelchair, the trained model being machine-learned based on a plurality of first learning images of the wheelchair and the occupant and coordinate information of one or more body parts of the wheelchair and the occupant captured in each of the plurality of first learning images, and for causing a computer to function to estimate the coordinate information of one or more body parts of the wheelchair and the occupant from a second image of the wheelchair and the occupant. Effect of the Invention
[0017] According to the present invention, the postures of a wheelchair and a passenger can be estimated together from images of the wheelchair and the passenger. [Brief description of the drawings]
[0018] [Figure 1] FIG. 4 is a diagram showing an example of a flow of a learning stage according to the first embodiment. [Diagram 2] FIG. 1 is a diagram showing an example of a flow of a utilization stage of a trained model according to the first embodiment. [Diagram 3] 1 is a block diagram showing an example of the configuration of a posture estimation system according to a first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of coordinate information data according to the first embodiment. [Diagram 5] FIG. 4 is a diagram showing an example of posture information data according to the first embodiment. [Figure 6] FIG. 4 is an image diagram showing an example of an output value of posture information according to the first embodiment. [Figure 7] An explanatory diagram explaining the steps involved in machine learning using CNN. [Figure 8] An explanatory diagram explaining the learning procedure of machine learning using CNN. [Figure 9] A diagram showing an example of the network structure of Mask R-CNN. [Figure 10] FIG. 11 is a block diagram showing an example of the configuration of a posture estimation system according to a second embodiment. [Figure 11] FIG. 11 is a diagram showing a first example of derivable evaluation indexes according to the second embodiment. [Figure 12] FIG. 11 is a diagram showing a second example of derivable evaluation indexes according to the second embodiment. [Figure 13] FIG. 11 is a diagram showing a third example of derivable evaluation indexes according to the second embodiment. [Figure 14] FIG. 11 is a diagram showing a fourth example of derivable evaluation indexes according to the second embodiment. [Figure 15] FIG. 11 is a diagram showing a fifth example of derivable evaluation indexes according to the second embodiment. [Figure 16] FIG. 11 is a diagram showing a sixth example of a derivable evaluation index according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0019] Hereinafter, one embodiment of the present invention will be described with reference to the drawings. <First embodiment> First, a first embodiment of the present invention will be described. A posture estimation system 1 according to this embodiment estimates the posture of a wheelchair and a person riding in the wheelchair (hereinafter referred to as a "passenger") from a captured image of the wheelchair and the passenger. For example, the posture estimation system 1 performs machine learning based on a plurality of first captured images for learning in which the wheelchair and the passenger are captured, and coordinate information of specific parts of the wheelchair and the passenger captured in each of the plurality of first captured images for learning, and generates a trained model for estimating the posture of the wheelchair and the passenger.
[0020] The specific body parts of the wheelchair and the occupant are preset parts including one or more body parts of the wheelchair and the occupant. The coordinate information of the body parts of the occupant includes, for example, the coordinate information of the neck, shoulder, elbow, wrist, armpit, thigh, knee, heel, etc. The coordinate information of the body parts of the wheelchair includes, for example, the axle of the wheelchair, the contact point of the wheels, etc.
[0021] Then, the posture estimation system 1 uses this trained model to estimate coordinate information of multiple parts of the wheelchair and the occupant from a second captured image of the wheelchair and the occupant captured by an imaging device (e.g., a video camera). This allows the posture estimation system 1 to simultaneously estimate the postures of the wheelchair and the occupant as a whole based on their respective coordinate information.
[0022] First, an overview of posture estimation system 1 will be described with reference to Figures 1 and 2. Pose estimation system 1 is divided into two stages: a learning stage in which a trained model is generated, and a utilization stage in which the generated trained model is utilized. 1 is a diagram showing an example of a flow of a learning stage according to the present embodiment. The learning stage is further divided into two stages: generation of a learning dataset and generation of a trained model using the learning dataset.
[0023] In the stage of generating a learning dataset, a wheelchair and a passenger are photographed and a plurality of first captured images (raw data) for learning are prepared (step S101), and the plurality of first captured images for learning are processed to generate a learning dataset (step S102). The plurality of first captured images for learning are a plurality of still images of the wheelchair and the passenger photographed by an imaging device (such as a camera), or a plurality of frame images included in a video of the wheelchair and the passenger photographed by an imaging device (such as a video camera). The first captured images may be a part or all of a second captured image described later. For example, for each of the plurality of first captured images for learning, a developer creates three-dimensional coordinate information in which the three-dimensional coordinates of a plurality of parts of the wheelchair and the passenger are annotated, and generates a learning dataset in which the first captured image and the three-dimensional coordinate information are paired. The plurality of parts of the wheelchair and the passenger are predetermined parts, such as parts corresponding to human joints and parts related to the movable structure of the wheelchair.
[0024] In the stage of generating the trained model, the posture estimation system 1 inputs the generated training data set into a training program to perform machine learning (step S103), and outputs the trained model (step S104). As the training program, for example, a convolution neural network (CNN) or the like is used. More specifically, for example, a training program based on Mask R-CNN or the like may be applied.
[0025] 2 is a diagram showing an example of a flow of a usage stage in which the trained model according to this embodiment is used. The posture estimation system 1 executes a posture estimation process for estimating the posture of the wheelchair and the passenger using the trained model. The posture estimation system 1 captures (records) an image of the wheelchair and the passenger with the imaging device 10 (e.g., a video camera) (step S201). Next, the captured image (second captured image) of the wheelchair and the passenger captured by the imaging device 10 is input as input data to the trained model (step S202).
[0026] The posture estimation system 1 uses the trained model to output 3D coordinate information of multiple parts of the wheelchair and the occupant from the input second captured image as a product of AI (Artificial Intelligence) (step S203). The posture estimation system 1 also outputs posture information calculated from the 3D coordinate information of multiple parts of the wheelchair and the occupant (step S204). The posture information is information including distances and angles (e.g., joint angles) based on the positional relationship of the coordinates of the estimated parts. For example, the posture estimation system 1 associates the estimated coordinates of each part so as to correspond to a human skeletal model or the structure of the wheelchair, and calculates distances and angles (e.g., joint angles) based on the positional relationship of the coordinates of the associated parts. The posture estimation system 1 also outputs posture information obtained by reconstructing multiple parts of the wheelchair and the occupant in a 3D coordinate space based on the distances and angles (e.g., joint angles) based on the positional relationship of the calculated coordinates of each part.
[0027] In the following description, the trained model generated by the posture estimation system 1 is also referred to as a "posture estimation model." The first captured image, the second captured image, or the first captured image and the second captured image together are also referred to as "captured images." In addition, the three-dimensional coordinates in this embodiment are represented by "X, Y, Z" coordinates consisting of an X axis and a Z axis perpendicular to the horizontal plane, and a Y axis perpendicular to the X axis and the Z axis.
[0028] [Configuration of posture estimation system 1] Next, the configuration of posture estimation system 1 will be described in detail. FIG. 3 is a block diagram showing an example of the configuration of a posture estimation system 1 according to this embodiment. The posture estimation system 1 shown in the figure includes an imaging device 10 (an example of an imaging section) and an information processing device 20 (an example of a posture estimation device and a learning device). The imaging device 10 is a video camera or a digital camera capable of capturing moving images. For example, the imaging device 10 captures images of athletes playing a sport such as wheelchair tennis or wheelchair basketball, and outputs the captured captured images to the information processing device 20. This captured image (moving image) is composed of a plurality of frame images captured at a predetermined frame rate. Furthermore, time information indicating the timing at which each frame image was captured is associated with the image. Note that there may be a plurality of imaging devices 10.
[0029] The information processing device 20 is, for example, a computer device such as a PC (personal computer). The information processing device 20 executes a learning process for generating a posture estimation model, as described with reference to Figs. 1 and 2, and a posture estimation process for estimating the posture of the wheelchair and a passenger from an image captured by the imaging device 10 using the posture estimation model. The illustrated information processing device 20 includes a storage unit 21 and a control unit 22. The storage unit 21 is a storage device that stores information used for various processes of the information processing device 20. For example, the storage unit 21 includes an image information storage unit 211, a coordinate information storage unit 212, a posture information storage unit 213, a learning dataset storage unit 214, a learning program storage unit 215, and a learned model storage unit 216.
[0030] The image information storage unit 211 stores data of the captured image captured by the imaging device 10. For example, the image information storage unit 211 stores data of a captured image (video) of a wheelchair and a passenger acquired from the imaging device 10 by the image acquisition unit 221 described later. The captured image of the wheelchair and the passenger is, for example, a captured image of a player playing a sport such as wheelchair tennis or wheelchair basketball. The captured image of the wheelchair and the passenger may be a captured image of a player of another wheelchair sport, or may be a captured image of a general player without being limited to a player. In addition, the captured image may be any captured image, not limited to wheelchair sports, as long as it is a captured image of a passenger riding in a wheelchair. The captured image (video) stored in the image information storage unit 211 is associated with a plurality of frame images and time information.
[0031] The coordinate information storage unit 212 stores three-dimensional coordinate information of a plurality of parts of the wheelchair and the occupant estimated from the captured image by the information processing device 20. For example, the coordinate information storage unit 212 stores three-dimensional coordinate information estimated for each frame image of the captured image.
[0032] FIG. 4 is a diagram showing an example of coordinate information data stored in the coordinate information storage unit 212. The coordinate information stores time information of a frame image and three-dimensional coordinate information of each part of the passenger and wheelchair estimated from the frame image in association with each other. The time information may be, for example, information based on the date and time of each frame (year, month, day, hour, minute, second, frame number, etc.), or may be a time code assigned to each frame. The three-dimensional coordinate information of each part is, for example, the three-dimensional coordinate values (X, Y, Z) of the passenger's neck, right shoulder, left shoulder, right elbow, left elbow, right wrist, left wrist, thigh, right thigh, left thigh, right knee, left knee, right heel, and left heel, and the right axle, right ground, left axle, and left ground of the wheelchair.
[0033] 3, the posture information storage unit 213 stores posture information regarding the posture calculated based on three-dimensional coordinate information of multiple parts of the wheelchair and the occupant estimated from the captured image by the information processing device 20. The posture information includes information such as distances and angles (for example, joint angles) based on the positional relationship of each part.
[0034] Fig. 5 is a diagram showing an example of posture information data stored in posture information storage unit 213. The illustrated posture information data example is an example of joint angle information calculated based on three-dimensional coordinate information of each body part. The joint angle information is stored in association with time information of a frame image and joint angle information calculated based on three-dimensional coordinate information of each body part of the wheelchair and occupant estimated from the frame image. The time information may be, for example, information based on the date and time of each frame (year, month, day, hour, minute, second, frame number, etc.) as in the example shown in Fig. 4, or may be a time code assigned to each frame.
[0035] The joint angle information is expressed by the rotation angles of roll (r), pitch (p), and yaw (y) in a three-dimensional coordinate space of a line segment (vector) connecting a certain part to another part based on a combination of predetermined parts, for example. For example, roll corresponds to rotation around the Z axis, pitch corresponds to rotation around the X axis, and yaw corresponds to rotation around the Y axis. In the illustrated example, the joint angle information includes the rotation angles of roll (r), pitch (p), and yaw (y) of left shoulder-right shoulder, right shoulder-right elbow, right elbow-right wrist, left shoulder-left elbow, left elbow-left wrist, neck-thigh, left thigh-right thigh, right thigh-right knee, right knee-right heel, left thigh-left knee, left knee-left heel, right axle-left axle, right axle-right ground, and left axle-left ground.
[0036] 6 is an image diagram showing an example of the output value of the posture information according to this embodiment. For example, this diagram shows an example of an image in which multiple parts of the wheelchair and the occupant are reconstructed in a three-dimensional coordinate space based on distances and angles (for example, joint angles) based on the positional relationship of the three-dimensional coordinates of each part. This image is generated as posture information and stored in the posture information storage unit 213.
[0037] 3, the training dataset storage unit 214 stores a training dataset for generating a posture estimation model. As described above, the training dataset is associated with a plurality of training captured images of a wheelchair and a passenger, and three-dimensional coordinate information of a plurality of body parts of the wheelchair and the passenger captured in each of the plurality of training captured images.
[0038] The learning program storage unit 215 stores, as a learning program, for example, a learning program based on a convolutional neural network (CNN), more specifically, Mask R-CNN, or the like.
[0039] The trained model memory unit 216 stores a posture estimation model generated by machine learning by inputting the learning dataset stored in the training dataset memory unit 214 into the training program stored in the training program memory unit 215.
[0040] Next, a description will be given of the configuration of the control unit 22. The control unit 22 is a functional configuration realized by a processor including, for example, a CPU (Central Processing Unit) or the like executing a program, and comprehensively controls the information processing device 20. For example, the control unit 22 includes an image acquisition unit 221, an image processing unit 222, a coordinate estimation unit 223, a posture information generation unit 224, and a trained model generation unit 225.
[0041] The image acquiring unit 221 (an example of an acquiring unit) acquires image information including captured images captured by the imaging device 10. For example, the imaging device 10 acquires captured images (videos) of athletes competing in a sport such as wheelchair tennis or wheelchair basketball from the imaging device 10, and stores the images in the image information storage unit 211. Note that the image acquiring unit 221 (an example of an acquiring unit) may acquire image information including captured images from a computer-readable recording medium, and store the image information in the image information storage unit 211.
[0042] The image processing unit 222 (an example of an extraction unit) recognizes an image area including a wheelchair and a passenger from the captured image acquired by the image acquisition unit 221, and extracts a partial image corresponding to the image area. For example, the image processing unit 222 recognizes an image area including a wheelchair and a passenger from the captured image by using feature information indicating the feature amounts of the wheelchair and the passenger (person). Furthermore, when the image processing unit 222 recognizes multiple image areas including wheelchairs and passengers from the captured image, it extracts multiple partial images.
[0043] The coordinate estimation unit 223 (an example of an estimation unit) estimates three-dimensional coordinate information of multiple body parts of the wheelchair and the occupant from a captured image of the wheelchair and the occupant, using the posture estimation model stored in the trained model storage unit 216. For example, the coordinate estimation unit 223 inputs a partial image extracted by the image processing unit 222 to the posture estimation model, and estimates three-dimensional coordinate information of multiple body parts of the wheelchair and the occupant from the input captured image, using the posture estimation model.
[0044] In addition, when the posture estimation model is capable of detecting the image area of the wheelchair and the passenger and estimating the three-dimensional coordinate information, the coordinate estimation unit 223 may input the captured image acquired by the image acquisition unit 221 to the posture estimation model, thereby estimating the three-dimensional coordinate information of a plurality of parts of the wheelchair and the passenger from the input captured image. For example, the coordinate estimation unit 223 may input the captured image acquired by the image acquisition unit 221 to the posture estimation model, detect the image area of the wheelchair and the passenger from the input captured image, extract partial images of the wheelchair and the passenger, and then estimate the three-dimensional coordinate information of a plurality of parts of the wheelchair and the passenger for each partial image. In other words, the coordinate estimation unit 223 may be configured to have a function of an extraction unit that extracts partial images of the wheelchair and the passenger from the input captured image, and a function of an estimation unit that estimates the three-dimensional coordinate information of a plurality of parts of the wheelchair and the passenger.
[0045] Then, the coordinate estimation unit 223 outputs the estimated three-dimensional coordinate information (coordinate values) for each body part as a product of the AI, and stores it in the coordinate information storage unit 212. For example, the coordinate estimation unit 223 estimates three-dimensional coordinate information of multiple body parts of the wheelchair and the occupant for each frame image, and stores the estimated three-dimensional coordinate information for each body part in the coordinate information storage unit 212 in association with time information for each frame image (see FIG. 4).
[0046] The posture information generating unit 224 (an example of a posture information generating unit) generates posture information relating to the posture of the wheelchair and the occupant based on the three-dimensional coordinate information of a plurality of body parts of the wheelchair and the occupant estimated by the coordinate estimating unit 223. For example, the posture information generating unit 224 calculates the joint angles (rotation angles of roll (r), pitch (p), and yaw (y) in a three-dimensional coordinate space) for each combination of predetermined body parts for each frame image, and stores them as posture information in the posture information storage unit 213 (see FIG. 5).
[0047] Furthermore, the posture information generating unit 224 generates an image in which each of the multiple body parts of the wheelchair and the occupant is reconstructed in a three-dimensional coordinate space based on distances and joint angles based on the positional relationship of the three-dimensional coordinates of each body part of the wheelchair and the occupant (see FIG. 6). The posture information generating unit 224 may store the image generated for each frame image in the posture information storage unit 213 as posture information.
[0048] The trained model generation unit 225 (an example of a training unit) performs machine learning using a dataset including a plurality of training captured images of a wheelchair and a passenger, and three-dimensional coordinate information of a plurality of body parts of the wheelchair and the passenger captured in each of the plurality of training captured images. As a result, the trained model generation unit 225 generates a posture estimation model (an example of a trained model) for estimating the three-dimensional coordinate information of a plurality of body parts of the wheelchair and the passenger. For example, the trained model generation unit 225 performs machine learning by inputting the training dataset stored in the training dataset storage unit 214 into the training program stored in the training program storage unit 215, thereby generating a posture estimation model. Then, the trained model generation unit 225 stores the generated posture estimation model in the trained model storage unit 216.
[0049] In addition, the trained model generation unit 225 may generate a posture estimation model for estimating three-dimensional coordinate information of multiple body parts of a wheelchair and a occupant in this embodiment by re-learning a human posture estimation model that has been machine-learned in advance to simply estimate the posture of a person other than a wheelchair occupant, using a training dataset that includes three-dimensional coordinate information of multiple body parts of the wheelchair and the occupant.
[0050] [Examples of learning programs] Here, a specific example of a learning program will be described. As described above, the learning program may be, for example, a convolutional neural network (CNN), more specifically, Mask R-CNN, etc. Below, the convolutional neural network (CNN) will be described as an example of machine learning.
[0051] About CNN FIG. 7 is an explanatory diagram illustrating the procedure for executing machine learning using CNN. In this diagram, the CNN is composed of I+1 layers L0 to LI. Layer L0 is also called the input layer, layers L1 to L(I-1) are called intermediate layers or hidden layers, and layer LI is called the output layer. I is determined by the structure of the CNN, for example, I=3 or 4.
[0052] In the CNN, an input image is input to the input layer L0. The input image is represented by a pixel matrix D11 in which the vertical and horizontal positions of the input image are the positions in the matrix. Each element of the pixel matrix D11 is input with the R (red), G (green), and B (blue) subpixel values as the subpixel values of the pixel corresponding to the position in the matrix. The first hidden layer, L1, is the layer where convolutional processing (also called filtering) and pooling processing are performed.
[0053] (Convolution processing) An example of the convolution process in the intermediate layer L1 will be described below. The convolution process is a process of applying a filter to an original image and outputting a feature map. Specifically, the input pixel values are divided into an R subpixel matrix D121, a B subpixel matrix D122, and a G subpixel matrix D123. In each of the subpixel matrices D121, D122, and D123 (each also referred to as a "subpixel matrix D12"), the elements of the submatrix with s rows and t columns are multiplied by the elements of a convolution matrix CM1 (also called a kernel) with s rows and t columns, and then added together to calculate a first pixel value. The first pixel values calculated in each of the subpixel matrices D12 are multiplied by a weighting coefficient and then added together to calculate a second pixel value. The second pixel value is set in each element of the convoluted image matrix D131 as a matrix element corresponding to the position of the submatrix. The position of the submatrix in each of the subpixel matrices D12 is shifted for each element (subpixel), and thus the second pixel value at each position is calculated, and all the matrix elements of the convoluted image matrix D131 are calculated.
[0054] 7 shows an example of a convolution matrix CM1 with 3 rows and 3 columns, where the first pixel value of the convolved pixel value D1311 is calculated for the submatrix with 3 rows and 3 columns from the second row to the fourth row and the second column to the fourth column of each subpixel matrix D12. A weighting coefficient is calculated and added to the first pixel value of each subpixel matrix D121, D122, and D123, whereby a second pixel value is calculated as the matrix element in the second row and the second column of the convolved image matrix D131. Similarly, the second pixel value of the matrix element in the third row and the second column of the convolved image matrix D131 is calculated from the submatrix with the third row to the fifth row and the second column to the fourth column. Similarly, convolved image matrices D132, . . . are calculated using other weighting coefficients or other convolution matrices.
[0055] (Pooling process) An example of the pooling process in the hidden layer L1 will be described below. The pooling process is a process for reducing an image while retaining the features of the image. Specifically, for the convoluted image matrix D131, a representative value of matrix elements in each region PM of u rows and v columns is calculated. The representative value is, for example, a maximum value. The representative value is set for each element of the CNN image matrix D141 as a matrix element corresponding to the position of the region. By shifting the regions in the convoluted image matrix D131 for each region PM, a representative value at each position is calculated, and all matrix elements of the convoluted image matrix D131 are calculated.
[0056] For example, FIG. 7 shows an example of a 2-row, 2-column region PM, where the maximum value of the second pixel values in the 2-row, 2-column region from the 3rd row to the 4th row and the 3rd column to the 4th column of the convolution image matrix D131 is calculated as a representative value. This representative value is set to the matrix element in the 2nd row, 2nd column of the CNN image matrix D141. Similarly, a representative value of the matrix element in the 3rd row, 2nd column of the CNN image matrix D141 is calculated from the submatrix in the 5th row to the 6th row and the 2nd column to the 4th column. Similarly, the CNN image matrix D142, is calculated from the convolution image matrix D132,.
[0057] Each matrix element (N elements) of the CNN image matrices D141, D142, ... is arranged in a predetermined order to generate a vector x. In FIG. 7, the element x of the vector x is n (n=1, 2, 3, ...N) is represented by N nodes.
[0058] The hidden layer Li represents the i-th hidden layer (i = 2 to I-1). From the i-th hidden layer node, a vector u (i) is the function f(u (i) ) as the input value for the vector z (i) The vector u (i) is the vector z output from the i-1th hidden layer node. (i-1) Then, the weight matrix W (i) Multiply it from the left and give vector b (i) The function f(u (i) ) is the activation function, and the vector b (i) is the bias. Also, the vector u (0) is the vector x.
[0059] The node of the output layer L4 is z (I-1) and the output is M y m (m=1, 2, . . . M). In other words, the output layer LI of the CNN gives y m Vector y(=(y 1 , y 2 , y 3 , y M )) is output. As a result, when the pixel values of the input image are input as input variables, the CNN outputs the vector y as an output variable. The vector y represents an evaluation value.
[0060] FIG. 8 is an explanatory diagram illustrating a learning procedure of machine learning using CNN. This figure is an explanatory diagram of the case where the CNN in FIG. 7 performs machine learning. For the pixel values of the images in the training dataset, the vector x output from the first hidden layer is defined as vector X. The vector representing the determined class of the training dataset is defined as vector Y.
[0061] Weight matrix W (i) An initial value is set for . When an input image is input to the input layer, and as a result, a vector X is input to the second hidden layer, a vector y(X) corresponding to vector X is output from the output layer. The error E between vector y(X) and vector Y is calculated using a loss function. The gradient ΔE of the i-th layer i is the output from each layer z i and the error signal δ i The error signal δ is calculated using i is the error signal δ i-1 This process of transmitting the error signal from the output layer to the input layer is also called backpropagation. Weight matrix W (i) is the gradient ΔE i Similarly, in the first hidden layer, the convolution matrix CM or the weighting coefficients are updated.
[0062] (Setting the trained model) The trained model generation unit 225 sets, for the CNN, the number of layers, the number of nodes in each layer, the node connection method between each layer, the activation function, the error function, the gradient descent algorithm, the pooling region, the kernel, the weighting coefficient, and the weighting matrix. The trained model generation unit 225 sets, for example, three layers (I=3) as the number of layers. As the number of nodes in each layer (also referred to as the "node number"), the trained model generation unit 225 sets the number of elements of vector x (node number N) to 800, the number of nodes in the second intermediate layer (i=2) to 500, and the number of nodes in the output layer (i=3) to 10. However, the present invention is not limited to this, and the total number may be four or more layers, and a different value may be set for the number of nodes.
[0063] The trained model generation unit 225 sets 20 convolution matrices CM of 5 rows and 5 columns and regions PM of 2 rows and 2 columns. However, the present invention is not limited to this, and a different number of matrices or a different number of convolution matrices CM may be set. Also, a different number of matrices or regions PM may be set. The trained model generation unit 225 may perform more convolution processing or pooling processing.
[0064] The trained model generation unit 225 sets full connections as connections for each layer of the neural network. However, the present invention is not limited to this, and connections for some or all layers may be set to non-full connections. The trained model generation unit 225 sets a sigmoid function as an activation function for all layers. However, the present invention is not limited to this, and the activation function for each layer may be other activation functions such as a step function, linear combination, soft sine, soft plus, ramp function, truncated power function, polynomial, absolute value, radial basis function, wavelet, maxout, etc. Also, the activation function of a certain layer may be a different type from that of other layers.
[0065] The trained model generation unit 225 sets squared loss (mean squared error) as the error function. However, the present invention is not limited to this, and the error function may be cross entropy, τ-quantile loss, Huber loss, or ε sensitivity loss (ε tolerance error function). In addition, the trained model generation unit 225 sets SGD (stochastic gradient descent) as an algorithm (gradient descent algorithm) for calculating the gradient. However, the present invention is not limited to this, and Momentum (inertia term) SDG, AdaGrad, RMSprop, AdaDelta, Adam (adaptive moment estimation), etc. may be used as the gradient descent algorithm.
[0066] [Mask R-CNN] Next, we will explain the overview of machine learning using Mask R-CNN. Mask R-CNN performs object detection on an image and applies CNN to the detected area. Figure 9 is a diagram showing an example of a network structure of Mask R-CNN. In the "Convolutional backbone" 101, a network (CNN) including multiple convolution layers is used to extract features from an input image and output the extraction results. A feature map is used as the feature.
[0067] In the "RPG (Region Proportional Network)" 102, a candidate region of the detection target (e.g., a wheelchair and a passenger) is detected based on the feature amount output from the "Convolutional backbone" 101. For example, the RPG 102 outputs the probability that the detection target is included in the candidate region and the position information of the candidate region on a feature map.
[0068] Next, in the "RolAlign layer" 103, partial regions corresponding to candidate regions are extracted from the feature map and converted into a fixed size feature map. For each partial region, the "Keypoint branch" 104 estimates the posture of the detection target (e.g., wheelchair and passenger), and outputs 3D coordinate information of each part of the detection target (e.g., wheelchair and passenger).
[0069] In addition, each layer is fully connected in a "fully connected layer" 105, classifying the partial regions that correspond to the candidate regions (Classification), and estimating the position (rectangular region) of the detection target (e.g., wheelchair and occupant) (Box Regression).
[0070] (Variations in learning) As the learning program, R(Region)-CNN, Fast R-CNN, Faster R-CNN, etc. may be used. As the learning program, Stacked Hourglass Networks, PoseResnet, HRNet, Cascaded Pyramid Network, etc. may be used. Other neural networks such as Perceptron neural networks, Recurrent Neural Networks (RNN), Residual Networks (ResNet), etc. may be set. Also, a trained model of supervised learning such as a decision tree, regression tree, random forest, gradient boosting tree, linear regression, logistic regression, or SVM (support vector machine) may be used.
[0071] In addition, the input image for each input unit input to the learning program is not limited to a single image, but may be multiple images (video) over time. For example, a video may be divided into a certain number of frames and used as input images. This allows an AI that takes spatiotemporal information into account, eliminating ambiguity and improving temporal consistency. For example, posture estimation can be performed with high accuracy even when a part of a joint is hidden (when there is occlusion) or when the image is blurred due to a sudden movement (when there is motion blur). As a learning program, 3D CNN (3D Convolutional Neural Networks), which combines spatial information (2D) and time information (1D) for the input video and performs 3D convolution, or RNN (Recurrent Neural Network) may be used.
[0072] [Evaluation of posture estimation accuracy using trained models] Here, we will explain the evaluation results of the accuracy of posture estimation using a posture estimation model (trained model) that was machine-learned using a dataset that includes multiple learning images of a wheelchair and its occupant, and three-dimensional coordinate information of multiple parts of the wheelchair and its occupant that are captured in each of the multiple learning images.
[0073] (Training Dataset) Machine learning requires a large-scale dataset, but it is difficult to prepare a large-scale dataset that includes 3D coordinate information of multiple body parts of wheelchairs and passengers. Therefore, here, a small-scale dataset that includes 3D coordinate information of multiple body parts of wheelchairs and passengers (people) was created and re-trained based on a human posture estimation model that estimates the posture of a person that has been pre-trained with a large-scale dataset. This small-scale dataset was created by annotating multiple specific body parts (feature points) of wheelchairs and passengers of about 6,000 people on about 2,000 wheelchair sports images. Since it is preferable that the wheelchairs in the wheelchair sports images are similar in shape, images of wheelchair basketball, wheelchair tennis, wheelchair badminton, and wheelchair rugby were used.
[0074] (Learning Program) For the training program, a Mask R-CNN-based pose estimation model that had been pre-trained on a large-scale dataset containing only people was used, and then re-trained on a dataset containing 3D coordinate information for multiple body parts of the wheelchair and the occupant (person).
[0075] (Evaluation Results) Accuracy was evaluated using Average Precision (AP), an index commonly used to evaluate posture estimation methods. The accuracy of the estimation results for the 3D coordinates of each body part was judged based on Object Keypoint Similarity (OKS), which indicates the similarity between the correct posture and the estimated posture. OKS indicates the average similarity between the correct coordinates and the estimated coordinates of each body part (feature point) per person. When using a posture estimation model retrained with a dataset containing 3D coordinate information for multiple body parts of the wheelchair and the occupant, AP was 92.8%, indicating that 3D coordinate information for multiple body parts of the wheelchair and the occupant can be estimated with high accuracy.
[0076] As described above, in the posture estimation system 1 of this embodiment, the information processing device 20 (an example of a posture estimation device) estimates three-dimensional coordinate information of one or more body parts of the wheelchair and the occupant from the captured images of the wheelchair and the occupant, using a posture estimation model (an example of a trained model) that has been machine-learned based on a plurality of learning images of the wheelchair and the occupant and coordinate information of one or more body parts of the wheelchair and the occupant captured in each of the plurality of learning images.
[0077] As a result, the posture estimation system 1 can estimate the three-dimensional coordinates of multiple parts of the wheelchair and the passenger from images alone by using machine learning, and can estimate the posture of the wheelchair and the passenger as a whole from images of the wheelchair and the passenger. In addition, in this embodiment, the posture estimation process is performed on the wheelchair and the passenger as a single object, so the processing speed is improved compared to when the posture estimation process is performed separately. For example, the calculation speed is improved because the wheelchair and the passenger are calculated as a single object. In addition, by performing the posture estimation process on the wheelchair and the passenger as a single object, a posture estimation model (an example of a trained model) that takes into account the positional relationship between the wheelchair and the passenger can be generated, improving the accuracy of posture estimation.
[0078] In addition, the information processing device 20 (an example of a posture estimation device) may extract partial images of multiple wheelchairs and occupants from an image of multiple wheelchairs and occupants, and estimate three-dimensional coordinate information of one or more parts of the wheelchairs and occupants from each of the multiple partial images.
[0079] As a result, the posture estimation system 1 can estimate the three-dimensional coordinates of a plurality of wheelchairs and occupants from an image, and can therefore estimate the postures of a plurality of wheelchairs and occupants from an image in which the plurality of wheelchairs and occupants are captured.
[0080] The configuration as an extracting unit that extracts a partial image may be included in either the image processing unit 222 or the coordinate estimation unit 223 provided in the information processing device 20. For example, the image processing unit 222 may recognize an image area including a wheelchair and a passenger from the captured image acquired by the image acquisition unit 221, and extract a partial image corresponding to the image area. Alternatively, the coordinate estimation unit 223 may use a posture estimation model to extract a partial image corresponding to an image area including a wheelchair and a passenger from the captured image.
[0081] In this embodiment, an example (a so-called top-down approach) has been described in which the wheelchair and the passenger (person) are detected, and then the posture of each is estimated, but the present invention is not limited to this. For example, the posture estimation process according to this embodiment can be realized by identifying all key points in the captured image acquired by the image acquisition unit 221, and then matching and connecting the key points (corresponding to each part of the wheelchair and the passenger) for each set of the wheelchair and the passenger (person) (a so-called bottom-up approach).
[0082] Moreover, the information processing device 20 (an example of a posture estimation device) generates posture information relating to the posture of the wheelchair and the occupant, based on the estimated three-dimensional coordinate information.
[0083] This enables the posture estimation system 1 to digitize the postures of the wheelchair and the occupant, and to visualize the postures.
[0084] The information processing device 20 (an example of a learning device) also functions as a learning device that generates a posture estimation model (an example of a trained model) for estimating three-dimensional coordinate information of one or more body parts of the wheelchair and the passenger from the images of the wheelchair and the passenger, by performing machine learning using a dataset including a plurality of learning captured images of the wheelchair and the passenger and coordinate information of one or more body parts of the wheelchair and the passenger captured in each of the plurality of learning captured images. The information processing device 20 uses the generated posture estimation model to estimate three-dimensional coordinate information of one or more body parts of the wheelchair and the passenger from the images captured by the imaging device 10 (an example of an imaging unit).
[0085] This allows the posture estimation system 1 to generate a posture estimation model (an example of a trained model) capable of estimating the three-dimensional coordinates of multiple parts of the wheelchair and the occupant only from the image. Therefore, the posture estimation system 1 can estimate the postures of the wheelchair and the occupant only from the image using machine learning.
[0086] The posture estimation model according to this embodiment is a trained model for estimating three-dimensional coordinate information of one or more body parts of a wheelchair and a wheelchair occupant. This posture estimation model is machine-learned based on a plurality of learning captured images of the wheelchair and the occupant and the three-dimensional coordinate information of one or more body parts of the wheelchair and the occupant captured in each of the plurality of learning captured images, and causes a computer to function to estimate the three-dimensional coordinate information of one or more body parts of the wheelchair and the occupant from the captured images of the wheelchair and the occupant.
[0087] As a result, the posture estimation model in this embodiment can estimate the three-dimensional coordinates of multiple parts of the wheelchair and occupant from images alone, and can estimate the posture of the wheelchair and occupant as a whole from images of the wheelchair and occupant.
[0088] In this embodiment, an example is described in which the posture of the wheelchair and the passenger is estimated from an image using a posture estimation model generated by machine learning, but the present invention is not limited to this. For example, the posture of the wheelchair and the passenger may be estimated from an image using a program that embodies an algorithm regarding the relationship between the images of the wheelchair and the passenger captured in each of a plurality of captured images and the three-dimensional coordinate information of one or more parts of the wheelchair and the passenger.
[0089] In other words, in the posture estimation system 1 according to this embodiment, the information processing device 20 (an example of a posture estimation device) estimates three-dimensional coordinate information of one or more body parts of the wheelchair and the occupant from the captured images of the wheelchair and the occupant, based on a plurality of captured images of the wheelchair and the occupant and coordinate information of one or more body parts of the wheelchair and the occupant captured in each of the plurality of captured images.
[0090] As a result, posture estimation system 1 can estimate the three-dimensional coordinates of multiple parts of the wheelchair and occupant from images alone without using machine learning, and can estimate the posture of the wheelchair and occupant as a whole from images captured of the wheelchair and occupant.
[0091] <Second embodiment> Next, a posture estimation system 1A according to a second embodiment will be described with reference to the drawings. In this embodiment, a posture estimation system 1A is described that derives an evaluation index of a rider's skill by using three-dimensional coordinate information and posture information of a plurality of parts of the wheelchair and the rider estimated by the posture estimation system 1 according to the first embodiment. The rider's skill is, for example, the rider's wheelchair operation skill or the rider's motor skill.
[0092] 10 is a block diagram showing an example of the configuration of a posture estimation system 1A according to this embodiment. The posture estimation system 1A shown in the figure includes an imaging device 10 and an information processing device 20A. The information processing device 20A includes a storage unit 21A and a control unit 22A.
[0093] The storage unit 21A stores information used for various processes of the information processing device 20A. The storage unit 21A includes an image information storage unit 211, a coordinate information storage unit 212, a posture information storage unit 213, a learning data set storage unit 214, a learning program storage unit 215, a trained model storage unit 216, and an evaluation index storage unit 217. Note that the image information storage unit 211, the coordinate information storage unit 212, the posture information storage unit 213, the learning data set storage unit 214, the learning program storage unit 215, and the trained model storage unit 216 are the same as those described in the first embodiment, and their description will be omitted.
[0094] The evaluation index storage unit 217 stores evaluation index information related to evaluation indexes derived by the evaluation index derivation unit 226 described later. For example, the evaluation index storage unit 217 stores evaluation index information related to the wheelchair paddling motion, the torque and power of the wheelchair, the three-dimensional coordinates of the occupant's center of gravity or the inclination of the body axis, the height of the occupant's raised hands, the posture of tools or implements (e.g., a tennis racket) held or worn by the occupant, and the like.
[0095] The control unit 22A is a functional configuration realized by, for example, a processor including a CPU or the like executing a program, and performs overall control of the information processing device 20A. As with the control unit 22 of the above-mentioned first embodiment, the control unit 22A performs a generation process of a posture estimation model (trained model) and a posture estimation process using the posture estimation model, and in addition, derives an evaluation index of the wheelchair operation skill of the passenger and an evaluation index of the passenger's movement.
[0096] The illustrated control unit 22A includes an image acquisition unit 221, an image processing unit 222, a coordinate estimation unit 223, a posture information generation unit 224, a trained model generation unit 225, and an evaluation index derivation unit 226. Note that the image acquisition unit 221, the image processing unit 222, the coordinate estimation unit 223, the posture information generation unit 224, and the trained model generation unit 225 are the same as those described in the first embodiment, and description thereof will be omitted.
[0097] The evaluation index derivation unit 226 derives an evaluation index for evaluating the skill of a wheelchair occupant, based on the posture information generated by the posture information generation unit 224. For example, the evaluation index derivation unit 226 derives an evaluation index for evaluating the wheelchair operation skill of the occupant.
[0098] FIG. 11 is a diagram showing a first example of an evaluation index that can be derived according to this embodiment. The illustrated example is a schematic diagram of a wheelchair and a passenger viewed from the side. The evaluation index derivation unit 226 detects a specific operation of the wheelchair by the passenger as one of the evaluation indexes. One example of the specific operation is a wheelchair paddling motion (pushing motion). For example, the evaluation index derivation unit 226 may detect the wheelchair paddling motion by the passenger based on a change in arm joint angle based on the three-dimensional coordinates of the passenger's arm joints (shoulder, elbow, wrist).
[0099] The evaluation index derivation unit 226 may detect the rowing motion by utilizing the fact that the acceleration of the movement of the wheelchair increases when the rider rows the wheelchair. The acceleration of the movement of the wheelchair starts to increase when the pushing motion starts, reaches a maximum immediately after the rider releases the wheelchair wheels, and then decreases. Therefore, the evaluation index derivation unit 226 can detect the rowing motion by detecting the peak value of the movement acceleration of the wheelchair.
[0100] FIG. 12 is a diagram showing a second example of an evaluation index that can be derived according to this embodiment. The illustrated example is a schematic diagram of a wheelchair and a rider viewed from the side. The evaluation index derivation unit 226 may derive the torque and power of the wheelchair due to the rider's pedaling motion as one of the evaluation indexes. For example, the evaluation index derivation unit 226 tracks the position (coordinate) of the rider's wrist to calculate the angular velocity of the rotation of the wheelchair wheels from the amount of change in the position of the wrist relative to the wheelchair axle. The evaluation index derivation unit 226 also tracks the position (coordinate) of the wheelchair axle to calculate the moving speed. The evaluation index derivation unit 226 derives the torque and power of the wheelchair based on the calculated angular velocity and moving speed, the radius of the wheels, etc.
[0101] 13 is a diagram showing a third example of an evaluation index that can be derived according to this embodiment. The illustrated example is a schematic diagram of a wheelchair and a passenger viewed from above. The evaluation index derivation unit 226 may derive the direction, acceleration, and speed of the movement of the wheelchair as one of the evaluation indexes. For example, the evaluation index derivation unit 226 tracks the coordinates of the passenger's center of gravity or the central coordinates of the wheelchair axles (the central coordinates of the line segment connecting the coordinates of the right axle and the coordinates of the left axle), and derives the direction, acceleration, and speed of the movement of the wheelchair.
[0102] 14 is a diagram showing a fourth example of an evaluation index that can be derived according to this embodiment. The illustrated example is a schematic diagram of a wheelchair and a passenger viewed from the front. The evaluation index derivation unit 226 may derive the inclination of the passenger's center of gravity or body axis as one of the evaluation indexes. For example, the evaluation index derivation unit 226 detects the passenger's center of gravity and body axis based on a line segment connecting the coordinates of the neck and the coordinates of the thighs, and derives the three-dimensional coordinates of the passenger's center of gravity or the inclination of the body axis.
[0103] In addition, in order to detect the inclination of the passenger's body axis, the information processing device 20A may generate a posture estimation model by machine learning using a learning dataset that includes images in which the body axis is not tilted and images in which the body axis is tilted, and detect the inclination of the passenger's body axis using the generated posture estimation model.
[0104] FIG. 15 is a diagram showing a fifth example of an evaluation index that can be derived according to this embodiment. The illustrated example is a schematic diagram of a wheelchair and a rider viewed from the front. The evaluation index derivation unit 226 may derive the height of the rider's raised hand as one of the evaluation indexes. For example, the evaluation index derivation unit 226 tracks the position (coordinate) of the rider's wrist, and derives the height of the rider's raised hand based on the three-dimensional coordinate when the wrist position is at its highest. This evaluation method is effective, for example, in evaluating wheelchair basketball players.
[0105] FIG. 16 is a diagram showing a sixth example of an evaluation index that can be derived according to this embodiment. The illustrated example is a schematic diagram of a wheelchair and a rider viewed from the front. In this example, the rider is holding a wheelchair tennis racket. The evaluation index derivation unit 226 may detect the racket held by the rider and derive the racket's posture (position and orientation) as one of the evaluation indexes. For example, in order to detect the racket held by the rider, the information processing device 20A may generate a posture estimation model by machine learning with a learning dataset including an image of the rider holding the racket, and detect the racket held by the rider using the generated posture estimation model. In addition, the information processing device 20A may generate a posture estimation model by machine learning with the learning dataset including three-dimensional coordinate information of each part of the racket (for example, the grip end, head top, sweet spot, etc. of the racket), and use the generated posture estimation model to estimate three-dimensional coordinate information of each part of the racket held by the rider and generate posture information of the racket.
[0106] Furthermore, the evaluation index derivation unit 226 may derive the total posture of the rider and the racket, or the total posture of the wheelchair, the rider, and the racket, as one of the evaluation indexes. For example, the information processing device 20A may estimate three-dimensional coordinate information of each part of the wheelchair, the rider, and the racket using a posture estimation model generated by machine learning with a learning dataset including an image of a rider holding a racket, and derive the total posture of the rider and the racket, or the total posture of the wheelchair, the rider, and the racket, as one of the evaluation indexes based on the estimated three-dimensional coordinate information.
[0107] 16 shows an example of detecting a wheelchair tennis racket and deriving the posture of the tennis racket (or the total posture of the occupant and racket, or the total posture of the wheelchair, occupant, and racket) as one of the evaluation indices, but this is not limited to a tennis racket and can be applied to any object such as equipment or tools held or worn by a occupant. For example, it may be a table tennis racket, ice hockey racket, softball bat, wheelchair fencing sword, etc. held by a wheelchair occupant. It may also be a softball glove worn by a wheelchair occupant.
[0108] As described above, in the posture estimation system 1A of this embodiment, the information processing device 20A (an example of a posture estimation device) derives evaluation indices for evaluating the skills of the occupant (such as the wheelchair operation skills and the occupant's motor skills) based on three-dimensional coordinate information and posture information of multiple parts of the wheelchair and the occupant.
[0109] As a result, the posture estimation system 1A can easily evaluate the wheelchair operation skill or athletic skill of a wheelchair sports player or a wheelchair user by photographing the wheelchair player or a wheelchair user with a camera, or can help the evaluator to easily perform the evaluation.
[0110] Furthermore, the information processing device 20A estimates three-dimensional coordinate information of at least one part of the object (equipment or tool) from the captured images of the wheelchair, the occupant, and the object (equipment or tool) held or worn by the occupant, based on a plurality of captured images of the wheelchair, the occupant, and the object (equipment or tool) held or worn by the occupant, and three-dimensional coordinate information of at least one part of the object, the occupant, and the object captured in each of the plurality of captured images.The information processing device 20A then derives an evaluation index for evaluating the skill of the occupant, based on the three-dimensional coordinate information of at least one part of the object (equipment or tool).
[0111] This enables the posture estimation system 1A to easily evaluate the skills of a wheelchair sports player, including the use of equipment or devices held or worn by the player, or to help the evaluator to easily evaluate the skills of the wheelchair sports player.
[0112] Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention.
[0113] In the above embodiment, an example was shown in which three-dimensional coordinate information of a plurality of parts of the wheelchair and the occupant was estimated, but the estimation is not limited to three-dimensional coordinate information, and two-dimensional coordinate information may be estimated.
[0114] Furthermore, the imaging device 10 in the above embodiment is not limited to a dedicated camera such as a video camera or a digital camera, but may be an electronic device having a camera function as a part of its functions such as a smartphone. The imaging device 10 and the information processing device 20 may be communicatively connected by wire or wirelessly, or, instead of being communicatively connected, an image captured by the imaging device 10 may be transferred to the information processing device 20 via a storage medium such as an optical disk or a memory card.
[0115] Furthermore, the imaging device 10 and the information processing device 20 (or the information processing device 20A) may be configured as a single integrated device. For example, a smartphone or tablet-type computer device having a camera function may include the various units included in the imaging device 10 and the information processing device 20 (or the information processing device 20A). Furthermore, the posture estimation system 1 (or the posture estimation system 1A) may be configured to include a plurality of devices in which the imaging device 10 and the information processing device 20 (or the information processing device 20A) are integrated.
[0116] Further, in the above embodiment, the information processing device 20 (20A) is described as being one device, but is not limited thereto, and may be configured as a plurality of devices. For example, in the above embodiment, the information processing device 20 is described as being equipped with both a configuration as a learning device that generates a posture estimation model and a configuration as a posture estimation device that estimates a posture using the posture estimation model, but the configuration as the learning device and the configuration as the posture estimation device may be configured as separate devices. For example, among the configurations equipped in the information processing device 20 (or the information processing device 20A), the learning data set storage unit 214, the learning program storage unit 215, the trained model storage unit 216, and the trained model generation unit 225 may be equipped in a computer device as a learning device, and the image information storage unit 211, the coordinate information storage unit 212, the posture information storage unit 213, the evaluation index storage unit 217, the image acquisition unit 221, the image processing unit 222, the coordinate estimation unit 223, the posture information generation unit 224, and the evaluation index derivation unit 226 may be equipped in a computer device as a posture estimation device.
[0117] The information processing device 20 (20A) may be configured such that a part or the whole of the storage unit 21 (21A) is provided in an external device. In this case, the external device may be a storage device connectable to the information processing device 20 (20A) via a network. The information processing device 20 (20A) may be configured such that a part of the functions of the control unit 22 (22A) is provided in the external device.
[0118] The imaging device 10 or the information processing device 20 (or the information processing device 20A) included in the above-mentioned posture estimation system 1 (or the posture estimation system 1A) has a computer system therein. A program for implementing the functions of each component included in the above-mentioned imaging device 10 or the information processing device 20 (or the information processing device 20A) may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to perform processing in each component included in the above-mentioned imaging device 10 or the information processing device 20 (or the information processing device 20A). Here, "reading a program recorded in a recording medium into a computer system and executing it" includes installing the program into a computer system. The "computer system" here includes hardware such as an OS and peripheral devices. The "computer system" may also include a plurality of computer devices connected via a network including a communication line such as the Internet, a WAN, a LAN, or a dedicated line. The "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, or a storage device such as a hard disk built into a computer system. In this way, the recording medium storing the program may be a non-transitory recording medium such as a CD-ROM.
[0119] The recording medium also includes a recording medium installed inside or outside and accessible from a distribution server to distribute the program. The program may be divided into a plurality of parts, downloaded at different times, and then combined by each component of the imaging device 10 or the information processing device 20 (or the information processing device 20A), or each divided program may be distributed by a different distribution server. Furthermore, the "computer-readable recording medium" includes a recording medium that holds a program for a certain period of time, such as a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network. The above program may be a recording medium for realizing part of the above-mentioned functions. Furthermore, the program may be a so-called difference file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.
[0120] Also, a part or all of the imaging device 10 or the information processing device 20 (or the information processing device 20A) may be realized as an integrated circuit such as an LSI (Large Scale Integration). Also, each component in the imaging device 10 or the information processing device 20 (or the information processing device 20A) of this embodiment may be individually made into a processor, or may be integrated into a processor partly or entirely. Also, the method of integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. Also, when an integrated circuit technology that replaces LSI appears due to advances in semiconductor technology, an integrated circuit based on that technology may be used.
[0121] In the above embodiment, the posture estimation technology has been described mainly in the field of wheelchair sports, but the technology can also be applied to fields other than sports, such as those shown below. (1) Application to Barrier-Free Design The posture estimation system 1A can quantitatively evaluate the load on a wheelchair occupant by using, for example, images from security cameras installed in towns and facilities as input images and deriving evaluation indices such as the inclination of the wheelchair and the pedaling power of the occupant. This makes it possible for the posture estimation system 1A to be used in the design of barrier-free towns and facilities. (2) Wheelchair Accident Prevention System This system can be applied to wheelchair accident prevention systems intended for use in hospitals, nursing homes, shopping centers, etc. For example, the posture estimation system 1 (1A) can classify the movements made while operating a wheelchair by estimating the posture (skeleton) of a wheelchair user, and can be used to detect and prevent falls. In addition, the posture estimation system 1 (1A) can be used to issue accident prevention alerts and estimate the impact force in the event of an accident by deriving evaluation indices such as the speed of movement and the magnitude of acceleration and deceleration. (3) Rehabilitation Applications The posture estimation system 1 (1A) can automatically measure the movements of the wheelchair and the occupant, thereby monitoring the degree and effect of improvement due to the occupant's rehabilitation during rehabilitation using a wheelchair. (4) Application to game controllers There is a technology that uses the motion capture data as input and uses physical movements as a game controller, but by using posture estimation technology by posture estimation system 1 (1A), the physical movements of a wheelchair user can also be used as a game controller. (5) Estimation of fatigue level The posture estimation system 1A may derive the fatigue level of the occupant as an evaluation index based on changes in the moving speed and the moving acceleration of the wheelchair and the occupant, or the torque and power of the wheelchair described above with reference to Fig. 12. For example, the posture estimation system 1A may identify the fatigue level of the occupant as an evaluation index for different types of wheelchairs or before and after a design change of the wheelchair, and provide feedback to the wheelchair manufacturer or designer. The examples shown in (1) to (5) are only some of the application examples of the posture estimation technology according to the present invention, and can be applied to various fields where wheelchairs are used. [Explanation of symbols]
[0122] 1,1A... Posture estimation system, 10... Imaging device, 20,20A... Information processing device, 21,21A... Memory unit, 22,22A... Control unit, 211... Image information memory unit, 212... Coordinate information memory unit, 213... Posture information memory unit, 214... Learning data set memory unit, 215... Learning program memory unit, 216... Trained model memory unit, 217... Evaluation index memory unit, 221... Image acquisition unit, 222... Image processing unit, 223... Coordinate estimation unit, 224... Posture information generation unit, 225... Trained model generation unit, 226... Evaluation index derivation unit
Claims
1. an estimation unit that estimates coordinate information of one or more body parts of the wheelchair and the occupant from a second captured image in which the wheelchair and the occupant are captured, based on a plurality of first captured images in which the wheelchair and the occupant of the wheelchair are captured and coordinate information of one or more body parts of the wheelchair and the occupant captured in each of the plurality of first captured images; A posture estimation system comprising:
2. a learning unit that performs machine learning using a dataset including a plurality of first captured images for learning in which the wheelchair and the occupant are captured, and coordinate information of one or more body parts of the wheelchair and the occupant captured in each of the plurality of first captured images for learning, to generate a trained model for estimating the coordinate information of one or more body parts of the wheelchair and the occupant from a second captured image in which the wheelchair and the occupant are captured; Equipped with The estimation unit is Using the trained model generated by the training unit, coordinate information of one or more body parts of the wheelchair and the occupant is estimated from a second captured image captured by an imaging unit. The pose estimation system of claim 1 .
3. an extraction unit that extracts partial images of the wheelchairs and the occupants from a second captured image in which the wheelchairs and the occupants are captured, The estimation unit is estimating coordinate information of one or more body parts of the wheelchair and the occupant from each of the plurality of partial images; The posture estimation system according to claim 1 or 2.
4. a posture information generating unit that generates posture information regarding postures of the wheelchair and the occupant based on the coordinate information estimated by the estimation unit; The pose estimation system according to claim 1 , further comprising:
5. an evaluation index derivation unit that derives an evaluation index for evaluating a skill of the passenger based on the posture information generated by the posture information generation unit; The pose estimation system of claim 4 .
6. The estimation unit is based on a plurality of captured images in which the wheelchair, the occupant, and an object held or worn by the occupant are captured, and on coordinate information of one or more parts of the wheelchair, the occupant, and the object captured in each of the plurality of first captured images, estimating coordinate information of at least one part of the object from a second captured image in which the wheelchair, the occupant, and the object are captured; The evaluation index derivation unit is Deriving the evaluation index based on coordinate information of at least one or more parts of the object. The pose estimation system of claim 5 .
7. an estimation unit that estimates coordinate information of one or more body parts of the wheelchair and the occupant from a second captured image in which the wheelchair and the occupant are captured, based on a plurality of first captured images in which the wheelchair and the occupant of the wheelchair are captured and coordinate information of one or more body parts of the wheelchair and the occupant captured in each of the plurality of first captured images; A posture estimation device comprising:
8. A posture estimation method in a posture estimation system, comprising: An acquisition unit acquires an image including a wheelchair and an occupant of the wheelchair; an estimation unit estimating, from the second captured image acquired by the acquisition unit, coordinate information of one or more body parts of the wheelchair and the occupant, using a trained model that has been machine-learned based on a plurality of first captured images for learning in which the wheelchair and the occupant are captured, and coordinate information of one or more body parts of the wheelchair and the occupant captured in each of the plurality of first captured images for learning; The pose estimation method has the following structure:
9. a learning unit that performs machine learning using a dataset including a plurality of first captured images for learning in which a wheelchair and an occupant of the wheelchair are captured, and coordinate information of one or more body parts of the wheelchair and the occupant captured in each of the plurality of first captured images for learning, to generate a trained model for estimating coordinate information of one or more body parts of the wheelchair and the occupant from a second captured image in which the wheelchair and the occupant are captured, and outputting the estimated coordinate information; A learning device comprising:
10. A trained model for estimating coordinate information of one or more body parts of a wheelchair and a passenger of the wheelchair, a machine learning system that performs machine learning based on a plurality of first captured images for learning in which the wheelchair and the occupant are captured and coordinate information of one or more body parts of the wheelchair and the occupant captured in each of the plurality of first captured images for learning, estimates the coordinate information of one or more body parts of the wheelchair and the occupant from a second captured image in which the wheelchair and the occupant are captured, and outputs the estimated coordinate information; A trained model for making a computer work.
Citation Information
Patent Citations
Image processing apparatus and image processing program
JP2008003794A
Attitude detector and behavior detector
JP2008146583A
Image processing device, image processing method and image processing program
JP2018206321A
Learning device, learning method, learning program, and object recognition device
JP2020123105A
