Specific program, specific method, and information processing device

Abstract

To calculate a feature amount of a person.SOLUTION: An information processor acquires image data captured by a camera and analyzes image frames constituting the acquired image data to generate skeleton information on a person included in the image frame and attribute information on the person included in the image frame. The information processor sets a first feature amount related to the height of the person to the person in the image frame on the basis of the generated attribute information on the person. The information processor sets a second feature amount related to the height of the person to the person in the image frame on the basis of the generated skeleton information and identifies a parameter of the camera on the basis of a difference between the first feature amount and the second feature amount.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a specific program and the like. [Background technology]

[0002] Video footage captured by cameras can be used in various investigations and can provide important information. For example, by analyzing camera images, it is possible to estimate not only a specific person's clothing and hairstyle, but also their height and other characteristics.

[0003] To estimate a person's height from a camera image, the camera's position and lens parameters are used, but it is unlikely that the parameters of an installed camera are recorded. In the following explanation, camera-related parameters will be referred to as "camera parameters" where appropriate.

[0004] For example, there is a conventional technique for estimating camera parameters using an image of a person whose height is known. FIG. 25 is a diagram for explaining the conventional technique. In the conventional technique, the camera parameters are estimated by utilizing the fact that the height of person 1a is known. Furthermore, in this conventional technique, the estimated camera parameters are used to calculate the height of person 1b, whose height is unknown.

[0005] FIG. 26 is a diagram for explaining estimation of camera parameters by the conventional technique. In the conventional technique, the height c of the camera 5, the angle θ of the camera 5, and the focal length f of the camera 5 are used as camera parameters. In the conventional technique, the image data 4 is analyzed to estimate the coordinates (x f ,y f ) and the coordinates above the head of person 2a (x h ,y h ) to identify the

[0006] In the prior art, the coordinates of the camera 5 and the feet of the person 2a (x f ,y f ) and the intersection point (X f ,Yf ,Z f ) is calculated as the coordinates of the feet of person 2b in the world coordinate system. As a premise, since the height of person 2a is known, the coordinates (X h ,Y h ,Z h In the conventional technology, the coordinates (x f ,y f ), (x h ,y h ) and the coordinates of the world coordinate system (X f ,Y f ,Z f ), (X h ,Y h ,Z h ) and estimate the camera parameters. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Shengzhe Li, et al. “A simplified nonlinear regression method for human height estimation in video surveillance” EURASIP Journal on Image and Video Processing 2015 Summary of the Invention [Problem to be solved by the invention]

[0008] However, the above-mentioned conventional technology has a problem in that if the height of a person in a camera image is not known, it is not possible to estimate the camera parameters and therefore it is not possible to calculate the feature values ​​of the person.

[0009] In one aspect, the present invention aims to provide an identification program, an identification method, and an information processing device that are capable of calculating a feature amount of a person. [Means for solving the problem]

[0010] In the first proposal, a computer is caused to execute the following process. The computer acquires image data captured by a camera and analyzes image frames constituting the acquired image data to generate skeletal information of a person included in the image frame and attribute information of the person included in the image frame. The computer assigns a first feature related to the person's height to the person in the image frame based on the generated attribute information of the person. The computer assigns a second feature related to the person's height to the person in the image frame based on the generated skeletal information, and identifies camera parameters based on the difference between the first feature and the second feature. [Effects of the Invention]

[0011] The feature amount of a person can be calculated. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of a system according to the present embodiment. [Figure 2] FIG. 2 is a diagram showing an example of video data. [Figure 3] FIG. 3 is a diagram for explaining the processing of the information processing device according to the present embodiment. [Figure 4] FIG. 4 is a functional block diagram showing the configuration of the information processing device according to this embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of the data structure of the learning model table. [Figure 6] FIG. 6 is a diagram illustrating an example of the data structure of the attribute table. [Figure 7] FIG. 7 is a diagram illustrating an example of the data structure of the evaluation table. [Figure 8] FIG. 8 is a diagram showing an example of a person detection result. [Figure 9] FIG. 9 is a diagram showing an example of a skeleton estimation result. [Figure 10] FIG. 10 is a diagram for explaining the processing of the parameter optimization unit. [Figure 11] FIG. 11 is a diagram for explaining the processing of the evaluation unit. [Figure 12] FIG. 12 is a flowchart illustrating a processing procedure of the information processing device according to the present embodiment. [Figure 13] FIG. 13 is a diagram for explaining the effect of the information processing device according to the present embodiment. [Figure 14] FIG. 14 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing apparatus of the embodiment. [Figure 15] FIG. 15 is a diagram showing video data of a person size with a large amount of information and video data of a person size with a small amount of information. [Figure 16] FIG. 16 is a diagram for explaining improvements in the first embodiment. [Figure 17] FIG. 17 is a diagram illustrating the process of the information processing device according to the second embodiment. [Figure 18] FIG. 18 is a diagram (1) for explaining the process of calculating the depth reduction ratio. [Figure 19] FIG. 19 is a diagram (2) for explaining the process of calculating the depth reduction ratio. [Figure 20] FIG. 20 is a diagram for explaining the process of placing a virtual person. [Figure 21] FIG. 21 is a functional block diagram illustrating a configuration of an information processing device according to the second embodiment. [Figure 22] FIG. 22 is a flowchart of a processing procedure of the information processing device according to the second embodiment. [Figure 23] FIG. 23 is a flowchart showing the processing procedure of the data extension process. [Figure 24] FIG. 24 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing apparatus of the second embodiment. [Figure 25] FIG. 25 is a diagram for explaining the prior art. [Figure 26] FIG. 26 is a diagram for explaining the estimation of camera parameters according to the conventional technique. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the identification program, identification method, and information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments. [Example]

[0014] Fig. 1 is a diagram showing an example of a system according to this embodiment. As shown in Fig. 1, this system includes a camera 10 and an information processing device 100. The camera 10 and the information processing device 100 are connected to each other via a network 30. Although the example shown in Fig. 1 shows only the camera 10, the system according to this embodiment may include other cameras.

[0015] The camera 10 captures video and transmits the captured video data to the information processing device 100. In the following description, the video data transmitted by the camera 10 to the information processing device 100 will be referred to as video data. In this embodiment, the description will be given using video data of a person.

[0016] Fig. 2 is a diagram showing an example of video data. As shown in Fig. 2, video data 20 includes a plurality of image frames in chronological order. Each image frame is assigned a frame number in ascending chronological order. One image frame is a still image captured by camera 10 at a certain timing.

[0017] The information processing device 100 acquires video data 20 from the camera 10 and sets camera parameters for the camera 10 based on the video data 20. The information processing device 100 calculates feature quantities of a person included in the video data based on the set camera parameters. In this embodiment, a case where height is calculated as a feature quantity of a person will be described, but other feature quantities of a person may also be calculated. For example, the information processing device 100 may calculate the length of a predetermined part of a person as a feature quantity of the person.

[0018] 3 is a diagram for explaining the processing of the information processing device according to this embodiment, in which an image frame F21 out of a plurality of image frames included in the video data 20 is used for explanation.

[0019] The coordinate system of the image frame F21 is the image coordinate system (x, y). The coordinate system in which the person actually exists is the world coordinate system (X, Y, Z). In the following explanation, the person in the image coordinate system shown in the image frame F21 will be referred to as person 21-1a, and the person in the world coordinate system will be referred to as person 21-2a.

[0020] The camera parameters of the camera 10 include the height c of the camera 10, the angle θ of the camera 10, and the focal length f of the camera 10. The camera parameters of the camera 10 are unknown, and the information processing device 100 sets the camera parameters to predetermined initial values.

[0021] The information processing device 100 analyzes the image frame F21 to identify skeletal data and attribute data of the person 21-1a. For example, the information processing device 100 inputs the image frame F21 into a machine-learned learning model to identify the skeletal data and attribute data of the person 21-1a. The learning model used by the information processing device 100 will be described later.

[0022] The skeletal data includes information about a plurality of joints of a person, and each joint is associated with a coordinate on an image frame. For example, the skeletal data corresponding to the image frame F21 includes the coordinates (x h1 ,y h1 ), foot coordinates (x f1 ,y f1 ) etc.

[0023] The attribute data includes information about a person's race, sex, etc. The information processing device 100 has an attribute table in which statistical values ​​such as average height and variance corresponding to the attribute data are set.

[0024] The information processing device 100 calculates the coordinates of the foot in the image coordinate system (x f1 ,y f1 ) onto the coordinates of the world coordinate system. For example, the information processing device 100 projects the coordinates (x f1 ,y f1 ) and the intersection point (X f1 ,Y f1 ,Z f1 ) is calculated as the coordinates of the feet of the person 21-2a in the world coordinate system.

[0025] The information processing device 100 identifies the average height and variance corresponding to the attribute data based on the attribute table. The information processing device 100 assigns a randomly allocated height L to the person 21-2a in the world coordinate axis system based on the identified average height and variance. The information processing device 100 calculates the coordinates of the head (X h1 ,Y h1 ,Z h1 ) is calculated.

[0026] The information processing device 100 calculates the coordinates of the head in the world coordinate system (X h1 ,Y h1 ,Z h1 ) into the coordinates of the image coordinate system. For example, the information processing device 100 projects the coordinates (X h1 ,Y h1 ,Z h1 ) and the coordinates (x' f1 ,y´ f1 ) is calculated and used as the coordinates of the head of the person 21-1a in the image coordinate system.

[0027] The information processing device 100 calculates the coordinates (x f1 ,y f1 ) to the coordinate (x' f1 ,y´ f1) is set as the “first feature amount.” The first feature amount corresponds to the height of the person 21-1a estimated based on the randomly assigned height L and the camera parameters of the camera 10.

[0028] The information processing device 100 calculates the coordinates (x f1 ,y f1 ) to the coordinates of the head (x h1 ,y h1 ) is set as a "second feature amount." The second feature amount corresponds to the height of the person 21-1a estimated based on the skeletal data.

[0029] The information processing device 100 optimizes the camera parameters of the camera 10 so that the difference between the first feature amount and the second feature amount becomes small while keeping the height L fixed.

[0030] After optimizing the camera parameters, the information processing device 100 changes the height L assigned to the person 21-2a and executes the above process again. The information processing device 100 repeatedly executes the process of identifying the optimized camera parameters for each different height L. As a result, a plurality of optimized camera parameters are calculated for each different height L.

[0031] The information processing device 100 evaluates the optimized camera parameters and acquires the camera parameters with the best evaluation value as the final result. The information processing device 100 calculates the height of the person 21-2a using the camera parameters acquired as the final result.

[0032] As described above, the information processing device 100 according to this embodiment randomly assigns a tentative height L based on the person's attribute data, and calculates a first feature amount of the person in the image calculated from the tentative height and a second feature amount of the person in the image calculated from the skeletal data. The information processing device 100 estimates camera parameters so as to minimize the error between the first feature amount and the second feature amount, and calculates the feature amount of the person. This makes it possible to estimate the camera parameters even if the person's height is unknown, and to use these camera parameters to improve the accuracy of the feature amount of the person, such as the person's height.

[0033] Next, an example of the configuration of the information processing device 100 according to this embodiment will be described. Fig. 4 is a functional block diagram showing the configuration of the information processing device according to this embodiment. As shown in Fig. 4, this information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0034] The communication unit 110 is connected to the camera 10 via the network 30 and receives video data. For example, the communication unit 110 is realized by a network interface card (NIC) or the like. The communication unit 110 may also be connected to other external devices or the like via the network 30.

[0035] The input unit 120 is an input device that inputs various types of information to the information processing device 100. The input unit 120 corresponds to a keyboard, a mouse, a touch panel, or the like.

[0036] The display unit 130 is a display device that displays information output from the control unit 150. The display unit 130 corresponds to a liquid crystal display, an organic EL (Electro Luminescence) display, a touch panel, or the like.

[0037] The storage unit 140 includes a video buffer 141, a learning model table 142, an attribute table 143, and an evaluation table 144. The storage unit 140 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0038] The video buffer 141 is a buffer that stores the video data 20 transmitted from the camera 10. The video data 20 includes a plurality of image frames in chronological order. Each image frame is assigned a frame number in ascending chronological order.

[0039] The learning model table 142 is a table that includes various learning models. Fig. 5 is a diagram showing an example of the data structure of the learning model table. As shown in Fig. 5, the learning model table 142 includes a person detection model 142a, an attribute determination model 142b, and a skeleton estimation model 142c.

[0040] The person detection model 142a is a machine learning model that, when a time-series image frame included in video data is input, detects a person area included in the image frame and tracks the person area. A person ID for identifying the person is assigned to each person detected from the image frame. The person detection model 142a is realized by a machine learning model such as DeepSort.

[0041] The attribute determination model 142b is a machine learning model that, when a person's area (face image, etc.) in an image frame is input, outputs attribute data of the corresponding person.

[0042] The skeleton estimation model 142c is a machine learning model that outputs skeleton data of a person when a region of the person in an image frame (a whole-body image) is input. The skeleton estimation model 142c can be realized by a machine learning model such as OpenPose.

[0043] The person detection model 142a, the attribute determination model 142b, and the skeleton estimation model 142c correspond to a neural network (NN) or the like. It is assumed that machine learning has been performed in advance on each of the person detection model 142a, the attribute determination model 142b, and the skeleton estimation model 142c based on a training dataset. For example, the training dataset for the attribute determination model 142b is a training dataset that has a plurality of training data sets in which a person's face image is used as input data and attribute data (race, gender) is used as a correct answer label.

[0044] The attribute table 143 is a table that defines statistical values ​​corresponding to attribute data. FIG. 6 is a diagram showing an example of the data structure of the attribute table. As shown in FIG. 6, the attribute table 143 associates attribute data with statistical values. The attribute data includes race and sex. The statistical values ​​include average height and variance.

[0045] The average height indicates the average value of the heights of people of the corresponding race and gender. The variance indicates the variance (degree of dispersion) of the heights of people of the corresponding race and gender. For example, the attribute table 143 shows that the average height for the race "Asian" and the gender "Male" is "172 cm" and the variance is "8".

[0046] The evaluation table 144 holds various information related to camera parameters. Fig. 7 is a diagram showing an example of the data structure of the evaluation table. As shown in Fig. 7, the evaluation table 144 has the number of executions, heights of persons ID1 to IDn, c, θ, f, and evaluation values.

[0047] The number of executions indicates the number of times the camera parameters have been optimized. In this embodiment, the number of executions "1" corresponds to randomly allocating height L, fixing height L, and performing a series of processes to optimize the camera parameters once. In the number of executions "2," height L is reassigned and the camera parameters are optimized. The same applies to the number of executions "3" and onward.

[0048] The height of a person ID indicates the height randomly assigned to the person identified by the person ID. For example, the height of person ID 1 indicates the height randomly assigned to the person with person ID "1". A person ID is information that uniquely identifies a person. Although not explained in Figure 3, if multiple people are detected from the same image frame, a person ID is assigned to each person. c, θ, and f are set to the values ​​of the optimized camera parameters. The evaluation value is an evaluation value for the camera parameters, and the smaller the value, the better the camera parameters.

[0049] Returning to the description of Fig. 4, the control unit 150 includes an acquisition unit 151, a detection unit 152, an attribute determination unit 153, a skeleton estimation unit 154, a parameter optimization unit 155, an evaluation unit 156, and a feature calculation unit 157. The control unit 150 is realized by, for example, a central processing unit (CPU) or a micro processing unit (MPU). The control unit 150 may also be implemented by, for example, an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0050] Acquisition unit 151 acquires video data 20 from camera 10 via communication unit 110. Acquisition unit 151 registers the acquired video data 20 in video buffer 141. Acquisition unit 151 continuously acquires video data 20 and registers it in video buffer 141 while camera 10 is capturing images.

[0051] The detection unit 152 acquires image frames in chronological order from the video buffer 141 and inputs the image frames to the person detection model 142a to detect the areas of people included in the image frames. FIG. 8 is a diagram showing an example of the person detection result. In the example shown in FIG. 8, the areas of people 21-1a, 22-1a, 23-1a, 24-1a, and 25-1a are detected from the image frame F21. Person IDs are assigned to the areas of people 21-1a, 22-1a, 23-1a, 24-1a, and 25-1a.

[0052] The detection unit 152 outputs information on the detection result to the attribute determination unit 153, the skeleton estimation unit 154, and the parameter optimization unit 155. The information on the detection result includes information on the person's area (image within the area), the person ID assigned to each area, the frame number of the image frame, etc.

[0053] The detection unit 152 repeatedly executes the above process for each image frame of the video data 20 stored in the video buffer 141 in chronological order.

[0054] The attribute determination unit 153 extracts a face image from a person's area included in the information of the detection result, and inputs the face image to the attribute determination model 142b to determine the person's race and gender. The attribute determination unit 153 determines the person's race and gender for each person's area included in the information of the detection result. The attribute determination unit 153 generates attribute data that associates a person ID with the person's race and gender, and outputs the attribute data to the parameter optimization unit 155.

[0055] The attribute determination unit 153 repeatedly executes the above process every time it acquires information on the detection result from the detection unit 152.

[0056] The skeleton estimation unit 154 estimates the skeleton of a person by inputting information on the area of ​​the person (whole-body image) included in the information on the detection result into the skeleton estimation model 142c. FIG. 9 is a diagram showing an example of a skeleton estimation result. As shown in FIG. 9, the skeleton estimation result 40 includes a plurality of joints 40-1, 40-2, 40-3, 40-4, 40-5, 40-6, 40-7, 40-8, 40-9, 40-10, 40-11, 40-12, and 40-13. Although not shown, each of the joints 40-1 to 40-13 includes coordinates in the image coordinate system and a joint type. For example, the joint 40-1 corresponds to the head. The joint 40-12 (40-13) corresponds to the foot.

[0057] The skeleton estimation unit 154 estimates the skeleton for each person's area included in the information of the detection result. The skeleton estimation unit 154 generates skeleton data in which the person ID is associated with the person's skeleton estimation result, and outputs the data to the parameter optimization unit 155, the evaluation unit 156, and the feature calculation unit 157.

[0058] The skeleton estimation unit 154 repeatedly executes the above process every time it acquires information on the detection result from the detection unit 152.

[0059] The parameter optimization unit 155 is a processing unit that optimizes camera parameters by executing the following processing. The processing of the parameter optimization unit 155 will be explained with reference to FIG. 3. The parameter optimization unit 155 optimizes the coordinates (x h1 ,y h1 ), foot coordinates (x f1 ,y f1 ) to identify the

[0060] The parameter optimization unit 155 identifies the race and gender of the person 21-1a based on the attribute data. The parameter optimization unit 155 compares the race and gender of the person 21-1a with the attribute table 143 to identify the average height and variance of the race and gender corresponding to the person 21-1a. The parameter optimization unit 155 randomly determines the height L to be assigned to the person 21-2a in the world coordinate axis system based on the average height and variance. For example, the parameter optimization unit 155 randomly determines the height L to be assigned to the person 21-2a by sampling random numbers from a normal distribution having the identified average height and variance.

[0061] The parameter optimization unit 155 sets initial values ​​for the camera parameters. For example, the parameter optimization unit 155 sets the initial values ​​as c=300, θ=30, and f=500. The user may change the initial values ​​by operating the input unit 120.

[0062] The parameter optimization unit 155 calculates the coordinates of the foot in the image coordinate system (x f1 ,y f1) onto the coordinates of the world coordinate system. For example, the parameter optimization unit 155 projects the coordinates (x f1 ,y f1 ) and the intersection point (X f1 ,Y f1 ,Z f1 ) is calculated as the coordinates of the feet of the person 21-2a in the world coordinate system.

[0063] The parameter optimization unit 155 calculates the coordinates (X f1 ,Y f1 ,Z f1 ) and the height L, the coordinates of the head of the person 21-2a (X h1 ,Y h1 ,Z h1 ) is calculated.

[0064] The parameter optimization unit 155 calculates the coordinates of the head in the world coordinate system (X h1 ,Y h1 ,Z h1 ) into the coordinates of the image coordinate system. For example, the parameter optimization unit 155 back-projects the coordinates (X h1 ,Y h1 ,Z h1 ) and the coordinates (x' f1 ,y´ f1 ) is calculated and used as the coordinates of the head of the person 21-1a in the image coordinate system.

[0065] The parameter optimization unit 155 calculates the coordinates of the foot in the image coordinate system (x f1 ,y f1 ) to the coordinate (x' f1 ,y´ f1 ) in the image coordinate system is set as the first feature amount. f1 ,y f1 ) to the coordinates of the head (x h1 ,y h1 ) is set as the second feature.

[0066] The parameter optimization unit 155 also performs the above process for the persons 22-1a to 25-1a other than the person 21-1a included in the image frame F21 shown in FIG. 3, thereby setting the first feature amount and the second feature amount.

[0067] 10 is a diagram for explaining the processing of the parameter optimization unit. As described above, the parameter optimization unit 155 calculates the coordinates of the feet (x f1 ,y f1 ) and the head coordinates (x h1 ,y h1 The parameter optimization unit 155 also randomly assigns a height L1 to the person 21-2a depending on the race and sex of the person 21-2a, and determines the coordinates (x' f1 ,y´ f1 ) is calculated. In order to distinguish it from the heights assigned to other persons described later, the height L assigned to the person 21-2a is expressed as height L1. The first feature amount of the person 21-1a is calculated by calculating the coordinates (x f1 ,y f1 ) to the coordinate (x' f1 ,y´ f1 The second feature of the person 21-1a is the distance to the coordinate (x f1 ,y f1 ) to the coordinates of the head (x h1 ,y h1 ) is the distance to

[0068] The parameter optimization unit 155 calculates the coordinates of the feet (x f2 ,y f2 ) and the head coordinates (x h2 ,y h2 For convenience, the person in the world coordinate system corresponding to the person 22-1a is referred to as person 22-2a. The parameter optimization unit 155 randomly assigns a height L2 to the person 22-2a according to the race and sex, and determines the coordinates (x' f2 ,y´ f2 The first feature of the person 22-1a is calculated by the coordinates (x f2 ,y f2 ) to the coordinate (x'f2 ,y´ f2 The second feature of the person 22-1a is the distance to the coordinate (x f2 ,y f2 ) to the coordinates of the head (x h2 ,y h2 ) is the distance to

[0069] The parameter optimization unit 155 calculates the coordinates of the feet (x f3 ,y f3 ) and the head coordinates (x h3 ,y h3 For convenience, the person in the world coordinate system corresponding to the person 23-1a is referred to as person 23-2a. The parameter optimization unit 155 randomly assigns a height L3 to the person 23-2a according to the race and sex, and determines the coordinates (x' f3 ,y´ f3 The first feature of the person 23-1a is calculated by the coordinates (x f3 ,y f3 ) to the coordinate (x' f3 ,y´ f3 The second feature of the person 23-1a is the distance to the coordinate (x f3 ,y f3 ) to the coordinates of the head (x h3 ,y h3 ) is the distance to

[0070] The parameter optimization unit 155 calculates the coordinates of the feet (x f4 ,y f4 ) and the head coordinates (x h4 ,y h4 For convenience, the person in the world coordinate system corresponding to the person 24-1a is referred to as person 24-2a. The parameter optimization unit 155 randomly assigns a height L4 to the person 24-2a according to the race and sex, and determines the coordinates (x' f4 ,y´ f4 The first feature of the person 24-1a is calculated by the coordinates (x f4 ,y f4 ) to the coordinate (x' f4 ,y´ f4The second feature of the person 24-1a is the distance to the coordinate (x f4 ,y f4 ) to the coordinates of the head (x h4 ,y h4 ) is the distance to

[0071] The parameter optimization unit 155 calculates the coordinates of the feet (x f5 ,y f5 ) and the head coordinates (x h5 ,y h5 For convenience, the person in the world coordinate system corresponding to the person 25-1a is referred to as person 25-2a. The parameter optimization unit 155 randomly assigns a height L5 to the person 25-2a according to the race and sex, and determines the coordinates (x' f5 ,y´ f5 The first feature of the person 25-1a is calculated by the coordinates (x f5 ,y f5 ) to the coordinate (x' f5 ,y´ f5 The second feature of person 25-1a is the distance to the coordinate (x f5 ,y f5 ) to the coordinates of the head (x h5 ,y h5 ) is the distance to

[0072] In the following description, the difference between the first feature amount and the second feature amount of person 21-1a will be referred to as the "difference in feature amount of person 21-1a." The difference between the first feature amount and the second feature amount of person 22-1a will be referred to as the "difference in feature amount of person 22-1a." The difference between the first feature amount and the second feature amount of person 23-1a will be referred to as the "difference in feature amount of person 23-1a."

[0073] The difference between the first feature amount and the second feature amount of person 24-1a is referred to as the "difference in feature amount of person 24-1a." The difference between the first feature amount and the second feature amount of person 25-1a is referred to as the "difference in feature amount of person 25-1a."

[0074] The parameter optimization unit 155 optimizes the camera parameters so as to reduce the differences in the feature amounts of the persons 21-1a to 25-1a while fixing the heights L1 to L5 assigned to the persons 21-1a to 25-1a. The parameter optimization unit 155 registers the heights L1 to L5 assigned to the persons 21-1a to 25-1b and information on the optimized camera parameters as a record with the number of executions "1" in the evaluation table 144.

[0075] The parameter optimization unit 155 reassigns the heights L1 to L5 assigned to the persons 21-1a to 25-1a, and resets the camera parameters to their initial values. The parameter optimization unit 155 optimizes the camera parameters so as to reduce differences in feature amounts among the persons 21-1a to 25-1a. The parameter optimization unit 155 registers the heights L1 to L5 assigned to the persons 21-1a to 25-1a and information on the optimized camera parameters as a record for the number of executions "2" in the evaluation table 144. The parameter optimization unit 155 repeatedly executes the above process, thereby registering records for each number of executions in the evaluation table 144. The value input to the evaluation value in the evaluation table 144 is calculated by the evaluation unit 156, which will be described later.

[0076] Returning to the explanation of FIG. 4, the evaluation unit 156 evaluates each of the camera parameters corresponding to each execution count registered in the evaluation table 144. FIG. 11 is a diagram for explaining the processing of the evaluation unit. In FIG. 11, the processing of the evaluation unit 156 will be explained using person 21-1a included in image frame F22. Image frame F22 is the image frame following image frame F21 described in FIG. 3 etc.

[0077] A case will be described in which the evaluation unit 156 calculates the evaluation value of the camera parameters for the execution count "1." The evaluation unit 156 acquires the height L1 randomly assigned to the person 21-1a and the camera parameters based on the record in the evaluation table 144 corresponding to the execution count "1."

[0078] The evaluation unit 156 calculates the coordinates (x h1-1 ,y h1-1 ), foot coordinates (x f1-1 ,y f1-1 The evaluation unit 156 uses the height L1 acquired from the evaluation table 144 and the camera parameters to project onto the world coordinate system and back-project from the world coordinate system in the same manner as the parameter optimization unit 155, thereby determining the coordinates (x' h1-1 ,y´ h1-1 ) to identify the

[0079] The evaluation unit 156 calculates the coordinates (x f1-1 ,y f1-1 ) to the coordinate (x' f1-1 ,y´ f1-1 ) and the coordinate (x f1-1 ,y f1-1 ) to the coordinates of the head (x h1-1 ,y h1-1 ) and calculate the difference in distance.

[0080] The evaluation unit 156 also repeatedly executes the above process for image frames F23 to F23+M (M is a predetermined natural number) to calculate the difference related to the person 21-1a. The evaluation unit 156 calculates the variance based on each calculated difference as the variance related to the person 21-1a. If the camera parameters are appropriate, the difference calculated from each image frame will be constant and the variance will be small.

[0081] The evaluation unit 156 calculates the variance for the other persons 22-1a to 25-1a in the same manner as for the person 21-1a. The evaluation unit 156 calculates the total value of the variances for the persons 21-1a to 25-1a as the evaluation value of the camera parameters for the execution count "1" and registers it in the evaluation table 144.

[0082] The evaluation unit 156 similarly calculates the evaluation values ​​of the camera parameters for the other execution counts and registers them in the evaluation table 144. Of the multiple camera parameters registered in the evaluation table 144, the camera parameter with the smallest evaluation value becomes the optimal camera parameter that is finally determined. In the example shown in Fig. 7, the camera parameter corresponding to the execution count "2" has the smallest evaluation value, so the camera parameters (c = 300, θ = 25, f = 540) corresponding to the execution count "2" become the camera parameters that are finally identified.

[0083] The evaluation unit 156 outputs the camera parameters that result in the smallest evaluation value to the feature amount calculation unit 157 .

[0084] The feature amount calculation unit 157 calculates the height of the person included in the image frame based on the camera parameters acquired from the evaluation unit 156. The feature amount calculation unit 157 identifies the coordinates of the person's head and feet in the image coordinate system based on the skeletal data. The feature amount calculation unit 157 projects the coordinates of the person's head and feet in the image coordinate system onto the coordinates of the person's head and feet in the world coordinate system based on the camera parameters. The feature amount calculation unit 157 calculates the distance from the coordinates of the feet to the coordinates of the head in the world coordinate system as the person's height. The feature amount calculation unit 157 may display information on the calculated feature amount (height) of the person on the display unit 130 or may transmit it to an external device.

[0085] The feature amount calculation unit 157 may calculate the length of a predetermined part of a person as a feature amount of the person based on the skeletal data and the camera parameters.

[0086] Next, an example of a processing procedure of the information processing device according to this embodiment will be described. Fig. 12 is a flowchart showing the processing procedure of the information processing device according to this embodiment. The acquisition unit 151 of the information processing device 100 receives video data from the camera 10 and registers it in the video buffer 141 (step S101).

[0087] The detection unit 152 of the information processing device 100 detects the area of ​​each person from the image frame (step S102). The attribute determination unit 153 of the information processing device 100 generates attribute data of each person (step S103). The skeleton estimation unit 154 of the information processing device 100 generates skeleton data of each person (step S104).

[0088] The parameter optimization unit 155 of the information processing device 100 identifies the average height and the variance based on the attribute data and the attribute table 143 (step S105). The parameter optimization unit 155 randomly determines the height to be assigned to the person based on the average height and the variance (step S106).

[0089] The parameter optimization unit 155 sets a first feature amount and a second feature amount for each person (step S107). The parameter optimization unit 155 optimizes the camera parameters for each person so that the difference between the first feature amount and the second feature amount is minimized (step S108).

[0090] The evaluation unit 156 of the information processing device 100 calculates evaluation values ​​of the camera parameters (step S109). If the process of calculating the camera parameters has not been repeated N times (step S110, No), the information processing device 100 proceeds to step S106. On the other hand, if the process of calculating the camera parameters has been repeated N times (step S110, Yes), the information processing device 100 proceeds to step S111.

[0091] The evaluation unit 156 selects the camera parameters that result in the smallest evaluation value (step S111). The feature amount calculation unit 157 of the information processing device 100 calculates the feature amount of the person using the selected camera parameters (step S112).

[0092] Next, the effects of the information processing device 100 according to this embodiment will be described. The information processing device 100 randomly assigns a tentative height L from the person's attribute data, and calculates a first feature amount of the person in the image calculated from the tentative height and a second feature amount of the person in the image calculated from the skeletal data. The information processing device 100 estimates camera parameters so as to minimize the error between the first feature amount and the second feature amount, and calculates the feature amount of the person. As a result, even if the person's height is unknown, the camera parameters can be estimated, and the accuracy of the feature amount of the person, such as the person's height, can be improved using these camera parameters.

[0093] The information processing device 100 assigns a random virtual height to a person based on the average height and variance corresponding to the person's race and gender, and identifies a first feature amount of the person in the image calculated from the virtual height. This makes it possible to set a likely height according to the race and gender, and identify the first feature amount, even if the person's height is unknown.

[0094] The information processing device 100 calculates the evaluation value of each camera parameter and selects the camera parameter with the smallest evaluation value as the final camera parameter, thereby further improving the accuracy of estimating the camera parameters.

[0095] Fig. 13 is a diagram for explaining the effect of the information processing device according to this embodiment. The vertical axis of bar graph G3 in Fig. 13 indicates the magnitude of error (mean absolute error). Bar B1 indicates the error when a person's height is calculated by estimating camera parameters using a known height. Bar B2 indicates the error in the person's height calculated by the information processing device 100. For both bars B1 and B2, the error is around 2 cm, and it can be confirmed that by using the information processing device 100, accuracy equivalent to that when the height is known can be achieved.

[0096] Next, an example of the hardware configuration of a computer that realizes the same functions as the information processing device 100 described in the above embodiment will be described. Fig. 14 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment.

[0097] 14, the computer 200 has a CPU 201 that executes various types of arithmetic processing, an input device 202 that accepts data input from a user, and a display 203. The computer 200 also has a communication device 204 that exchanges data with the camera 10, external devices, etc. via a wired or wireless network, and an interface device 205. The computer 200 also has a RAM 206 that temporarily stores various types of information, and a hard disk drive 207. The devices 201 to 207 are connected to a bus 208.

[0098] The hard disk drive 207 stores an acquisition program 207a, a detection program 207b, an attribute determination program 207c, a skeleton estimation program 207d, a parameter optimization program 207e, an evaluation program 207f, and a feature calculation program 207g. The CPU 201 reads out each of the programs 207a to 207g and loads them into the RAM 206.

[0099] The acquisition program 207a functions as an acquisition process 206a. The detection program 207b functions as a detection process 206b. The attribute determination program 207c functions as an attribute determination process 206c. The skeleton estimation program 207d functions as a skeleton estimation process 206d. The parameter optimization program 207e functions as a parameter optimization process 206e. The evaluation program 207f functions as an evaluation process 206f. The feature calculation program 207g functions as a feature calculation process 206g.

[0100] The processing of the acquisition process 206a corresponds to the processing of the acquisition unit 151. The processing of the detection process 206b corresponds to the processing of the detection unit 152. The processing of the attribute determination process 206c corresponds to the processing of the attribute determination unit 153. The processing of the skeleton estimation process 206d corresponds to the processing of the skeleton estimation unit 154. The processing of the parameter optimization process 206e corresponds to the processing of the parameter optimization unit 155. The processing of the evaluation process 206f corresponds to the processing of the evaluation unit 156. The processing of the feature calculation process 206g corresponds to the processing of the feature calculation unit 157.

[0101] It should be noted that each of the programs 207a to 207g does not necessarily have to be stored in the hard disk drive 207 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207g. [Example]

[0102] Before describing the second embodiment, improvements to the first embodiment will be described. In the first embodiment, unknown camera parameters are estimated for a moving person who appears in video data. Here, in the first embodiment, if only video data capturing a person moving within a narrow depth range can be acquired, the amount of information about the person's size that can be acquired when optimizing the camera parameters is reduced. If the amount of information about the person's size is reduced, it is not possible to increase the constraints during estimation, and the estimation accuracy of the camera parameters may decrease.

[0103] In the following description, video data of a person moving over a wide depth range will be referred to as "video data with a large amount of information about the person size," while video data of a person moving over a narrow depth range will be referred to as "video data with a small amount of information about the person size."

[0104] Fig. 15 shows video data with a large amount of information about people's sizes and video data with a small amount of information about people's sizes. In video data 50a in Fig. 15, people are present on both the foreground and the background of video data 50a, with a wide depth range for the people and a large amount of information about the people's sizes. On the other hand, in video data 50b, the people are present near the center, with a narrow depth range for the people and a small amount of information about the people's sizes.

[0105] Fig. 16 is a diagram for explaining improvements of this embodiment 1. In graph G4 of Fig. 16, the horizontal axis corresponds to a certain camera parameter x, and the vertical axis corresponds to a certain camera parameter y. For example, the position of the optimal solution of the camera parameters on graph G4 is assumed to be p1. The position of a certain local solution of the camera parameters on graph G4 is assumed to be p2.

[0106] When the information processing device 100 of the first embodiment estimates the camera parameters using the video data 50a, the estimated value on the graph G4 is at p3, and a value close to the optimal solution can be obtained. On the other hand, when the information processing device 100 estimates the camera parameters using the video data 50b, the estimated value on the graph G4 is at p4, and the value is closer to the local solution than the optimal solution.

[0107] As described above, in the first embodiment, if video data with a large amount of information about a person size cannot be acquired, the accuracy of estimating parameters decreases.

[0108] Next, a description will be given of an information processing device according to Example 2. The information processing device according to Example 2 virtually moves a person included in video data, thereby increasing the amount of information on the size of the person included in the video data and improving the estimation accuracy of camera parameters.

[0109] FIG. 17 is a diagram for explaining the processing of the information processing device according to the second embodiment. For example, the information processing device acquires video data 55 and executes the processing of steps S1, S2, and S3. It is assumed that one person is shown in the video data 55. The video data 55 includes image frames in a time series. For convenience, FIG. 17 shows the same person at different times. Person 55a indicates the person in the image frame at time t, and person 55b indicates the person in the image frame at time t+1.

[0110] Step S1 in Fig. 17 will be described. In step S1, the information processing device estimates the depth reduction ratio of the person size. The information processing device estimates the depth reduction ratio based on the change in size between person 55a and person 55b, and the change in y coordinate between person 55a and person 55b.

[0111] Step S2 in Fig. 17 will be described. In step S2, the information processing device analyzes the video data 55 and extracts candidate points on the ground. The example shown in Fig. 17 shows a case where the information processing device extracts candidate points p1-1, p1-2, p1-3, p1-4, p1-5, p1-6, p1-7, and p1-8 on the ground.

[0112] Step S3 in Fig. 17 will be described. The information processing device identifies candidate points at which to place virtual people from candidate points p1-1 to p1-8 on the ground. In the example shown in Fig. 17, a case will be described in which the information processing device places virtual people 56a and 56b at candidate points p1-4 and p1-8. The information processing device calculates the size of the virtual person 56a placed at candidate point p1-4 based on the size of the person 55a and the depth reduction ratio calculated in step S1. The information processing device calculates the size of the virtual person 56a placed at candidate point p1-8 based on the size of the person 55a and the depth reduction ratio calculated in step S1. The information processing device places the virtual people 56a and 56b of the calculated sizes at candidate points p1-4 and p1-8.

[0113] The information processing device estimates camera parameters based on the people 55a and 55b and the virtual people 56a and 56b. By using information on the virtual people 56a and 56b in addition to information on the people 55a and 55b, the amount of information on the person sizes increases, making it possible to prevent the solution for the camera parameters from becoming a local solution.

[0114] Next, an example of the process of calculating the depth reduction ratio described in step S1 of FIG. 17 will be described. The information processing device determines, based on the video data, whether or not a person depicted in the video data has movement in the depth direction. For example, the information processing device detects a person's area from each image frame included in the video data and identifies the y coordinate of the person's feet. If the amount of change in the y coordinate per predetermined time is equal to or greater than a threshold, the information processing device determines that the movement of the person depicted in the video data has depth. On the other hand, if the amount of change in the y coordinate over the predetermined time is less than the threshold, the information processing device determines that the movement of the person depicted in the video data does not have depth.

[0115] FIG. 18 is a diagram (1) for explaining the process of calculating the depth reduction rate. The horizontal axis of graph G5 in FIG. 18 corresponds to time, and the vertical axis corresponds to the y coordinate of the foot of a person included in video data 60 in FIG. 18. If the amount of change in the y coordinate per predetermined time in graph G5 is equal to or greater than a threshold, the information processing device determines that the person shown in video data 60 is moving in the depth direction. If the person shown in video data 60 is moving in the depth direction, the information processing device calculates the depth reduction rate based on the skeletal data of the person at each time.

[0116] In FIG. 18, as an example, the walking direction of the person is from the far side to the near side of the video data 60. Here, the y coordinate of the person's foot at time t1 is set to y1. The y coordinate of the person's foot at time t2 is set to y2. The height obtained from the person's skeletal data at time t1 is set to H1. The height obtained from the person's skeletal data at time t2 is set to H2. The information processing device calculates the depth reduction rate α based on equation (1).

[0117] Depth reduction ratio α = -(H2 - H1) / (y2 - y1) (1)

[0118] FIG. 19 is a diagram (2) for explaining the process of calculating the depth reduction rate. The horizontal axis of graph G6 in FIG. 19 corresponds to time, and the vertical axis corresponds to the y coordinate of the foot of a person included in video data 65 in FIG. 19. If the amount of change in the y coordinate per predetermined time in graph G6 is less than a threshold, the information processing device determines that the person shown in video data 65 is not moving in the depth direction. If the person shown in video data 65 is not moving in the depth direction, the information processing device calculates the depth reduction rate by comparing the areas of the person's body parts at a certain time.

[0119] In FIG. 19, as an example, the walking direction of the person is from the right side to the left side of the video data 65. In this Example 2, the area of ​​the human body part to be compared is the area of ​​the foot part. In the following explanation, the area of ​​the foot part is referred to as the "foot area." In area 65a of the video data 65, the left foot area of ​​the person is designated as V1, and the right foot area is designated as V2. The y coordinate of the part where the left foot area V1 contacts the ground is designated as y1. The y coordinate of the part where the right foot area V2 contacts the ground is designated as y2. The information processing device calculates the depth reduction rate α based on equation (2). For example, foot area V1 is the area of ​​the left shoe. Foot area V2 is the area of ​​the right shoe.

[0120] Depth reduction rate α=-(V2-V1) / (y2-y1)...(2)

[0121] Next, the process of extracting point candidates on the ground, which was explained in step S2 of Fig. 17, will be explained. The information processing device 100 performs semantic segmentation on the video data and assigns labels to all pixels in the image frame. The labels include information for identifying objects such as pixels and the ground. The information processing device refers to the labeling results and identifies pixels assigned with labels corresponding to the ground as areas of the ground.

[0122] The information processing device may input the video data into a learning model such as YOLO (You Look Only Once) to determine the scene of the video data. For example, if the scene of the video data is a parking lot, the information processing device detects line components included in the image frame and identifies the intersections of the detected line components as candidate points. In the example described in FIG. 17, the information processing device detects candidate points p1-1 to p1-8 corresponding to the intersections of the line components.

[0123] The information processing device may input video data into a learning model such as YOLO to determine the scene of the video data, and if the scene of the video data is not a specific scene such as a parking lot, may set a predetermined number of candidate points at predetermined intervals in the area of ​​the ground. The number of candidate points to be set is assumed to be preset.

[0124] Next, the process of placing a virtual person explained in step S3 of Fig. 17 will be described. Fig. 20 is a diagram for explaining the process of placing a virtual person. The information processing device determines a placement point from the candidate points p1-1 to p1-8 obtained in step S2. For example, the information processing device preferentially determines a candidate point on the other side of the position where the person is present as the placement point. In Fig. 20, as an example, the placement point is set to candidate point p1-4.

[0125] The information processing device calculates the coordinate y of the feet of the skeletal data of the person 55a at a certain time t and the coordinate y' of the placement point (candidate point p1-4) where the virtual person 56a is to be placed. The information processing device calculates the height H' of the virtual person 56a based on equation (3). α is the depth reduction ratio calculated in step S1.

[0126] The height of the virtual person is H' = α(y' - y) (3)

[0127] After calculating the height H' of the virtual person 56a, the information processing device adjusts the size of the skeletal data of the virtual person 56a based on the height H', and places the information of the virtual person 56a at the placement point (candidate point p1-4).

[0128] Next, a configuration example of an information processing device according to Example 2 will be described. Fig. 21 is a functional block diagram showing the configuration of the information processing device according to Example 2. As shown in Fig. 21, the information processing device 300 includes a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 350.

[0129] The explanations regarding the communication unit 110, input unit 120, display unit 130, and memory unit 140 are the same as the explanations regarding the communication unit 110, input unit 120, display unit 130, and memory unit 140 described in Figure 4, so the same symbols are used and the explanations are omitted.

[0130] The control unit 350 includes an acquisition unit 151, a detection unit 152, an attribute determination unit 153, a skeleton estimation unit 154, a parameter optimization unit 155, an evaluation unit 156, a feature calculation unit 157, and a data extension unit 350a. The control unit 350 is realized by, for example, a CPU or an MPU. The control unit 150 may also be implemented by, for example, an integrated circuit such as an ASIC or an FPGA.

[0131] The acquisition unit 151, detection unit 152, attribute determination unit 153, skeleton estimation unit 154, parameter optimization unit 155, evaluation unit 156, and feature calculation unit 157 are the same as those described in Fig. 4. The parameter optimization unit 155 further uses skeleton data of the virtual person generated by a data extension unit 350a (described later) to optimize the camera parameters.

[0132] The data expansion unit 350a executes the processes described with reference to FIGS. 17 to 20 to generate skeletal data of the virtual person placed at the placement point of the image frame.

[0133] For example, as described in step S1 of FIG. 17, and in FIGS. 18 and 19, the data expansion unit 350a estimates the depth reduction rate α based on the change in size between person 55a and person 55b, and the change in the y coordinate between person 55a and person 55b.

[0134] The data extension unit 350a analyzes the video data 55 and extracts point candidates on the ground, as described in step S2 of FIG.

[0135] As described in step S3 of FIG. 17 and FIG. 20, the data expansion unit 350a identifies candidate points on the ground at which to place a virtual person from candidate points p1-1 to p1-8. The data expansion unit 350a calculates the size of the virtual person 56a to be placed at candidate point p1-8 based on the size of the person 55a and the depth reduction rate α calculated in step S1. The data expansion unit 350a generates skeletal data of the virtual person 56a by adjusting the skeletal data of the person 55a to match the calculated size. For example, the data expansion unit 350a may generate skeletal data of the virtual person 56a by multiplying the length between each joint in the skeletal data of the person 55a by the depth reduction rate α. The data expansion unit 350a outputs the skeletal data of the virtual person 56a to the parameter optimization unit 155.

[0136] Next, a description will be given of an example of a processing procedure of the information processing device 300 according to the present embodiment 2. Fig. 22 is a flowchart showing the processing procedure of the information processing device according to the present embodiment 2. As shown in Fig. 22, the acquisition unit 151 of the information processing device 300 receives video data from the camera 10 and registers it in the video buffer 141 (step S201).

[0137] The detection unit 152 of the information processing device 300 detects the area of ​​each person from the image frame (step S202). The attribute determination unit 153 of the information processing device 300 generates attribute data of each person (step S203). The skeleton estimation unit 154 of the information processing device 300 generates skeleton data of each person (step S204).

[0138] The data extension unit 350a of the information processing device 300 executes the data extension process (step S205).

[0139] The parameter optimization unit 155 of the information processing device 300 identifies the average height and the variance based on the attribute data and the attribute table 143 (step S206). The parameter optimization unit 155 randomly determines the height to be assigned to the person based on the average height and the variance (step S207).

[0140] The parameter optimization unit 155 sets a first feature amount and a second feature amount for each person (step S208). The parameter optimization unit 155 optimizes the camera parameters for each person so that the difference between the first feature amount and the second feature amount is minimized (step S209).

[0141] The evaluation unit 156 of the information processing device 300 calculates evaluation values ​​of the camera parameters (step S210). If the process of calculating the camera parameters has not been repeated N times (step S211, No), the information processing device 300 proceeds to step S207. On the other hand, if the process of calculating the camera parameters has been repeated N times (step S211, Yes), the information processing device 300 proceeds to step S212.

[0142] The evaluation unit 156 selects the camera parameters that result in the smallest evaluation value (step S212). The feature amount calculation unit 157 of the information processing device 300 calculates the feature amount of the person using the selected camera parameters (step S213).

[0143] Next, an example of the processing procedure for the data extension processing described in step S205 of Fig. 22 will be described. Fig. 23 is a flowchart showing the processing procedure for the data extension processing. As shown in Fig. 23, the data extension unit 350a of the information processing device 300 acquires skeletal data of the same person included in each image frame (step S301).

[0144] If the movement of the person has depth (Yes at step S302), the data expansion unit 350a calculates the depth reduction rate α based on equation (1) (step S303), and proceeds to step S305.

[0145] On the other hand, if there is no depth in the movement of the person (step S302, No), the data expansion unit 350a calculates the depth reduction rate α based on equation (2) (step S304), and proceeds to step S305.

[0146] The data extension unit 350a extracts candidate points from the ground area of ​​the entire image frame (step S305). The data extension unit 350a identifies an arrangement point where the virtual person is to be arranged from the candidate points (step S306).

[0147] The data extension unit 350a calculates the height of the virtual person at the placement point based on equation (3) (step S307). The data extension unit 350a generates skeleton data of the virtual person at the placement point (step S308).

[0148] Next, the effects of the information processing device 300 according to the second embodiment will be described. The information processing device 300 calculates the depth reduction rate of the size of a person due to movement from each image frame included in the video data, and generates skeletal data of the virtual person when the virtual person is placed at the placement point based on the placement point on the ground included in the image frame and the depth reduction rate. This makes it possible to add information about the size of the person even when the amount of information about the size of the person is small in the video data. Adding information about the size of the person can prevent a decrease in the accuracy of parameter estimation.

[0149] The information processing device 300 identifies an area corresponding to the ground from the area of ​​the image frame, and identifies a placement point from the intersection of straight line components of the area corresponding to the ground, thereby making it possible to place the virtual person at an appropriate point in the image frame.

[0150] When the movement of a person shown in the video data has depth, the information processing device 300 calculates the depth reduction rate α based on formula (1). This makes it possible to calculate the depth reduction rate α using the skeletal data of the person at each time.

[0151] When the movement of a person shown in the video data does not have depth, the information processing device 300 calculates the depth reduction rate α based on equation (2). This makes it possible to calculate the depth reduction rate α using the person parts even when the movement of a person does not have depth.

[0152] Next, a description will be given of an example of a hardware configuration of a computer that realizes the same functions as the information processing device 300 described in the above-described embodiment 2. Fig. 24 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment 2.

[0153] 24, the computer 400 has a CPU 401 that executes various types of arithmetic processing, an input device 402 that accepts data input from a user, and a display 403. The computer 400 also has a communication device 404 that exchanges data with the camera 10, external devices, etc. via a wired or wireless network, and an interface device 405. The computer 400 also has a RAM 406 that temporarily stores various types of information, and a hard disk drive 407. The devices 401 to 407 are connected to a bus 408.

[0154] The hard disk drive 407 stores an acquisition program 407a, a detection program 407b, an attribute determination program 407c, a skeleton estimation program 407d, a parameter optimization program 407e, an evaluation program 407f, and a feature calculation program 407g. The hard disk drive 407 also stores a data extension program 407h. The CPU 401 reads out each of the programs 407a to 407h and expands them in the RAM 406.

[0155] The acquisition program 407a functions as an acquisition process 406a. The detection program 407b functions as a detection process 406b. The attribute determination program 407c functions as an attribute determination process 406c. The skeleton estimation program 407d functions as a skeleton estimation process 406d. The parameter optimization program 407e functions as a parameter optimization process 406e. The evaluation program 407f functions as an evaluation process 406f. The feature calculation program 407g functions as a feature calculation process 406g. The data extension program 407h functions as a data extension process 406h.

[0156] The processing of the acquisition process 406a corresponds to the processing of the acquisition unit 151. The processing of the detection process 406b corresponds to the processing of the detection unit 152. The processing of the attribute determination process 406c corresponds to the processing of the attribute determination unit 153. The processing of the skeleton estimation process 406d corresponds to the processing of the skeleton estimation unit 154. The processing of the parameter optimization process 406e corresponds to the processing of the parameter optimization unit 155. The processing of the evaluation process 406f corresponds to the processing of the evaluation unit 156. The processing of the feature calculation process 406g corresponds to the processing of the feature calculation unit 157. The processing of the data extension process 406h corresponds to the processing of the data extension unit 350a.

[0157] It should be noted that each of the programs 407a to 407h does not necessarily have to be stored in the hard disk drive 407 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 400. Then, the computer 400 may read and execute each of the programs 407a to 407h.

[0158] The following supplementary notes are further disclosed regarding the embodiments including the above examples.

[0159] (Note 1) The camera acquires the image data, generating skeletal information of a person included in the image frame and attribute information of the person included in the image frame by analyzing the image frames constituting the acquired image data; setting a first feature amount related to the height of the person for the person in the image frame based on the generated attribute information of the person; setting a second feature amount related to a height of the person for the person in the image frame based on the generated skeleton information; identifying a parameter of the camera based on a difference between the first feature amount and the second feature amount; A specific program that causes a computer to execute a process.

[0160] (Appendix 2) The specific program described in Appendix 1, characterized in that the attribute information is statistical information including an average height and variance according to the attributes of the person, and the process of setting the first feature quantity assigns a random height to the person based on the statistical information, and sets the height of the person in the image frame as the first feature quantity based on the height assigned to the person.

[0161] (Appendix 3) The identification program described in Appendix 1 is characterized in that the process of identifying the camera parameters sets the parameters so that the difference between the first feature and the second feature in the image frame is small.

[0162] (Supplementary Note 4) The process of setting the first feature value includes setting a plurality of first feature values ​​by randomly resetting a height of the person, and the process of identifying the parameters includes generating a plurality of parameters using the plurality of first feature values; The specific program according to claim 2, further causing a computer to execute a process of calculating a feature amount of the person based on any one of the plurality of parameters.

[0163] (Appendix 5) The specific program described in Appendix 4 is characterized in that the parameter calculation process uses multiple parameters to perform a process of calculating the difference for each of multiple image frames, and selects one parameter from the multiple parameters based on the variance of the differences.

[0164] (Supplementary Note 6) Calculating a reduction rate of the size of the person due to movement based on skeletal information and position of the person included in a first image frame and skeletal information and position of the person included in a second image frame among the plurality of image frames; Based on a placement point on the ground different from the position of the person included in the image frame and the reduction ratio, skeletal information of the person when the person is placed at the placement point is generated. The specific program according to claim 1, further comprising causing a computer to execute processing.

[0165] (Appendix 7) The identification program described in Appendix 6, characterized in that the computer further executes a process of identifying an area corresponding to the ground from the area of ​​the image frame, and identifying the placement point based on the intersection of straight line components of the area corresponding to the ground.

[0166] (Appendix 8) The specific program described in Appendix 6 is characterized in that the process of calculating the reduction ratio calculates a first change amount between the coordinates of the part of the person included in the first image frame that touches the ground and the coordinates of the part of the person included in the second image frame that touches the ground, calculates a second change amount between the height of the person included in the first image frame and the height of the person included in the second image frame, and calculates the reduction ratio based on the first change amount and the second change amount.

[0167] (Appendix 9) The specific program described in Appendix 6 is characterized in that the process of calculating the reduction ratio calculates a third change amount between the coordinates of the part of the person's right foot area included in the first image frame that contacts the ground and the coordinates of the part of the person's left foot area included in the first image frame that contacts the ground, calculates a fourth change amount between the area of ​​the right foot area and the area of ​​the left foot area, and calculates the reduction ratio based on the third change amount and the fourth change amount.

[0168] (Appendix 10) The camera acquires the image data, generating skeletal information of a person included in the image frame and attribute information of the person included in the image frame by analyzing the image frames constituting the acquired image data; setting a first feature amount related to the height of the person for the person in the image frame based on the generated attribute information of the person; setting a second feature amount related to a height of the person for the person in the image frame based on the generated skeleton information; identifying a parameter of the camera based on a difference between the first feature amount and the second feature amount; A method for identifying a target object, characterized in that the processing is carried out by a computer.

[0169] (Appendix 11) The identification method described in Appendix 10, characterized in that the attribute information is statistical information including an average height and variance according to the attributes of the person, and the process of setting the first feature quantity includes assigning a random height to the person based on the statistical information, and setting the height of the person in the image frame as the first feature quantity based on the height assigned to the person.

[0170] (Appendix 12) The identification method described in Appendix 10, characterized in that the process of identifying the camera parameters sets the parameters so that the difference between the first feature amount and the second feature amount in the image frame becomes small.

[0171] (Supplementary Note 13) The process of setting the first feature value includes setting a plurality of first feature values ​​by randomly resetting a height of the person, and the process of identifying the parameters includes generating a plurality of parameters using the plurality of first feature values; The identification method according to claim 11, further comprising causing a computer to execute a process of calculating a feature amount of the person based on any one of the plurality of parameters.

[0172] (Appendix 14) The identification method described in Appendix 13, characterized in that the process of calculating the parameters uses multiple parameters to perform the process of calculating the difference for multiple image frames, and selects one parameter from the multiple parameters based on the variance of the differences.

[0173] (Supplementary Note 15) Calculating a reduction rate of the size of the person due to movement based on skeletal information and position of the person included in a first image frame and skeletal information and position of the person included in a second image frame among the plurality of image frames; Based on a placement point on the ground different from the position of the person included in the image frame and the reduction ratio, skeletal information of the person when the person is placed at the placement point is generated. 11. The identification method according to claim 10, further comprising causing a computer to execute the processing.

[0174] (Appendix 16) The identification method described in Appendix 15, characterized in that the computer further executes a process of identifying an area corresponding to the ground from the area of ​​the image frame, and identifying the placement point based on the intersection of straight line components of the area corresponding to the ground.

[0175] (Appendix 17) The identification method described in Appendix 15, characterized in that the process of calculating the reduction ratio calculates a first change amount between the coordinates of the part of the person in the first image frame that touches the ground and the coordinates of the part of the person in the second image frame that touches the ground, calculates a second change amount between the height of the person in the first image frame and the height of the person in the second image frame, and calculates the reduction ratio based on the first change amount and the second change amount.

[0176] (Appendix 18) The process of calculating the reduction ratio is characterized by calculating a third change amount between the coordinates of the part where the person's right foot area included in the first image frame contacts the ground and the coordinates of the part where the person's left foot area included in the first image frame contacts the ground, calculating a fourth change amount between the area of ​​the right foot area and the area of ​​the left foot area, and calculating the reduction ratio based on the third change amount and the fourth change amount.

[0177] (Appendix 19) The camera acquires the image data, generating skeletal information of a person included in the image frame and attribute information of the person included in the image frame by analyzing the image frames constituting the acquired image data; setting a first feature amount related to the height of the person for the person in the image frame based on the generated attribute information of the person; setting a second feature amount related to a height of the person for the person in the image frame based on the generated skeleton information; identifying a parameter of the camera based on a difference between the first feature amount and the second feature amount; An information processing device comprising a control unit that executes processing.

[0178] (Appendix 20) The information processing device described in Appendix 19 is characterized in that the attribute information is statistical information including an average height and variance according to the attributes of the person, and the process of setting the first feature value assigns a random height to the person based on the statistical information, and sets the height of the person in the image frame as the first feature value based on the height assigned to the person.

[0179] (Appendix 21) The information processing device described in Appendix 19, characterized in that the process of identifying the camera parameters sets the parameters so that the difference between the first feature and the second feature in the image frame is small.

[0180] (Supplementary Note 22) The process of setting the first feature value includes setting a plurality of first feature values ​​by randomly resetting a height of the person, and the process of identifying the parameters includes generating a plurality of parameters using the plurality of first feature values; The information processing device according to claim 20, further comprising causing a computer to execute a process of calculating a feature amount of the person based on any one of the plurality of parameters.

[0181] (Appendix 23) The information processing device described in Appendix 22 is characterized in that the parameter calculation process uses multiple parameters to perform the process of calculating the difference for each of multiple image frames, and selects one parameter from the multiple parameters based on the variance of the differences.

[0182] (Supplementary Note 24) Calculating a reduction rate of the size of the person due to movement based on skeletal information and position of the person included in a first image frame and skeletal information and position of the person included in a second image frame among the plurality of image frames; Based on a placement point on the ground different from the position of the person included in the image frame and the reduction ratio, skeletal information of the person when the person is placed at the placement point is generated. 20. The information processing device according to claim 19, wherein the control unit further executes processing.

[0183] (Appendix 25) The information processing device described in Appendix 24, characterized in that the computer further executes a process of identifying an area corresponding to the ground from the area of ​​the image frame, and identifying the placement point based on the intersection of straight line components of the area corresponding to the ground.

[0184] (Appendix 26) The information processing device described in Appendix 24 is characterized in that the process of calculating the reduction ratio calculates a first change amount between the coordinates of the part of the person included in the first image frame that touches the ground and the coordinates of the part of the person included in the second image frame that touches the ground, calculates a second change amount between the height of the person included in the first image frame and the height of the person included in the second image frame, and calculates the reduction ratio based on the first change amount and the second change amount.

[0185] (Appendix 27) The information processing device described in Appendix 24 is characterized in that the process of calculating the reduction ratio calculates a third change amount between the coordinates of the part where the person's right foot area included in the first image frame touches the ground and the coordinates of the part where the person's left foot area included in the first image frame touches the ground, calculates a fourth change amount between the area of ​​the right foot area and the area of ​​the left foot area, and calculates the reduction ratio based on the third change amount and the fourth change amount. [Explanation of symbols]

[0186] 100 Information processing device 110 Communications Department 120 Input section 130 Display section 140 Storage section 141 Video Buffer 142 Learning Model Table 143 Attribute Table 144 Evaluation Table 150 control section 151 Acquisition Department 152 Detection unit 153 Attribute determination section 154 Skeleton Estimation Section 155 Parameter Optimization Department 156 Evaluation Department 157 Feature Calculation Unit 300 Information processing device 350 control section 350a Data Extension

Claims

1. The camera captures the image data, generating skeletal information of the person included in the image frame and statistical information including an average height and a variance according to attributes of the person included in the image frame by analyzing the image frames constituting the acquired image data; assigning a random height to the person based on the generated statistical information, and setting the height of the person in the image frame as a first feature amount based on the height assigned to the person; setting a second feature amount related to a height of the person for the person in the image frame based on the generated skeleton information; identifying a parameter of the camera based on a difference between the first feature amount and the second feature amount; A specific program that causes a computer to execute a process.

2. The identification program according to claim 1, characterized in that the process of identifying the camera parameters sets the parameters so that the difference between the first feature amount and the second feature amount in the image frame is small.

3. the process of setting the first feature amount includes setting a plurality of first feature amounts by randomly resetting a height of the person; the process of identifying the parameters includes generating a plurality of parameters using the plurality of first feature amounts; 2. The identification program according to claim 1, further causing a computer to execute a process of calculating a feature amount of the person based on any one of the plurality of parameters.

4. The specific program according to claim 3, characterized in that the parameter calculation process uses multiple parameters to perform a process of calculating the difference for each of multiple image frames, and selects one parameter from the multiple parameters based on the variance of the differences.

5. The camera captures the image data, generating skeletal information of the person included in the image frame and statistical information including an average height and a variance according to attributes of the person included in the image frame by analyzing the image frames constituting the acquired image data; assigning a random height to the person based on the generated statistical information, and setting the height of the person in the image frame as a first feature amount based on the height assigned to the person; setting a second feature amount related to a height of the person for the person in the image frame based on the generated skeleton information; identifying a parameter of the camera based on a difference between the first feature amount and the second feature amount; A method for identifying a target object, characterized in that the processing is carried out by a computer.

6. The camera captures the image data, generating skeletal information of the person included in the image frame and statistical information including an average height and a variance according to attributes of the person included in the image frame by analyzing the image frames constituting the acquired image data; assigning a random height to the person based on the generated statistical information, and setting the height of the person in the image frame as a first feature amount based on the height assigned to the person; setting a second feature amount related to a height of the person for the person in the image frame based on the generated skeleton information; identifying a parameter of the camera based on a difference between the first feature amount and the second feature amount; An information processing device comprising a control unit that executes processing.

Images