Update program, update method and information processing device

Abstract

To set camera photographing information for accurately calculating a height of a person.SOLUTION: An information processing apparatus sets a range of a height of a person included in an image frame captured by a camera based on positional information where the camera is installed. The information processing apparatus specifies the height of the person included in the image frame based on camera photographing information. The information processing apparatus updates the photographing information based on a height of a person not included in the range if the specified height of the person is not included in the range.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an update program and the like. [Background technology]

[0002] Video footage captured by cameras can be used in various investigations and can provide important information. For example, by analyzing camera images, it is possible to estimate not only a specific person's clothing and hairstyle, but also their height and other characteristics.

[0003] To estimate a person's height from a camera image, the camera's position and lens parameters are used, but it is unlikely that the parameters of an installed camera are recorded. In the following explanation, camera-related parameters will be referred to as "camera parameters" where appropriate.

[0004] There is a conventional technique for estimating camera parameters that uses a line segment of known length (body height). In this conventional technique, when the length of the line segment is unknown, the line segment of unknown length is included in the camera's shooting range, and the length of the line segment is estimated as one of the parameters at the same time as the camera parameters.

[0005] FIG. 15 is a diagram illustrating a conventional technique. In the conventional technique, when the height of person 1a is known, the known height remains constant even when person 1a moves within the shooting range, and camera parameters are estimated by utilizing this fact. On the other hand, when the height of person 1a is not known, the unknown height remains constant in video in which person 1a moves comprehensively within the shooting range, and the height of person 1a is estimated as one of the parameters together with the camera parameters. When the range of movement of person 1a is insufficient, video in which another person 1b, whose height is unknown, moves is also used, and the heights of people 1a and 1b are estimated as camera parameters together with the camera parameters. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Gaku Nakano “Camera Calibration Using Parallel Line Segments”Central Research Labs, NEC Corporation Summary of the Invention [Problem to be solved by the invention]

[0007] However, the above-mentioned conventional technology has a problem in that it is not possible to estimate camera parameters and therefore to calculate the height of a person.

[0008] 15, if the height of person 1a is unknown, video of person 1a moving comprehensively within the shooting range is used, but it is difficult to acquire video of person 1a moving comprehensively within the shooting range. Also, while it is possible to use video of multiple people moving, the number of camera parameters to be estimated increases, and there are cases where the estimation does not converge to appropriate camera parameters.

[0009] In one aspect, the present invention aims to provide an updating program, an updating method, and an information processing device that are capable of calculating a feature amount of a person. [Means for solving the problem]

[0010] In the first proposal, the computer executes the following process: The computer sets a height range for people included in an image frame captured by the camera based on information about the location where the camera is installed. The computer identifies the height of people included in the image frame based on the camera's shooting information. If the identified height of a person does not fall within the range, the computer updates the shooting information based on the height of people not included in the range. [Effects of the Invention]

[0011] It is possible to set camera shooting information for calculating a person's height with high accuracy. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of a system according to the present embodiment. [Figure 2] FIG. 2 is a diagram showing an example of video data. [Figure 3] FIG. 3 is a diagram (1) for explaining the processing of the information processing device according to the present embodiment. [Figure 4] FIG. 4 is a diagram (2) for explaining the processing of the information processing device according to the present embodiment. [Figure 5] FIG. 5 is a diagram (3) for explaining the processing of the information processing device according to the present embodiment. [Figure 6] FIG. 6 is a functional block diagram showing the configuration of an information processing device according to this embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of the data structure of the learning model table. [Figure 8] FIG. 8 is a diagram illustrating an example of the data structure of the attribute table. [Figure 9] FIG. 9 is a diagram showing an example of a person detection result. [Figure 10] FIG. 10 is a diagram showing an example of a skeleton estimation result. [Figure 11] FIG. 11 is a flowchart (1) showing the processing procedure of the information processing device according to the present embodiment. [Figure 12] FIG. 12 is a flowchart (2) showing the processing procedure of the information processing device according to the present embodiment. [Figure 13] FIG. 13 is a diagram showing the results of a comparison of the estimation accuracy between the prior art and the present invention. [Figure 14] FIG. 14 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the information processing apparatus of the embodiment. [Figure 15] FIG. 15 is a diagram for explaining the prior art. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment of the updating program, updating method, and information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings, although the present invention is not limited to these embodiments. [Example]

[0014] Fig. 1 is a diagram showing an example of a system according to this embodiment. As shown in Fig. 1, this system includes a camera 10 and an information processing device 100. The camera 10 and the information processing device 100 are connected to each other via a network 30. Although the example shown in Fig. 1 shows only the camera 10, the system according to this embodiment may include other cameras.

[0015] The camera 10 captures video within its capture range and transmits the captured video data to the information processing device 100. In the following description, the video data transmitted by the camera 10 to the information processing device 100 will be referred to as video data. In this embodiment, the description will be given using video data of a person.

[0016] Fig. 2 is a diagram showing an example of video data. As shown in Fig. 2, video data 20 includes a plurality of image frames in chronological order. Each image frame is assigned a frame number in ascending chronological order. One image frame is a still image captured by camera 10 at a certain timing.

[0017] The camera 10 transmits position data of the location where the camera 10 is installed to the information processing device 100. For example, the camera 10 measures latitude and longitude using a GPS (Global Positioning System) function, and transmits the measured latitude and longitude as position data to the information processing device 100. An administrator of the camera 10 may set the position data in the camera 10 in advance.

[0018] The information processing device 100 acquires position data and video data 20 from the camera 10, assigns a tentative average height determined from the position data to each person included in an image frame of the video data 20, and estimates camera parameters. The information processing device 100 extracts people whose heights determined from the camera parameter estimation results do not fall within a predetermined range based on the average height. The information processing device 100 repeatedly executes a process of adjusting the camera parameters by recalculating the camera parameters using the heights of the extracted people, and determines the camera parameters.

[0019] The information processing device 100 calculates the height of a person included in the video data 20 based on the determined camera parameters. In this embodiment, a case where the height of a person is calculated will be described, but other feature amounts of the person may also be calculated. For example, the information processing device 100 may calculate the length of a predetermined part of the person. The camera parameters are an example of shooting information.

[0020] 3, 4, and 5 are diagrams for explaining the processing of the information processing device according to this embodiment. Fig. 3 will be explained. In Fig. 3, the explanation will be given using image frame F21 out of the multiple image frames included in video data 20.

[0021] The coordinate system of the image frame F21 is the image coordinate system (x, y). The coordinate system in which the person actually exists is the world coordinate system (X, Y, Z). In the following explanation, the person in the image coordinate system shown in the image frame F21 will be referred to as person 21-1a, and the person in the world coordinate system will be referred to as person 21-2a.

[0022] The camera parameters of the camera 10 include the height c of the camera 10, the angle θ of the camera 10, and the focal length f of the camera 10. The camera parameters of the camera 10 are unknown, and the information processing device 100 sets predetermined initial values ​​for the camera parameters in advance. Note that the camera parameters are not limited to these, and other parameters that can define a geometric relationship may also be used. For example, other parameters include the intersection of the optical axis and the image (optical axis center coordinates), the rotation angle of the camera, etc.

[0023] The information processing device 100 identifies skeletal data of the person 21-1a by analyzing the image frame F21. For example, the information processing device 100 identifies skeletal data of the person 21-1a by inputting the image frame F21 into a machine-learned learning model. The learning model used by the information processing device 100 will be described later.

[0024] The skeletal data includes information about a plurality of joints of a person, and each joint is associated with a coordinate on an image frame. For example, the skeletal data corresponding to the image frame F21 includes the coordinates (x h1 ,y h1 ), foot coordinates (x f1 ,y f1 ) etc.

[0025] The information processing device 100 has an attribute table in which the territorial extent of each country is associated with the average height of people living in that country. Based on the location data received from the camera 10 and the attribute table, the information processing device 100 identifies the average height of people in the country where the camera 10 is installed (the average height of people captured in the image frame F21).

[0026] The information processing device 100 calculates the coordinates of the foot in the image coordinate system (x f1 ,y f1 ) onto the coordinates of the world coordinate system. For example, the information processing device 100 projects the coordinates (x f1 ,y f1 ) and the intersection point (X f1 ,Y f1 ,Z f1 ) is calculated as the coordinates of the feet of the person 21-2a in the world coordinate system.

[0027] The information processing device 100 assigns the height (average height) L, which is determined based on the position data and attribute information, to the person 21-2a in the world coordinate axis system. The information processing device 100 calculates the coordinates of the head of the person 21-2a (X h1 ,Y h1 ,Z h1 ) is calculated.

[0028] The information processing device 100 calculates the coordinates of the head in the world coordinate system (X h1 ,Y h1 ,Z h1 ) into the coordinates of the image coordinate system. For example, the information processing device 100 projects the coordinates (X h1 ,Y h1 ,Z h1 ) and the coordinates (x' f1 ,y´ f1 ) is calculated and used as the coordinates of the head of the person 21-1a in the image coordinate system.

[0029] The information processing device 100 calculates the coordinates (x f1 ,y f1 ) to the coordinate (x' f1 ,y´ f1 ) is set as the “first feature amount.” The first feature amount corresponds to the height of the person 21-1a estimated based on the assigned height L and the camera parameters of the camera 10.

[0030] The information processing device 100 calculates the coordinates (x f1 ,y f1 ) to the coordinates of the head (x h1 ,y h1 ) is set as a "second feature amount." The second feature amount corresponds to the height of the person 21-1a estimated based on the skeletal data.

[0031] 4 will be explained. The information processing device assigns heights L to the other persons 22-1a, 23-1a, 24-1a, and 25-1a included in the image frame F21 in the same manner as for the person 21-1a, and sets the first feature amount and the second feature amount for each of the persons 22-1a to 25-1a. Initially, the heights L assigned to the persons 21-1a to 25-1a are the same height (average height).

[0032] The coordinates of the feet in the image coordinate system obtained from the skeletal data of person 22-1a are (x f2 ,y f2 ) and the coordinates of the head are (x h2 ,y h2 ) The coordinates of the head obtained using the camera parameters and height L are (x' f2 ,y´ f2 The first feature of the person 22-1a is the coordinate (x f2 ,y f2 ) to the coordinate (x' f2 ,y´ f2 The second feature of the person 22-1a is the distance to the coordinate (x f2 ,y f2 ) to the coordinates of the head (x h2 ,y h2 ) is the distance to

[0033] The coordinates of the feet in the image coordinate system obtained from the skeletal data of person 23-1a are (x f3 ,y f3 ) and the coordinates of the head are (x h3 ,y h3 ) The coordinates of the head obtained using the camera parameters and height L are (x' f3 ,y´ f3 The first feature of the person 23-1a is the coordinate (x f3 ,y f3 ) to the coordinate (x' f3 ,y´ f3 The second feature of the person 23-1a is the distance to the coordinate (x f3 ,y f3 ) to the coordinates of the head (x h3 ,y h3 ) is the distance to

[0034] The coordinates of the feet in the image coordinate system obtained from the skeletal data of person 24-1a are (x f4 ,y f4 ) and the coordinates of the head are (x h4 ,y h4 ) The coordinates of the head obtained using the camera parameters and height L are (x' f4 ,y´ f4 The first feature of the person 24-1a is the coordinate (x f4 ,y f4 ) to the coordinate (x' f4 ,y´ f4 The second feature of the person 24-1a is the distance to the coordinate (x f4 ,y f4 ) to the coordinates of the head (x h4 ,y h4 ) is the distance to

[0035] The coordinates of the feet in the image coordinate system obtained from the skeletal data of person 25-1a are (x f5 ,y f5 ) and the coordinates of the head are (x h5 ,y h5 ) The coordinates of the head obtained using the camera parameters and height L are (x' f5 ,y´ f5 The first feature of the person 25-1a is the coordinate (x f5 ,y f5 ) to the coordinate (x' f5 ,y´ f5 The second feature of person 25-1a is the distance to the coordinate (x f5 ,y f5 ) to the coordinates of the head (x h5 ,y h5 ) is the distance to

[0036] With the heights L of the persons 21-1a to 25-1a fixed, the information processing device optimizes the camera parameters of the camera 10 so as to reduce the difference between the first feature amount and the second feature amount for each of the persons 21-1a to 25-1a.

[0037] We now move on to the description of FIG. 5. The information processing device calculates the heights of the persons 21-1a to 25-1a based on the camera parameters optimized by the above process. The information processing device identifies persons among the persons 21-1a to 25-1a whose calculated heights do not fall within a predetermined range. For example, the predetermined range is set to "initial value (average height) ±4". When optimizing the camera parameters for the second time or later, the information processing device uses information on persons whose calculated heights do not fall within the predetermined range.

[0038] In the example shown in FIG. 5, the heights of persons 21-1a to 25-1a calculated based on the optimized camera parameters are "173," "169," "167," "177," and "170," respectively. If the initial value is 172, the predetermined range is "168 to 176." Then, the information processing device identifies person 23-1a with a height of "167" and person 24-1a with a height of "177" as people whose heights do not fall within the predetermined range.

[0039] If the height of the identified person is equal to or greater than the initial value, the information processing device adds a predetermined value to the person's height and sets the added height as the second initial value. If the height of the identified person is less than the initial value, the information processing device subtracts a predetermined value from the person's height and sets the subtracted height as the second initial value. The predetermined value is 1.

[0040] For example, the height of person 23-1a is "167", which is less than the initial value. Therefore, the information processing device sets "166" as the second initial value for the height L of person 23-1a. The height of person 24-1a is "177", which is greater than or equal to the initial value. Therefore, the information processing device sets "178" as the second initial value for the height L of person 24-1a.

[0041] The information processing device optimizes the camera parameters of the camera 10 so as to reduce the difference between the first feature amount and the second feature amount for each of the persons 23-1a and 24-1a, while fixing the heights L of the persons 23-1a and 24-1a. The initial values ​​of the camera parameters for the second estimation are set to the results of the first estimation of the camera parameters.

[0042] As described above, in the first processing, the information processing device assigns a tentative average height to each person and estimates camera parameters. The information processing device 100 extracts people whose heights, identified from the camera parameter estimation results, do not fall within a predetermined range based on the average height. The information processing device 100 determines the camera parameters by recalculating the camera parameters from the second time onward using the heights of the extracted people. This allows camera parameters for accurately calculating the heights of people to be set in the camera 10.

[0043] The information processing device identifies people who are not within a predetermined range based on the average height, resets the height of such people, recalculates the camera parameters, and can converge the camera parameters even if there are people who deviate from the statistical average value.

[0044] The information processing device can estimate the height of each person even if the heights of all the people included in the image frame are unknown.

[0045] Next, an example of the configuration of the information processing device 100 according to this embodiment will be described. Fig. 6 is a functional block diagram showing the configuration of the information processing device according to this embodiment. As shown in Fig. 6, this information processing device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0046] The communication unit 110 is connected to the camera 10 via the network 30 and receives video data. For example, the communication unit 110 is realized by a network interface card (NIC) or the like. The communication unit 110 may also be connected to other external devices or the like via the network 30.

[0047] The input unit 120 is an input device that inputs various types of information to the information processing device 100. The input unit 120 corresponds to a keyboard, a mouse, a touch panel, or the like.

[0048] The display unit 130 is a display device that displays information output from the control unit 150. The display unit 130 corresponds to a liquid crystal display, an organic EL (Electro Luminescence) display, a touch panel, or the like.

[0049] The storage unit 140 includes a video buffer 141, a learning model table 142, and an attribute table 143. The storage unit 140 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0050] The video buffer 141 is a buffer that stores the video data 20 transmitted from the camera 10. The video data 20 includes a plurality of image frames in chronological order. Each image frame is assigned a frame number in ascending chronological order.

[0051] The learning model table 142 is a table that includes various learning models. Fig. 7 is a diagram showing an example of the data structure of the learning model table. As shown in Fig. 7, the learning model table 142 includes a person detection model 142a and a skeleton estimation model 142b.

[0052] The person detection model 142a is a machine learning model that, when a time-series image frame included in video data is input, detects a person area included in the image frame and tracks the person area. A person ID for identifying the person is assigned to each person detected from the image frame. The person detection model 142a is realized by a machine learning model such as DeepSort.

[0053] The skeleton estimation model 142b is a machine learning model that outputs skeleton data of a person when a region of the person in an image frame (a whole-body image) is input. The skeleton estimation model 142b can be realized by a machine learning model such as OpenPose.

[0054] The person detection model 142a and the skeleton estimation model 142b correspond to a neural network (NN), etc. It is assumed that machine learning has been performed in advance on the person detection model 142a and the skeleton estimation model 142b based on a learning data set.

[0055] Attribute table 143 is a table that defines the relationship between the territorial extent of each country and the average height of people living in the corresponding country. FIG. 8 is a diagram showing an example of the data structure of an attribute table. As shown in FIG. 8, attribute table 143 associates territorial extent with average height. The territorial extent indicates the territorial extent of the corresponding country. The territorial extent is specified by latitude and longitude. The average height indicates the average height of people living in the corresponding territorial extent. For example, it is shown that the average height corresponding to the territorial extent "territorial extent of Japan" is "172 cm." This means that the average height of people photographed by cameras installed in Japanese territory is 172 cm.

[0056] Returning to the description of Fig. 6, the control unit 150 includes an acquisition unit 151, a detection unit 152, a setting unit 153, a skeleton estimation unit 154, a parameter optimization unit 155, and an estimation unit 156. The control unit 150 is realized by, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The control unit 150 may also be implemented by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0057] Acquisition unit 151 acquires video data 20 from camera 10 via communication unit 110. Acquisition unit 151 registers the acquired video data 20 in video buffer 141. Acquisition unit 151 continuously acquires video data 20 and registers it in video buffer 141 while camera 10 is capturing images.

[0058] The detection unit 152 acquires image frames in chronological order from the video buffer 141 and inputs the image frames to the person detection model 142a to detect the areas of people included in the image frames. FIG. 9 is a diagram showing an example of the person detection result. In the example shown in FIG. 9, the areas of people 21-1a, 22-1a, 23-1a, 24-1a, and 25-1a are detected from the image frame F21. Person IDs are assigned to the areas of people 21-1a, 22-1a, 23-1a, 24-1a, and 25-1a.

[0059] The detection unit 152 outputs information on the detection result to the skeleton estimation unit 154 and the parameter optimization unit 155. The information on the detection result includes information on the person's area (image within the area), the person ID assigned to each area, the frame number of the image frame, etc.

[0060] The detection unit 152 repeatedly executes the above process for each image frame of the video data 20 stored in the video buffer 141 in chronological order.

[0061] The setting unit 153 identifies the average height of people within the territory of the country where the camera 10 is installed based on the position data acquired from the camera 10, and sets a predetermined range based on the identified average height. An example of the processing of the setting unit 153 will be described below.

[0062] The setting unit 153 compares the position data acquired from the camera 10 with the attribute table 143, identifies the range of territory that includes the latitude and longitude of the position data, and identifies the average height corresponding to the identified range of territory. The setting unit 153 sets the predetermined range to "average height ±4". For example, if the latitude and longitude of the position data are included in the range of Japanese territory, the setting unit 153 identifies the average height as "172" and sets the predetermined range to "168 to 176".

[0063] Incidentally, the setting unit 153 may identify attribute data of a person photographed by the camera 10 based on the location data, and set a predetermined range based on the attribute data. For example, the setting unit 153 holds a race table that associates location data with the race of a person living in the location data, and identifies the person's race as attribute data based on the race table and the location data. The setting unit 153 identifies an average height corresponding to the identified race, and sets a predetermined range based on the identified average height. The average height corresponding to the race is assumed to be set in advance in the table.

[0064] The setting unit 153 outputs the data of the average height and the data within a predetermined range to the parameter optimization unit 155 .

[0065] The skeleton estimation unit 154 estimates the skeleton of a person by inputting information on the area of ​​the person (whole-body image) included in the information on the detection result into the skeleton estimation model 142b. FIG. 10 is a diagram showing an example of a skeleton estimation result. As shown in FIG. 10, the skeleton estimation result 40 includes a plurality of joints 40-1, 40-2, 40-3, 40-4, 40-5, 40-6, 40-7, 40-8, 40-9, 40-10, 40-11, 40-12, and 40-13. Although not shown, each of the joints 40-1 to 40-13 includes coordinates in the image coordinate system and a joint type. For example, the joint 40-1 corresponds to the head. The joint 40-12 (40-13) corresponds to the foot.

[0066] The skeleton estimation unit 154 estimates the skeleton for each person's area included in the information of the detection result. The skeleton estimation unit 154 generates skeleton data in which the person ID is associated with the estimation result of the person's skeleton, and outputs the data to the parameter optimization unit 155 and the estimation unit 156.

[0067] The skeleton estimation unit 154 repeatedly executes the above process every time it acquires information on the detection result from the detection unit 152.

[0068] The parameter optimization unit 155 is a processing unit that optimizes camera parameters by executing the following processing. The processing of the parameter optimization unit 155 will be explained with reference to FIG. 3. The parameter optimization unit 155 optimizes the coordinates (x h1 ,y h1 ), foot coordinates (x f1 ,y f1 ) to identify the

[0069] The parameter optimization unit 155 calculates the coordinates of the foot in the image coordinate system (x f1 ,y f1 ) onto the coordinates of the world coordinate system. For example, the parameter optimization unit 155 projects the coordinates (x f1 ,y f1 ) and the intersection point (X f1 ,Y f1 ,Z f1 ) is calculated as the coordinates of the feet of the person 21-2a in the world coordinate system.

[0070] The parameter optimization unit 155 assigns the height (average height) L acquired from the setting unit 153 to the person 21-2a in the world coordinate axis system. The parameter optimization unit 155 calculates the coordinates of the head (X h1 ,Y h1 ,Z h1 ) is calculated.

[0071] The parameter optimization unit 155 calculates the coordinates of the head in the world coordinate system (X h1 ,Y h1 ,Z h1 ) into the coordinates of the image coordinate system. For example, the parameter optimization unit 155 back-projects the coordinates (X h1 ,Y h1 ,Z h1 ) and the coordinates (x' f1 ,y´ f1) is calculated and used as the coordinates of the head of the person 21-1a in the image coordinate system.

[0072] The parameter optimization unit 155 sets a first feature amount and a second feature amount for the person 21-1a based on the results of the above processing. The parameter optimization unit 155 also assigns heights L to the other people 22-1a, 23-1a, 24-1a, and 25-1a included in the image frame F21 in the same manner as for the person 21-1a, and sets first feature amounts and second feature amounts for each of the people 22-1a to 25-1a. The heights L assigned to the people 21-1a to 25-1a at the beginning are the same height (average height).

[0073] The parameter optimization unit 155 optimizes the camera parameters of the camera 10 so as to reduce the difference between the first feature amount and the second feature amount for each of the persons 21-1a to 25-1a, with the height L of each of the persons 21-1a to 25-1a being fixed.

[0074] Next, the parameter optimization unit 155 calculates the heights of the persons 21-1a to 25-1a, respectively, based on the camera parameters optimized by the above process. The parameter optimization unit 155 identifies persons among the persons 21-1a to 25-1a whose calculated heights do not fall within a predetermined range. For example, the predetermined range is set to "initial value (average height) ±4". When optimizing the camera parameters for the second time or later, the parameter optimization unit 155 uses information on persons whose calculated heights do not fall within the predetermined range.

[0075] For example, the processing of the parameter optimization unit 155 will be described using Fig. 5. The parameter optimization unit 155 calculates the heights of the people 21-1a to 25-1a based on the optimized camera parameters as "173", "169", "167", "177", and "170", respectively. If the initial value is 172, the predetermined range is "168 to 176". In this case, the parameter optimization unit 155 identifies the person 23-1a with a height of "167" and the person 24-1a with a height of "177".

[0076] If the height of the identified person is equal to or greater than the initial value, the parameter optimization unit 155 adds a predetermined value to the person's height and sets the added height as the second initial value. If the height of the identified person is less than the initial value, the parameter optimization unit 155 subtracts a predetermined value from the person's height and sets the subtracted height as the second initial value. The predetermined value is set to 1.

[0077] Among the persons 21-1a to 25-1a, the persons whose calculated heights do not fall within a predetermined range will be referred to as persons 23-1a and 24-1a in the following description. With the heights L of the persons 23-1a and 24-1a fixed, the parameter optimization unit 155 optimizes the camera parameters of the camera 10 for each of the persons 23-1a and 24-1a so as to reduce the difference between the first feature amount and the second feature amount. The initial values ​​of the camera parameters for the second estimation are set to the results of the camera parameter estimation for the first estimation. The parameter optimization unit 155 outputs the results of the optimized camera parameters to the estimation unit 156.

[0078] If there is no person among the persons 21-1a to 25-1a whose calculated height is not within a predetermined range, the parameter optimization unit 155 skips the second processing and outputs the first calculation result of the camera parameters to the estimation unit 156.

[0079] The estimation unit 156 calculates the height of the person included in the image frame based on the camera parameters acquired from the parameter optimization unit 155. The estimation unit 156 identifies the coordinates of the person's head and feet in the image coordinate system based on the skeletal data. The estimation unit 156 projects the coordinates of the person's head and feet in the image coordinate system onto the coordinates of the person's head and feet in the world coordinate system based on the camera parameters. The estimation unit 156 calculates the distance from the coordinates of the feet to the coordinates of the head in the world coordinate system as the person's height. The estimation unit 156 may display information on the calculated feature amount (height) of the person on the display unit 130 or may transmit it to an external device.

[0080] The estimation unit 156 may calculate the length of a predetermined part of a person as a feature amount of the person based on the skeletal data and the camera parameters.

[0081] Next, an example of a processing procedure of the information processing device according to this embodiment will be described. Figures 11 and 12 are flowcharts showing the processing procedure of the information processing device according to this embodiment. First, Figure 11 will be described. The acquisition unit 151 of the information processing device 100 receives video data from the camera 10 and registers it in the video buffer 141 (step S101).

[0082] The detection unit 152 of the information processing device 100 detects the area of ​​each person from the image frame (step S102). The skeleton estimation unit 154 of the information processing device 100 generates skeleton data of each person (step S103).

[0083] The setting unit 153 of the information processing device 100 identifies the average height based on the position data acquired from the camera 10 and the attribute table 143 (step S104). The setting unit 153 sets the average height as the initial value of the height of each person (step S105).

[0084] The parameter optimization unit 155 of the information processing device 100 sets a first feature amount and a second feature amount for each person (step S106). The parameter optimization unit 155 optimizes the camera parameters for each person so that the difference between the first feature amount and the second feature amount is minimized (step S107).

[0085] The parameter optimization unit 155 uses the optimized camera parameters to calculate the height of each person (step S108), and the process proceeds to step S109 in FIG.

[0086] We now move on to the description of Fig. 12. The parameter optimization unit 155 identifies a person whose height is outside the predetermined range (step S109). If the parameter optimization unit 155 can identify a person whose height is outside the predetermined range (step S110, Yes), the process proceeds to step S111. On the other hand, if the parameter optimization unit 155 cannot identify a person whose height is outside the predetermined range (if the heights of all people are within the predetermined range) (step S110, No), the process proceeds to step S114.

[0087] The parameter optimization unit 155 sets the second initial value for the identified person (step S111). The parameter optimization unit 155 sets a first feature amount and a second feature amount for the identified person (step S112).

[0088] The parameter optimization unit 155 optimizes the camera parameters so as to minimize the difference between the identified first feature amount and the identified second feature amount (step S113). The estimation unit 156 of the information processing device 100 calculates the height of each person using the optimized camera parameters (step S114). The estimation unit 156 outputs the height of each person (step S115).

[0089] Next, the effects of the information processing device 100 according to this embodiment will be described. In the first processing, the information processing device 100 assigns a tentative average height to each person and estimates camera parameters. The information processing device 100 identifies people whose heights, identified from the camera parameter estimation results, do not fall within a predetermined range based on the average height, and determines the camera parameters by recalculating the camera parameters from the second time onwards using the heights of the identified people. This makes it possible to set camera parameters in the camera 10 that allow for accurate calculation of a person's height.

[0090] Fig. 13 is a diagram showing the results of a comparison of the estimation accuracy between the conventional technology and the present invention. The vertical axis of graph G1 in Fig. 13 represents the absolute mean error (hereinafter simply referred to as error) of the estimated height. Br1 represents the error when a person's height is estimated using a conventional technique for video 20A captured by the first camera. Br2 represents the error when a person's height is estimated using information processing device 100 for video 20A captured by the first camera. As shown in Fig. 13, comparing Br1 and Br2, the error is smaller when the person's height is estimated using information processing device 100.

[0091] Br3 indicates an error when the person's height is estimated using a conventional technique for the video 20B captured by the second camera. Br4 indicates an error when the person's height is estimated using the information processing device 100 for the video 20B captured by the second camera. As shown in Fig. 13, comparing Br3 and Br4, the error is smaller when the person's height is estimated using the information processing device 100.

[0092] The information processing device 100 identifies the height of a person by projecting the coordinates of the feet and head of the person included in the image frame onto the coordinates of the world coordinate system based on the camera parameters, thereby enabling the height to be calculated accurately from the image frame according to the camera parameters.

[0093] The information processing device 100 identifies the coordinates of the person's feet and head on the image frame based on the height of the person in the world coordinate system that is not included in the predetermined range, and updates the camera parameters based on the difference between the identified coordinates and the coordinates of the person's feet and head identified from the image frame. In this way, the camera parameters can be optimized using only the height of the person that is outside the predetermined range, thereby making it possible to correctly estimate the camera parameters.

[0094] Incidentally, the information processing device 100 according to the present embodiment identifies people whose heights, determined from the camera parameter estimation results, are outside a predetermined range based on the average height, and recalculates the camera parameters from the second time onward using the heights of the identified people, but this is not limiting. The information processing device 100 may optimize the camera parameters by weighting, among the heights of multiple people included in the image frame, heights that are outside the predetermined range more heavily than heights that are within the predetermined range.

[0095] For example, among the persons 21-1a to 25-1a described in FIG. 5, the persons whose calculated heights do not fall within a predetermined range are designated as persons 23-1a and 24-1a. The difference between the first feature amount and the second feature amount for persons 21-1a, 22-1a, 23-1a, and 25-1a is designated as a first difference. The difference between the first feature amount and the second feature amount for persons 23-1a and 24-1a is designated as a second difference. The parameter optimization unit 155 may optimize the camera parameters so that the value obtained by multiplying the first difference and the second difference by a weight α becomes small. This allows the camera parameters to be optimized with emphasis on statistically small differences.

[0096] Next, an example of the hardware configuration of a computer that realizes the same functions as the information processing device 100 described in the above embodiment will be described. Fig. 14 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing device of the embodiment.

[0097] 15, the computer 200 has a CPU 201 that executes various types of arithmetic processing, an input device 202 that accepts data input from a user, and a display 203. The computer 200 also has a communication device 204 that exchanges data with the camera 10, external devices, etc. via a wired or wireless network, and an interface device 205. The computer 200 also has a RAM 206 that temporarily stores various types of information, and a hard disk drive 207. The devices 201 to 207 are connected to a bus 208.

[0098] The hard disk drive 207 stores an acquisition program 207a, a detection program 207b, a setting program 207c, a skeleton estimation program 207d, a parameter optimization program 207e, and an estimation program 207f. The CPU 201 reads out each of the programs 207a to 207f and loads them into the RAM 206.

[0099] The acquisition program 207a functions as an acquisition process 206a. The detection program 207b functions as a detection process 206b. The setting program 207c functions as a setting process 206c. The skeleton estimation program 207d functions as a skeleton estimation process 206d. The parameter optimization program 207e functions as a parameter optimization process 206e. The estimation program 207f functions as an estimation process 206f.

[0100] The processing of the acquisition process 206a corresponds to the processing of the acquisition unit 151. The processing of the detection process 206b corresponds to the processing of the detection unit 152. The processing of the setting process 206c corresponds to the processing of the setting unit 153. The processing of the skeleton estimation process 206d corresponds to the processing of the skeleton estimation unit 154. The processing of the parameter optimization process 206e corresponds to the processing of the parameter optimization unit 155. The processing of the estimation process 206f corresponds to the processing of the estimation unit 156.

[0101] It should be noted that each of the programs 207a to 207f does not necessarily have to be stored in the hard disk drive 207 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207f.

[0102] The following supplementary notes are further disclosed regarding the embodiments including the above examples.

[0103] (Appendix 1) Based on the location information of the camera, a range of heights of people included in the image frame captured by the camera is set, Identifying the height of the person included in the image frame based on the photographing information of the camera; If the height of the identified person is not within the range, the photographing information is updated based on the height of the person not within the range. An update program that causes a computer to execute a process.

[0104] (Appendix 2) The update program described in Appendix 1 is characterized in that the process of identifying the height of the person is performed by projecting the coordinates of the person's feet and head contained in the image frame onto coordinates in a world coordinate system based on the shooting information, thereby identifying the height of the person.

[0105] (Appendix 3) The update program described in Appendix 1 is characterized in that the process of updating the shooting information identifies a first feature from the feet to the head of the person on the image frame based on the height of the person in a world coordinate system that is not included in the range, and updates the shooting information based on the difference between the first feature and a second feature from the feet to the head of the person identified from the image frame.

[0106] (Appendix 4) The update program described in Appendix 1 is characterized in that the process of updating the shooting information is performed by giving greater weight to heights of multiple people included in the image frame that are not included in the range than to heights that are included in the range, thereby updating the shooting information.

[0107] (Appendix 5) Based on the location information of the camera, a range of heights of people included in the image frame captured by the camera is set, Identifying the height of the person included in the image frame based on the photographing information of the camera; If the height of the identified person is not within the range, the photographing information is updated based on the height of the person not within the range. An updating method characterized in that the processing is executed by a computer.

[0108] (Appendix 6) The update method described in Appendix 5 is characterized in that the process of identifying the height of the person involves projecting the coordinates of the person's feet and head contained in the image frame onto coordinates in a world coordinate system based on the shooting information, thereby identifying the height of the person.

[0109] (Appendix 7) The updating method described in Appendix 5 is characterized in that the process of updating the shooting information identifies a first feature from the feet to the head of the person on the image frame based on the height of the person in a world coordinate system that is not included in the range, and updates the shooting information based on the difference between the first feature and a second feature from the feet to the head of the person identified from the image frame.

[0110] (Appendix 8) The updating method described in Appendix 5 is characterized in that the process of updating the shooting information is performed by giving a greater weight to heights of multiple people included in the image frame that are not included in the range than to heights that are included in the range, thereby updating the shooting information.

[0111] (Supplementary Note 9) A setting unit that sets a range of heights of people included in an image frame captured by the camera based on location information of the camera; a parameter optimization unit that specifies a height of a person included in the image frame based on the photographing information of the camera, and when the specified height of the person is not included in the range, updates the photographing information based on the height of the person not included in the range; An information processing device comprising:

[0112] (Appendix 10) The information processing device described in Appendix 9 is characterized in that the parameter optimization unit identifies the height of the person by projecting the coordinates of the person's feet and head included in the image frame onto coordinates of a world coordinate system based on the shooting information.

[0113] (Appendix 11) The information processing device described in Appendix 9 is characterized in that the parameter optimization unit identifies a first feature from the feet to the head of the person on the image frame based on the height of the person in a world coordinate system that is not included in the range, and updates the shooting information based on the difference between the first feature and a second feature from the feet to the head of the person identified from the image frame.

[0114] (Appendix 12) The information processing device described in Appendix 9 is characterized in that the parameter optimization unit updates the shooting information by giving a greater weight to heights of multiple people included in the image frame that are not included in the range than to heights that are included in the range. [Explanation of symbols]

[0115] 100 Information processing device 110 Communications Department 120 Input section 130 Display section 140 Storage section 141 Video Buffer 142 Learning Model Table 143 Attribute Table 150 control section 151 Acquisition Department 152 Detection unit 153 Setting section 154 Skeleton Estimation Section 155 Parameter Optimization Department 156 Estimation Department

Claims

1. Based on the location information of the camera, a range of heights of people included in an image frame captured by the camera is set; a distance from the feet to the head of the person included in the image frame is set as a first feature amount; a head of the person included in the image frame is estimated based on the photographing information of the camera, and a distance from the estimated head to the feet is set as a second feature amount; optimizing the photographing information so that a difference between the first feature amount and the second feature amount becomes small; Identifying the height of the person based on the optimized shooting information; If the height of the identified person is not within the range, the photographing information is updated based on the height of the person not within the range. An update program that causes a computer to execute a process.

2. 2. The update program according to claim 1, wherein the process of identifying the height of the person involves projecting the coordinates of the feet and head of the person included in the image frame onto coordinates in a world coordinate system based on the shooting information, thereby identifying the height of the person.

3. The update program according to claim 1 , wherein the process of setting the second feature quantity specifies the second feature quantity based on a height of a person in a world coordinate system that is not included in the range.

4. 2. The update program according to claim 1, wherein the processing for updating the photographing information is performed by assigning a higher weight to heights of a plurality of people included in the image frame that are not included in the range than to heights that are included in the range, thereby updating the photographing information.

5. Based on the location information of the camera, a range of heights of people included in an image frame captured by the camera is set; a distance from the feet to the head of the person included in the image frame is set as a first feature amount; a head of the person included in the image frame is estimated based on the photographing information of the camera, and a distance from the estimated head to the feet is set as a second feature amount; optimizing the photographing information so that a difference between the first feature amount and the second feature amount becomes small; Identifying the height of the person based on the optimized shooting information; If the height of the identified person is not within the range, the photographing information is updated based on the height of the person not within the range. An updating method characterized in that the processing is executed by a computer.

6. a setting unit that sets a height range of a person included in an image frame captured by the camera based on location information of the camera; a parameter optimization unit that sets a distance from the feet to the head of the person included in the image frame as a first feature amount, estimates the head of the person included in the image frame based on shooting information of the camera, sets the distance from the estimated head to the feet as a second feature amount, optimizes the shooting information so as to reduce a difference between the first feature amount and the second feature amount, specifies a height of the person based on the optimized shooting information, and, if the specified height of the person is not included in the range, updates the shooting information based on the height of the person that is not included in the range; An information processing device comprising:

Images