Estimation device, estimation method, and program

Abstract

This estimation device comprises a control unit that estimates an estimation target amount, which is the size of an estimation target animal, by using one or more frames in which a difference between the value of the depth of the estimation target animal and the value of the depth of a reference object having a known size is equal to or less than a threshold value among one or more frames constituting a processing target video having at least one frame in which the estimation target animal and the reference object are imaged.　

Description

Estimation device, estimation method, and program

[0001] The present invention relates to an estimation device, an estimation method, and a program.

[0002] To estimate the height of a person in a video, there is a technique for replacing the video with multiple still image frames, extracting person and object frames, and estimating the height from the person and object in the still images (see Non-Patent Documents 1 and 2).

[0003] Kaiming He, Georgia Gkioxari, Piotr Dollar, Ross Girshick, “Mask R-CNN”, The IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2961-2969; Toshiaki Takeda et al., “Basic Study on Estimating Height of a Person in Images Captured by a Single-Shot Camera”, 2022 IEICE Information and Systems Conference Proceedings 2, D-11-8

[0004] However, if there is no object near the person in the still image frame, the error in the height estimation result may be large. Furthermore, there may be errors, recognition errors, or recognition failure due to fluctuations in recognition caused by external factors such as lighting or wind during shooting. In such cases, there is a problem of reduced accuracy in the estimation result. This situation is not limited to when the estimation target is a person, but also applies to non-human animals. Furthermore, the quantity to be estimated is not limited to when the quantity to be estimated is height, but also applies to other body dimensions of an animal, such as shoulder width.

[0005] In view of the above circumstances, an object of the present invention is to provide a technology for improving the accuracy of estimating the size of an object to be estimated that appears in a video.

[0006] One aspect of the present invention is an estimation device that includes a control unit that estimates an estimation target quantity, which is the size of an animal to be estimated, using one or more frames that constitute a video to be processed, which has at least one frame depicting the animal to be estimated and a reference object of known size, in which the difference between the depth value of the animal to be estimated and the depth value of the reference object is less than a threshold value.

[0007] One aspect of the present invention is an estimation method that estimates an estimation target quantity, which is the size of an animal to be estimated, using one or more frames that make up a video to be processed, which has at least one frame depicting the animal to be estimated and a reference object of known size, in which the difference between the depth value of the animal to be estimated and the depth value of the reference object is less than a threshold value.

[0008] One aspect of the present invention is a program for causing a computer to function as the above-described estimation device.

[0009] According to the present invention, it is possible to improve the accuracy of estimating the size of an object to be estimated that appears in a video.

[0010] FIG. 1 is an explanatory diagram illustrating an overview of the estimation device 1 in the first embodiment. FIG. 2 is an explanatory diagram (part 2) for explaining an example of the selection processing in the first embodiment. FIG. 3 is an explanatory diagram (part 4) for explaining an example of the selection processing in the first embodiment. FIG. 4 is an explanatory diagram for explaining an example of the estimation target quantity estimation processing in the first embodiment. FIG. 5 is a flowchart showing an example of the flow of processing executed by a control unit 11 included in the estimation device 1 in the first embodiment. FIG. 6 is a diagram showing an example of the hardware configuration of the estimation device 1 in each embodiment.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] First Embodiment Fig. 1 is an explanatory diagram illustrating an overview of an estimation device 1 according to a first embodiment. The estimation device 1 performs estimation processing on a moving image to be processed (hereinafter referred to as a "moving image to be processed"). The moving image to be processed may be input to the estimation device 1 from an external device, or may be stored in advance in the estimation device 1. The estimation processing is processing for estimating the size of an estimation target that appears in the moving image to be processed (hereinafter referred to as an "estimated amount of the estimation target").

[0013] The estimation target is a target whose estimation target quantity is to be estimated. In the following description, a case where the estimation target is a human will be described as an example, but the estimation target may be an animal other than a human. In addition, in the following description, a case where the estimation target quantity is the height of the estimation target will be described as an example, but the estimation target quantity may be other body dimensions such as the shoulder width of the estimation target.

[0014] The estimation device 1 includes a control unit 11 including at least a processor 91, such as a CPU (Central Processing Unit), and a memory 92, connected via a bus. The control unit 11 executes estimation processing on a moving image to be processed. The moving image to be processed is a moving image having at least one frame (still image) that captures a person to be estimated and a reference object of known size. In other words, at least one of the frames constituting the moving image to be processed captures a pair of the person to be estimated and the reference object.

[0015] Note that the term "known size" means that the size has been stored in advance before the execution of the estimation process in the estimation device 1, or that information indicating the size has been input before the execution of the estimation process. In other words, the term "known size" means that the size has been determined in advance for the estimation device 1.

[0016] The estimation device 1 performs the following estimation processes: frame division process, bounding box estimation process, selection process, and estimation target quantity estimation process. The frame division process is a process of dividing a moving image to be processed into one or more frames. Any well-known technique may be used to divide the moving image into frames.

[0017] The bounding box estimation process is a process of estimating a bounding box surrounding a person to be estimated (hereinafter referred to as an "estimation target box") and a bounding box surrounding a reference object (hereinafter referred to as a "reference box") for each frame constituting a moving image to be processed. Note that depending on the frame, either the person to be estimated or the reference object may not be shown. In this case, the estimation device 1 will be unable to estimate at least one of the estimation target box or the reference box in the bounding box estimation process.

[0018] When the estimation device 1 is unable to estimate at least one of the estimation target box or the reference box in any of the frames to be processed, it outputs an abnormal value as the estimation result for that frame to be processed. Note that when at least one of the estimation target box or the reference box cannot be estimated in any of the frames to be processed, this means that, as described above, there is no person or reference object to be estimated in the frame to be processed. The abnormal value is a predetermined value that indicates that at least one of the estimation target box or the reference box cannot be estimated.

[0019] The technique for a computer to distinguish between the person to be estimated and the reference object in the frame and estimate the bounding box may be, for example, the technique described in Non-Patent Document 1 or 2.

[0020] The selection process is a process of selecting one or more frames from among the frames constituting the video to be processed to be used for estimating the estimation target quantity. Hereinafter, the frames selected by the selection process are referred to as frames to be used. In the selection process of this embodiment, the estimation device 1 selects one or more frames that contain both the person to be estimated and a reference object, and in which the difference in depth between the person to be estimated and the reference object shown in the frame is less than a threshold. The depth of the person to be estimated represents the distance between the camera that captured the video to be processed and the person to be estimated. The depth of the reference object represents the distance between the camera that captured the video to be processed and the reference object.

[0021] The estimation device 1 generates depth images using each frame constituting a moving image to be processed. A method for generating depth images using each frame constituting a moving image to be processed may use, for example, the technique described in Reference 1 below. Note that the method for generating depth images for each frame constituting a moving image to be processed is not limited to the technique described in the reference below, and machine learning may also be used. (Reference 1: Rene Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun, “Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer,” IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, pp. 1623-1637, VOL. 44, 2022)

[0022] As described above, when selecting a frame to be used, information indicating the bounding boxes of the person to be estimated and the reference object is required. Therefore, the bounding box estimation process must be performed in advance. Therefore, the estimation device 1 outputs an abnormal value as the estimation result for a frame for which the bounding boxes of the person to be estimated and the reference object have not been obtained in the bounding box estimation process. The estimation device 1 does not select, as a frame to be used, a frame for which an abnormal value has been output in the bounding box estimation process. In this way, in the selection process, it is also determined whether each frame is a frame to be used based on the result of the bounding box estimation process.

[0023] Next, the selection process performed by the estimation device 1 will be described with reference to FIGS. 2 to 5. FIG. 2 is an explanatory diagram (part 1) for explaining an example of the selection process in the first embodiment. (A) of FIG. 2 shows a frame (original image) at a certain time, and (B) of FIG. 2 shows a depth image of the frame shown in (A) of FIG. 2. FIG. 2 shows a case where a person is located farther away from the camera position than the object (a case where the depth difference between the person and the reference object is large). In other words, FIG. 2 shows a case where the distance between the person and the object is different from the camera position. In this case, if the person's height is estimated based on the size of the reference object, it is conceivable that the error from the actual height will be large.

[0024] For example, when the estimation device 1 calculates the difference between the depth value of a person appearing in the depth image shown in Fig. 2B and the depth value of a reference object and compares the calculated difference value with a threshold, it is expected that the difference value will be greater than the threshold. Therefore, the estimation device 1 does not select, as a target frame, a frame whose difference value is not greater than the threshold.

[0025] 3 is an explanatory diagram (part 2) for explaining an example of the selection process in the first embodiment. (A) in FIG. 3 shows a frame (original image) at a certain time, and (B) in FIG. 3 shows a depth image of the frame shown in (A) in FIG. 3. FIG. 3 shows a case where a person and an object are at the same distance from the camera position (where the depth difference between the person and the reference object is small). In this case, if the person's height is estimated based on the size of the reference object, it is conceivable that the error from the actual height will be small.

[0026] For example, when the estimation device 1 calculates the difference between the depth value of a person in the depth image shown in Fig. 3B and the depth value of a reference object and compares the calculated difference value with a threshold, it is assumed that the difference value is equal to or smaller than the threshold. Therefore, the estimation device 1 selects a frame whose difference value is equal to or smaller than the threshold as a target frame to be used.

[0027] FIG. 4 is an explanatory diagram (part 3) for explaining an example of the selection process in the first embodiment. (A) in FIG. 4 shows a frame (original image) at a certain time, and (B) in FIG. 4 shows a depth image of the frame shown in (A) in FIG. 4. FIG. 4 shows a case where a person is closer to the camera position than the object (a case where the depth difference between the person and the reference object is large). In other words, FIG. 4 shows a case where the distance between the person and the object is different from the camera position. In this case, if the person's height is estimated based on the size of the reference object, it is conceivable that the error from the actual height will be large.

[0028] For example, when the estimation device 1 calculates the difference between the depth value of a person appearing in the depth image shown in Fig. 4B and the depth value of a reference object and compares the calculated difference value with a threshold, it is expected that the difference value will be greater than the threshold. Therefore, the estimation device 1 does not select, as a target frame, a frame whose difference value is not greater than the threshold.

[0029] FIG. 5 is an explanatory diagram (part 4) for explaining an example of the selection process in the first embodiment. The horizontal axis in FIG. 5 represents the frame number, and the vertical axis represents the relative depth data. The smaller the value on the horizontal axis of the graph in FIG. 5 (i.e., the further to the left in FIG. 5), the smaller the frame number. The smaller the frame number, the earlier the frame was captured. Therefore, the frame number is equivalent to the capture time.

[0030] As shown in FIG. 5 , the difference between the depth value of the reference object and the depth value of the person is large in frame numbers "F0" and "F100," while the difference between the depth value of the reference object and the depth value of the person is small in frame number "F50." The situation in which relative depth data for frame number "F0" is obtained is assumed to be the situation shown in FIG. 2 . The situation in which relative depth data for frame number "F50" is obtained is assumed to be the situation shown in FIG. 3 . The situation in which relative depth data for frame number "F100" is obtained is assumed to be the situation shown in FIG. 4 . The estimation device 1 selects a frame in which the difference between the depth value of the reference object and the depth value of the person is within a threshold (e.g., frame number "F50") as a target frame for use. Using the above-described method, the estimation device 1 selects one or more target frames for use from among the frames constituting the moving image to be processed.

[0031] Returning to FIG. 1 , the description will continue. The estimation target quantity estimation process is a process for estimating an estimation target quantity based on a target frame and one or more selected frames. When multiple frames are selected as target frames in the selection process, the estimation device 1 may select one frame that satisfies a condition indicating a small difference as the target frame. The condition indicating a small difference is that the difference value is less than a second threshold value that is smaller than the threshold value. Thus, for example, when multiple frames are selected as target frames in the selection process, the estimation device 1 may select one frame with the smallest difference value as the target frame.

[0032] FIG. 6 is an explanatory diagram illustrating an example of the estimation target quantity estimation process in the first embodiment. In FIG. 6, a reference object 901 represents the bounding box of the reference object. A box 902 is the bounding box of the person to be estimated. The X axis in FIG. 6 represents the horizontal direction, and the Y axis in FIG. 6 represents the vertical direction. "Object center coordinates" refer to the coordinates of the center of the bounding box of the reference object. Hereinafter, the center of the bounding box of the reference object will be referred to as the object center. The object Y-axis width in FIG. 6 refers to the length of the reference object in the Y-axis direction. Note that the person Y-axis width in FIG. 6 refers to the length of the person to be estimated in the Y-axis direction.

[0033] The estimation device 1 estimates the estimation target quantity based on one selected use target frame. Note that the process of estimating the estimation target quantity captured in one still image (frame) based on the still image may use the technology described in Non-Patent Document 2, for example.

[0034] It is also possible that the posture of the person depicted in the frame selected as the target frame is not upright, or that some of the predetermined feature points along the person's skeletal structure are missing. For example, if the person depicted in the frame selected as the target frame is bent, the posture needs to be corrected. Alternatively, if some of the predetermined feature points along the person's skeletal structure are missing in the frame selected as the target frame, the missing parts need to be corrected or complemented. Therefore, the estimation device 1 may use the technology described in References 2 or 3 below to correct the person's posture or correct or complement parts of the predetermined feature points along the person's skeletal structure in order to estimate the estimation target quantity. (Reference 2: International Publication No. WO 2021 / 1661482) (Reference 3: International Publication No. WO 2022 / 168252)

[0035] 7 is a flowchart showing an example of the flow of processing executed by the control unit 11 included in the estimation device 1 according to the first embodiment. The control unit 11 acquires a processing target moving image stored in an external device or the internal memory 92 (step S101). The control unit 11 executes a frame division process on the acquired processing target moving image (step S102). By executing the frame division process, one or more frames constituting the processing target moving image are generated.

[0036] Next, the control unit 11 executes a bounding box estimation process for each of the generated one or more frames (step S103). Next, the control unit 11 executes a selection process using one or more frames on which the bounding box estimation process has been executed (step S104). Specifically, the control unit 11 first generates a depth image using each of the one or more frames on which the bounding box estimation process has been executed. Note that the control unit 11 does not select a frame from among the one or more frames for which an abnormal value has been output in the bounding box estimation process as a frame to be used, and therefore does not need to generate a depth image. As a result, one or more depth images are generated.

[0037] Next, the control unit 11 calculates a difference between the depth value of the person and the depth value of the reference object in each of one or more depth images. The control unit 11 compares the calculated difference value in each of one or more depth images with a threshold, and selects one or more frames whose difference value is equal to or less than the threshold as target frames. Here, the control unit 11 selects the target frame with the smallest difference value as the target frame to be finally used.

[0038] The control unit 11 executes an estimation target quantity estimation process using the selected use target frame (step S105). Then, the control unit 11 controls the operation of a predetermined output destination such as the output unit 15 (described later) to output the estimation target quantity obtained in the process of step S105 to the predetermined output destination (step S106). The estimation target quantity output in step S106 is the estimation target quantity desired by the user (for example, height).

[0039] In the example of FIG. 7, the processes from step S102 to step S105 are an example of the estimation process.

[0040] The estimation device 1 configured as described above includes a control unit 11 that estimates an estimation target quantity, which is the size of the animal to be estimated and a predefined size, using one or more frames constituting a video to be processed, the video having at least one frame showing an animal to be estimated and a reference object of known size, in which the difference between the depth value of the animal to be estimated and the depth value of the reference object is equal to or less than a threshold. This reduces estimation errors in the estimation target quantity due to differences in depth, thereby improving the accuracy of estimating the size of the target object shown in the video.

[0041] Furthermore, the control unit 11 does not generate depth images for frames in which at least one of the bounding box of the target animal or the bounding box of the reference object cannot be estimated, i.e., for frames in which abnormal values ​​are output. This eliminates the need for the control unit 11 to generate depth images for frames that are not used to estimate the target quantity. This reduces the processing load on the control unit 11.

[0042] Second Embodiment In the first embodiment, a configuration was described in which an estimation device estimates an estimation target quantity using one target frame. In this case, it is expected that a fluctuation error occurs in the estimated depth value relative to the true depth value. As a result, the estimation accuracy of the estimation target quantity may decrease depending on the selected target frame. Therefore, in the second embodiment, a configuration for solving such a problem will be described.

[0043] In the second embodiment, the configuration of the estimation device 1 is the same as in the first embodiment. Differences in processing from the estimation device 1 in the first embodiment will be described below. The estimation device 1 in the second embodiment differs from the first embodiment in the estimation target quantity estimation process. Specifically, the control unit 11 of the estimation device 1 estimates the estimation target quantity in each of multiple use target frames selected in the selection process. In this way, while the control unit 11 in the first embodiment estimates the estimation target quantity using one use target frame, the control unit 11 in the second embodiment estimates the estimation target quantity using multiple use target frames.

[0044] The control unit 11 calculates statistical values ​​of a plurality of estimation target quantities estimated in each of a plurality of use target frames. The statistical values ​​are, for example, average values, median values, etc. The control unit 11 calculates the statistical values ​​of all the estimated estimation target quantities as the final estimation target quantity.

[0045] According to the estimation device 1 of the second embodiment configured as above, it is possible to obtain the same effects as those of the first embodiment.

[0046] Furthermore, according to the estimation device 1 of the second embodiment, it is possible to suppress variations in estimation by taking statistical values, thereby further improving the accuracy of estimation by the estimation process.

[0047] <Hardware Description> Fig. 8 is a diagram showing an example of the hardware configuration of the estimation device 1 in each embodiment. As described above, the estimation device 1 includes the control unit 11 and executes a program. By executing the program, the estimation device 1 functions as a device including the control unit 11, input unit 12, communication unit 13, storage unit 14, and output unit 15.

[0048] More specifically, the processor 91 reads out a program stored in the storage unit 14 and stores the read out program in the memory 92. When the processor 91 executes the program stored in the memory 92, the estimation device 1 functions as a device including a control unit 11, an input unit 12, a communication unit 13, a storage unit 14, and an output unit 15.

[0049] The control unit 11 controls the operation of various functional units included in the estimation device 1. The control unit 11 controls, for example, the operation of the output unit 15. The control unit 11 executes, for example, an estimation process. The control unit 11 records, in the storage unit 14, various pieces of information generated by the operation of the control unit 11.

[0050] The input unit 12 includes input devices such as a mouse, a keyboard, and a touch panel. The input unit 12 may be configured as an interface that connects these input devices to the estimation device 1. The input unit 12 accepts input of various types of information to the estimation device 1.

[0051] The communication unit 13 includes a communication interface for connecting the estimation device 1 to an external device. The communication unit 13 communicates with the external device via a wired or wireless connection. The external device is, for example, a device that is the sender of the moving image to be processed. In such a case, the communication unit 13 acquires the moving image to be processed by communicating with the device that is the sender of the moving image to be processed.

[0052] The moving image to be processed does not necessarily have to be input via the communication unit 13 , but may be input to the input unit 12 .

[0053] The storage unit 14 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The storage unit 14 stores various information related to the estimation device 1. The storage unit 14 stores information input via, for example, the input unit 12 or the communication unit 13. The storage unit 14 stores various information generated by the execution of processing by, for example, the control unit 11. The storage unit 14 stores information indicating the size of a reference object in advance. The size of the reference object is, for example, height and width.

[0054] The output unit 15 outputs various types of information. The output unit 15 is configured to include a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The output unit 15 may be configured as an interface that connects these display devices to the estimation device 1. The output unit 15 outputs, for example, information input to the input unit 12. The output unit 15 outputs, for example, an estimation target quantity estimated by the estimation process.

[0055] (Modification 1 common to the first and second embodiments) The estimation device 1 may be implemented using a plurality of information processing devices communicably connected via a network. In this case, each functional unit of the estimation device 1 may be distributed and implemented among the plurality of information processing devices.

[0056] (Modification 2 common to the first and second embodiments) The control unit 11 may be mounted on a smartphone, a mobile phone, or a personal computer. That is, the estimation device 1 may be applied to a smartphone, a mobile phone, or a personal computer.

[0057] Note that all or part of the functions of the estimation device 1 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via a telecommunications line.

[0058] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0059] The present invention can be applied to a technique for estimating the body size of an animal that appears in a video consisting of one or more frames.

[0060] REFERENCE SIGNS LIST 1... estimation device, 11... control unit, 12... input unit, 13... communication unit, 14... storage unit, 15... output unit, 91... processor, 92... memory

Claims

1. Among one or more frames constituting a video to be processed having at least one frame depicting an animal to be estimated and a reference object of a known size, using one or more frames in which the difference between the depth value of the animal to be estimated and the depth value of the reference object is equal to or less than a threshold value, a control unit that estimates an estimated target amount that is the size of the animal to be estimated. An estimation device comprising:

2. The control unit generates a depth image using each of one or more frames constituting the video to be processed, and calculates the difference between the depth value of the animal to be estimated and the depth value of the reference object in each of the generated plurality of depth images. The estimation device according to claim 1.

3. The control unit executes a process of estimating the bounding boxes of the animal to be estimated and the reference object, and does not generate a depth image for a frame in which at least one of the bounding box of the animal to be estimated or the bounding box of the reference object cannot be estimated. The estimation device according to claim 2.

4. When there are a plurality of frames in which the difference between the depth value of the animal to be estimated and the depth value of the reference object is equal to or less than a threshold value, the control unit estimates the estimated target amount using one frame that satisfies a condition indicating that the difference is small. The estimation device according to any one of claims 1 to 3.

5. When there are a plurality of frames in which the difference between the depth value of the animal to be estimated and the depth value of the reference object is equal to or less than a threshold value, the control unit estimates the estimated target amount for the plurality of frames, and uses the statistical value of the plurality of estimated target amounts as the final estimated target amount to estimate the estimated target amount. The estimation device according to any one of claims 1 to 3.

6. An estimation method of estimating an estimated target amount that is the size of an animal to be estimated, using one or more frames in which the difference between the depth value of the animal to be estimated and the depth value of a reference object of a known size is equal to or less than a threshold value, among one or more frames constituting a video to be processed having at least one frame depicting the animal to be estimated and the reference object.

7. A program for causing a computer to function as the estimation device according to claim 1.

Images