Area detection program, device, and method

Abstract

This region detection device is configured to: identify a person's center line in a perpendicular direction in a three-dimensional space by projecting a detected two-dimensional bounding box from a two-dimensional space onto the three-dimensional space by using internal and external parameters of a camera; estimate the width of the three-dimensional bounding box on the basis of the height of the three-dimensional bounding box, which is the length of the person's center line, and statistical information indicating average values of the height and width of the three-dimensional bounding box, the statistical information being calculated in advance from three-dimensional models of a gymnast in various postures; and interpolate the 2D-BB in an image by projecting a three-dimensional bounding box, which is specified by the person's center line and the height and width of the 3D-BB, from the three-dimensional space onto the two-dimensional space by using the internal and external parameters of the camera.

Description

[Technical field]

[0001] The disclosed technology relates to an area detection program, an area detection device, and an area detection method. [Background technology]

[0002] Conventionally, there is a technology that recognizes three-dimensional information of a person, such as skeletal information, from an image of the person by using a recognition model that has been generated in advance by machine learning. In such a technology, a bounding box that encloses the area of ​​the person that is the target of the recognition process is detected.

[0003] For example, a 3D model generation device has been proposed that generates a 3D model of a subject from images captured by multiple cameras. The device acquires a silhouette image for each viewpoint from a multi-viewpoint image, and generates a low-resolution voxel model with a first voxel size from the multiple silhouette images using a volume intersection method. The device also classifies the low-resolution voxel models based on their characteristics, and determines a second size smaller than the first size for each low-resolution voxel model based on the classification result. The device then generates a high-resolution voxel model with a voxel size of the second size for each 3D bounding box of the low-resolution voxel model, and outputs a 3DCG model of the subject based on the high-resolution voxel model.

[0004] There is also a technique for generating an image from an intermediate viewpoint by using perspective projection to estimate corresponding points at other viewpoints from multi-viewpoint images captured from a plurality of viewpoints. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-071749 [Non-patent literature]

[0006] [Non-Patent Document 1] Hideo Saito, Makoto Kimura, Satoshi Yaguchi, Naho Inamoto, "View Interpolation of Multiple Cameras Based on Projective Geometry", 2002. Summary of the Invention [Problem to be solved by the invention]

[0007] When a bounding box is detected as a region showing a target person from an image using a machine learning model, the bounding box may not be detected or may be erroneously detected. If such a bounding box is not detected or may be erroneously detected in any of the images in a multi-view image for acquiring three-dimensional information of a person, the three-dimensional information cannot be acquired with high accuracy in the subsequent processing.

[0008] Furthermore, when dealing with a person who can assume a variety of poses, such as a gymnast, simply applying perspective projection may not be enough to properly interpolate undetected or erroneously detected bounding boxes.

[0009] In one aspect, the disclosed technology aims to appropriately interpolate undetected or erroneously detected bounding boxes in multi-view images. [Means for solving the problem]

[0010] In one aspect, the disclosed technology acquires images captured by a plurality of image capture devices that capture images of a person from different directions. The disclosed technology also inputs the acquired images into a machine learning model that has been generated in advance by machine learning to detect an area of ​​the person included in the image, and detects an area showing the person from each of the acquired images. The disclosed technology then interpolates an area showing the person in a second image of the acquired images based on the area of ​​the person detected in a first image of the acquired images and parameters of each of the plurality of image capture devices. Effect of the Invention

[0011] As one aspect, it has an effect of being able to appropriately interpolate undetected or erroneously detected bounding boxes in multi-view images. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 is a schematic diagram showing a connection between the area detection device and a camera. [Diagram 2] FIG. 2 is a functional block diagram of the area detection device. [Diagram 3] FIG. 1 is a diagram for explaining a two-dimensional bounding box. [Figure 4] FIG. 13 is a diagram for explaining differences in the width of a two-dimensional bounding box depending on the viewpoint. [Diagram 5] FIG. 13 is a diagram for explaining the specification of a three-dimensional person center line. [Figure 6] FIG. 13 is a diagram for explaining two-dimensional bounding box interpolation. [Figure 7] FIG. 13 is a diagram for explaining the width and height of a three-dimensional bounding box. [Figure 8] FIG. 13 is a diagram for explaining statistical information regarding a three-dimensional bounding box. [Figure 9] FIG. 1 is a block diagram showing a schematic configuration of a computer functioning as an area detection device. [Figure 10] 13 is a flowchart illustrating an example of an area detection process. [Figure 11] FIG. 1 illustrates an example of two-dimensional bounding box interpolation. [Figure 12] 1A and 1B are diagrams for explaining an example of a technique for applying a multi-view image in which a two-dimensional bounding box is detected. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. In the following embodiment, a case where a target person is a gymnast will be described, but the disclosed technology can be applied to various targets such as athletes of other sports and general pedestrians.

[0014] As shown in Fig. 1, area detection device 10 according to this embodiment is connected to each of a plurality of cameras 30n that capture images of gymnast 90 at viewpoints n from different directions. In the example of Fig. 1, n = 1, 2, 3, and camera 301 capturing images from viewpoint 1, camera 302 capturing images from viewpoint 2, and camera 303 capturing images from viewpoint 3 are connected to area detection device 10. Note that the number of cameras 30n connected to area detection device 10 is not limited to the example of Fig. 1.

[0015] The cameras 30n are installed at different positions on approximately the same horizontal plane at angles that allow the gymnast 90 to be within the shooting range. That is, the cameras 30n are arranged side by side in the horizontal direction so as to surround the gymnast 90. Note that, here, "approximately on the same horizontal plane" refers to a case where the heights of the cameras 30n from the floor surface can be considered to be approximately the same, and where the difference in height of the cameras 30n from the floor surface is equal to or less than a predetermined value. The images captured by the cameras 30n are input sequentially to the area detection device 10. Note that time information is associated with each frame included in the images captured by each camera 30n, and the images captured by each camera 30n can be synchronized based on this time information.

[0016] 2, the area detection device 10 functionally includes an acquisition unit 12, a detection unit 14, and an interpolation unit 16. Furthermore, a detection model 20 is stored in a predetermined storage area of ​​the area detection device 10.

[0017] The acquisition unit 12 acquires, as a multi-viewpoint image, a set of images indicated by frames whose time information corresponds in the video input from the camera 30n to the area detection device 10. In the following, among the images included in the multi-viewpoint image, an image captured by the camera 30n is referred to as an image 40n.

[0018] The detection unit 14 inputs an image 40n included in the multi-view image acquired by the acquisition unit 12 to the detection model 20, and detects a two-dimensional bounding box (hereinafter, also referred to as "2D-BB") as an area showing a person from each of the images 40n. The detection model 20 is generated in advance using images in which a correct 2D-BB is given for images showing gymnasts in various poses as training data. The detection model 20 is an example of a "machine learning model" of the disclosed technology. For example, as shown in FIG. 3, the detection unit 14 detects a circumscribing rectangle of an area showing a gymnast 90 in the image 40n as a 2D-BB 42n.

[0019] The interpolation unit 16 interpolates the 2D-BB 42n in the second image of the multi-viewpoint image based on the 2D-BB 42n detected from the first image of the multi-viewpoint image and the parameters of each of the cameras 30n. In this embodiment, the 2D-BB 42k is detected from the image 40k captured by the camera 30k of the viewpoint k, and the 2D-BB 42_miss is not detected from the image 40_miss captured by the camera 30_miss of the viewpoint_miss. In this case, the image 40k is an example of the first image, and the image 40_miss is an example of the second image.

[0020] Here, when using the detection model 20, which is a machine learning model as described above, the 2D-BB 42n may be undetected or erroneously detected from the image 40n. This may occur when a part of the gymnast 90 in the image 40n is obscured by an obstacle, or when the posture of the target gymnast 90 is not similar to any of the gymnasts in the images used to train the detection model 20. Consider acquiring three-dimensional information of the gymnast 90 based on an image in the 2D-BB 42n detected from each image 40n included in the multi-view image and a recognition model that recognizes three-dimensional information such as skeletal information generated in advance by machine learning. In this case, if an image in which the 2D-BB 42n is undetected or erroneously detected is included in the multi-view image, the recognition accuracy of the three-dimensional information will decrease. Therefore, it is desirable to interpolate the undetected or erroneously detected BB 42n.

[0021] However, when targeting a person who can assume various postures, such as a gymnast, simply applying perspective projection may not be sufficient to properly interpolate undetected or erroneously detected 2D-BB 42n. A more detailed explanation will be given with reference to FIG. 4. As shown in FIG. 4, a three-dimensional bounding box (hereinafter also referred to as "3D-BB") 44 is assumed to be a rectangular parallelepiped consisting of horizontal and vertical planes, and is the smallest rectangular parallelepiped that completely surrounds a gymnast 90 in three-dimensional space. At the viewpoint n of each camera 30n, the 3D-BB 44 is projected onto each image 40n to become the 2D-BB 42n.

[0022] When the cameras 30n are arranged side by side in the horizontal direction so as to surround the gymnast 90, the height (vertical length) of the 2D-BB 42n in each image 40n is the same regardless of the viewpoint of each camera 30n. In FIG. 4, the heights of the 2D-BB 42n and the 3D-BB 44 are represented by lines connecting the star marks. Meanwhile, the width (horizontal length) of the 2D-BB 42n in each image 40n differs depending on the viewpoint of each camera 30n. In the example of FIG. 4, the width of the 2D-BB 421 detected from the image 401 and the width of the 2D-BB 422 detected from the image 402 differ due to the difference in the viewpoint. Therefore, for example, since the 2D-BB 423 has not been detected in the image 403, even if an attempt is made to simply apply perspective projection as in the prior art and to interpolate from the detected 2D-BB 421 and 422, the width of the 2D-BB 423 cannot be appropriately determined. Therefore, the 2D-BB 423 in the image 403 cannot be interpolated.

[0023] Therefore, the interpolation unit 16 specifies the height of 2D-BB 42_miss in image 40_miss where 2D-BB is not detected, based on the height of 2D-BB 42k in image 40k where 2D-BB is detected and the parameters of each camera 30k. Then, the interpolation unit 16 estimates the width of 2D-BB 42_miss in image 40_miss where 2D-BB is not detected, based on the statistical information on the posture of the gymnast 90 and the parameters of camera 30_miss.

[0024] A specific description will be given with reference to Fig. 5 and Fig. 6. The interpolation unit 16 converts the 2D-BB 42k detected from the image 40k into B k =[x k ,y k ,w k ,h k ]. x k and y k are the coordinates of the upper left corner point of the 2D-BB 42k in the image 40k, and w and h are the width and height of the 2D-BB 42k, respectively. miss =[x miss ,y miss ,w miss ,h miss ]. That is, B miss Specifying the missing value corresponds to interpolating 2D-BB42_miss. In the examples of Fig. 5 and Fig. 6, k = 1, 2, and _miss = 3.

[0025] The interpolation unit 16 calculates the coordinates [x k +w k / 2,y k ], and the coordinates of the lower end point [x k +w k / 2,y k +h k Then, the interpolation unit 16 converts the coordinates of the upper end point and the lower end point into three-dimensional coordinates using parameters of the camera 30k for converting the three-dimensional coordinates into coordinates on the image plane captured by the camera 30k. For example, the interpolation unit 16 may convert the coordinates using cv::sfm::trianglePoints defined in OpenCV (https: / / docs.opencv.org / 3.4 / d0 / dbd / group__triangulation.html). In this case, the interpolation unit 16 converts the coordinates of points P3d top and P3d bot The three-dimensional coordinates of the camera 30k are calculated using the parameter matrix ProjMat cam_k The calculation is performed using the following equations (1) and (2). In three-dimensional space, P3d topand P3d bot The line connecting these is called the person's center line.

[0026] P3d top =[X top ,Y top ,Z top ] =cv::sfm::trianglePoints({[x 1 +w 1 / 2,y 1 ],[x 2 +w 2 / 2,y 2 ],...,[x k +w k / 2,y k ]},{ProjMat cam_1 ,ProjMat cam_2 ,···,ProjMat cam_k}) (1) P3d bot =[X bot ,Y bot ,Z bot ] =cv::sfm::trianglePoints({[x 1 +w 1 / 2,y 1 +h 1 ],[x 2 +w 2 / 2,y 2 +h 2 ],...,[x k +w k / 2,y k +h k ]},{ProjMat cam_1 ,ProjMat cam_2 ,···,ProjMat cam_k}) (2)

[0027] In addition, the interpolation unit 16 performs the interpolation process as shown in FIG. top and P3d botThe three-dimensional coordinates of the 2D-BB 42_miss are converted into coordinates on the image 40_miss based on the parameters of the camera 30_miss. In this way, the interpolation unit 16 calculates the coordinates of the upper and lower end points of the vertical center line of the 2D-BB 42_miss. For example, the interpolation unit 16 converts the parameter matrix ProjMat cam_miss , and a parameter s representing the scale ratio between the three-dimensional coordinates and the size of the image 40, coordinate conversion is performed as shown in the following equations (3) and (4).

[0028] [s×(x miss +w miss / 2), s×y miss ,s] =ProjMat cam_miss ×[X top ,Y top ,Z top ,1] (3) [s×(x miss +w miss / 2),s×(y miss +h miss ),s] =ProjMat cam_miss ×[X bot ,Y bot ,Z bot ,1] (4)

[0029] Based on the above coordinate transformation, the interpolation unit 16 calculates y miss , h miss , and x miss +w miss / 2 is calculated as shown in the following equations (5) to (7).

[0030] y miss =s×y miss / s (5) h miss =s×(y miss +h miss ) / sy miss (6) x miss +w miss / 2=s×(x miss +w miss / 2) / s (7)

[0031] The interpolation unit 16 specifies the line connecting the specified upper end point and lower end point as the vertical center line of 2D-BB42_miss, and specifies the length of the center line as the height of 2D-BB42_miss.

[0032] The interpolation unit 16 also estimates the width of 2D-BB42_miss based on the identified height of 2D-BB42_miss and statistical information related to the posture of the gymnast 90. The statistical information may be, for example, the average of the sum of the height and width of 3D-BB44 surrounding the gymnast in each of a number of different postures taken by the gymnast. In the following, as shown in Fig. 7, the height of the 3D-BB44 is Height_3D, the larger of the widths is Width_max_3D, and the smaller is Width_min_3D.

[0033] The statistical information will be described in more detail. First, as shown in FIG. 8, a life-size 3D model of a gymnast is prepared for multiple postures (poses) obtained from motion capture, manual annotation, publicly available data, etc. In the example of FIG. 8, 3D models for M poses are prepared. Then, for each 3D model, 3D-BB44 is identified, Height_3D, Width_max_3D, and Width_min_3D are calculated, and the average Mean_3D shown in the following formula (8) is calculated as statistical information.

[0034] Mean_3D=(1 / M)Σ M ((Width_max_3D+Width_min_3D) / 2+Height_3D) (8)

[0035] The interpolation unit 16 calculates the statistical information Mean_3D and the already calculated Z top and Z bot Using these, the width Width_3D of the 3D-BB 44 of the target gymnast 90 is calculated as shown in the following formula (9). Width_3D=Mean_3D-||Z top -Z bot|| (9)

[0036] When the 3D-BB is perspectively projected onto an image, the ratio of the height to the width of the projected 2D-BB is the same as the ratio of the height to the width of the 3D-BB before projection. Therefore, the interpolation unit 16 calculates the ratio of the calculated Width_3D and the already calculated Z top , Z bot , and h miss Using the above, the width w of 2D-BB42_miss is calculated as shown in the following equation (10). miss Furthermore, the interpolation unit 16 calculates the calculated w miss And by equation (7), x miss Calculate. w miss =(Width_3D / ||Z top -Z bot ||)×h miss (10) x miss =x miss +w miss / 2-w miss / 2 (11)

[0037] The interpolation unit 16 calculates [x miss ,y miss ,w miss ,h miss In the image 40_miss, the interpolation unit 16 interpolates the 2D-BB 42_miss specified by 2DBB 42k. The interpolation unit 16 then outputs the interpolated 2D-BB 42_miss and 2DBB 42k together as a multi-viewpoint image in which 2D-BB has been detected.

[0038] The area detection device 10 may be realized by, for example, a computer 50 shown in Fig. 9. The computer 50 includes a CPU (Central Processing Unit) 51, a memory 52 as a temporary storage area, and a non-volatile storage unit 53. The computer 50 also includes an input / output I / F (Interface) 54 to which external devices such as a camera 30n, an input device, and a display device are connected, and an R / W (Read / Write) unit 55 that controls reading and writing of data from and to a storage medium 59. The computer 50 also includes a communication I / F 56 that is connected to a network such as the Internet. The CPU 51, the memory 52, the storage unit 53, the input / output I / F 54, the R / W unit 55, and the communication I / F 56 are connected to one another via a bus 57.

[0039] The storage unit 53 may be realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like. The storage unit 53 serving as a storage medium stores a region detection program 60 for causing the computer 50 to function as the region detection device 10. The region detection program 60 has an acquisition process 62, a detection process 64, and an interpolation process 66. The storage unit 53 also has an information storage area 70 in which information constituting the detection model 20 is stored.

[0040] The CPU 51 reads the area detection program 60 from the storage unit 53, loads it in the memory 52, and sequentially executes the processes of the area detection program 60. The CPU 51 operates as the acquisition unit 12 shown in FIG. 2 by executing the acquisition process 62. The CPU 51 operates as the detection unit 14 shown in FIG. 2 by executing the detection process 64. The CPU 51 operates as the interpolation unit 16 shown in FIG. 2 by executing the interpolation process 66. The CPU 51 reads information from the information storage area 70 and loads the detection model 20 in the memory 52. ​​As a result, the computer 50 that has executed the area detection program 60 functions as the area detection device 10. The CPU 51 that executes the program is hardware.

[0041] The functions realized by the area detection program 60 can also be realized by, for example, a semiconductor integrated circuit, more specifically, a GPU (Graphics Processing Unit) or an ASIC (Application Specific Integrated Circuit).

[0042] Next, the operation of the area detection device 10 according to the present embodiment will be described. When a multi-viewpoint image is input from the camera 30n to the area detection device 10, the area detection device 10 executes the area detection process shown in Fig. 10. Note that the area detection process is an example of an area detection method of the disclosed technology.

[0043] In step S10, the acquisition unit 12 acquires the multi-view image input to the area detection device 10. Next, in step S12, the detection unit 14 inputs each image 40n included in the acquired multi-view image to the detection model 20 and detects 2D-BB 42n from each of the images 40n. Next, in step S14, the detection unit 14 determines whether or not there is an image 40n in which 2D-BB 42n has not been detected among the images 40n included in the multi-view image. If there is an image 40n in which 2D-BB 42n has not been detected, the process proceeds to step S16, and if there is no image 40n, the process proceeds to step S24.

[0044] In step S16, the interpolation unit 16 calculates the coordinates [x k +w k / 2,y k ], and the coordinates of the lower end point [x k +w k / 2,y k +h k Next, in step S18, the interpolation unit 16 converts the coordinates of the upper end point and the lower end point into three-dimensional coordinates using the parameter matrix of the camera 30k (denoted as "OK camera" in FIG. 10) to determine a point P3d top and P3d bot Then, the interpolation unit 16 determines P3d top and P3d bot The line connecting these is identified as the person's center line.

[0045] Next, in step S20, the interpolation unit 16 top and P3d bot The three-dimensional coordinates of 2D-BB42_miss are projected onto image 40_miss (denoted as "miss image" in FIG. 10) based on the parameter matrix of camera 30_miss (denoted as "miss camera" in FIG. 10). As a result, the vertical center line of 2D-BB42_miss is identified, and the length of the center line is identified as the height of 2D-BB42_miss.

[0046] Next, in step S22, the interpolation unit 16 estimates the width of 2D-BB42_miss based on the identified height of 2D-BB42_miss and statistical information related to the posture of the gymnast 90. As a result, 2D-BB42_miss identified by the vertical center line and height of 2D-BB42_miss identified in step S20 above and the width of 2D-BB42_miss estimated in this step is interpolated in the image 40_miss.

[0047] Next, in step S24, the interpolation unit 16 outputs a multi-view image in which the 2D-BB 42n is detected from each image 40n. The detected 2D-BB 42n includes the 2D-BB interpolated in step S22. Next, in step S26, the acquisition unit 12 determines whether or not the next multi-view image has been input to the region detection device 10. If the next multi-view image has been input to the region detection device 10, the process returns to step S10; if not, the region detection process ends.

[0048] FIG. 11 shows an example of interpolating a 2D-BB that was not detected in a multi-view image including four images. In FIG. 11, cam_id:n (n=1, 2, 3, 4) represents an image captured by a camera with id n. Also, frame:852 is a frame number associated with each image, and corresponds to the time information in the above embodiment. As shown in the left diagram of FIG. 11, the missing 2D-BB in cam_id:3 included in the multi-view image with frame number 852 is interpolated as shown in the right diagram of FIG. 11 by applying this embodiment.

[0049] As described above, the area detection device according to the present embodiment acquires a multi-view image, which is a set of images captured by a plurality of cameras that capture a person from different directions. The area detection device also inputs each of the images included in the acquired multi-view image into a detection model that has been generated in advance by machine learning so as to detect a bounding box indicating the area of ​​the person included in the image, and detects a bounding box from each of the images. The area detection device then interpolates a bounding box in a second image among the acquired images based on the bounding box detected from a first image among the acquired images and the parameters of each of the multiple cameras.

[0050] More specifically, the region detection device projects the detected 2D-BB from two dimensions to three dimensions using the internal and external parameters of the camera, and identifies the vertical center line of the person in the three-dimensional space. The region detection device then estimates the width of the 3D-BB based on statistical information indicating the average height and width of the 3D-BB, which have been calculated in advance from three-dimensional models of gymnasts in various postures, and the height of the 3D-BB, which is the length of the person's center line. Furthermore, the region detection device interpolates the 2D-BB in the image by projecting the 3D-BB identified from the person's center line, the height and width of the 3D-BB from three dimensions to two dimensions using the internal and external parameters of the camera. This makes it possible to appropriately interpolate undetected bounding boxes in a multi-view image.

[0051] The multi-view image with 2D-BB detection output from the region detection device according to the present embodiment is used for learning-type skeleton recognition of a gymnast, for example, as shown in FIG. 12. In the example of FIG. 12, a skeleton recognition model is generated in advance by machine learning using a pair of a multi-view image with known 2D-BB and the correct three-dimensional coordinates (hereinafter referred to as "3D joint coordinates") of each joint of the gymnast represented by the multi-view image as training data. The skeleton recognition model is, for example, a neural network. Then, when a multi-view image with 2D-BB detection output from the region detection device according to the present embodiment is acquired as a multi-view image of a gymnast to be recognized, the multi-view image is input to the machine-learned skeleton recognition model, and 3D joint coordinates are output. Then, the 3D joint coordinates output from the skeleton recognition model are used as the primary skeleton recognition result, and the 3D joint coordinates obtained by searching the position of each joint based on constraints such as the length between each joint and the positional relationship are output as the fitting result.

[0052] In the above embodiment, the case where multiple cameras are arranged on approximately the same horizontal plane has been described, but the present invention is not limited to this. For example, depending on the sport played by the athlete, a multi-view image taken by multiple cameras arranged on approximately the same vertical plane may provide a more accurate recognition result in skeletal recognition, etc. In this case, the width of the 3D-BB is specified based on the width of the 2D-BB detected from the first image and the parameters of the camera that took the first image, and the height of the 3D-BB is estimated based on the width of the 3D-BB and statistical information on the three-dimensional model of the gymnast.

[0053] In the above embodiment, the first image is an image in which 2D-BB is detected, and the second image is an image in which 2D-BB is not detected, that is, the case where 2D-BB is interpolated in an image in which 2D-BB is missing has been described, but the present invention is not limited to this. For example, each image included in a multi-view image may be a first image, and the other image may be a second image, and the 2D-BB detected in the first image may be corrected based on the 2D-BB that is interpolated in the same manner as in the above embodiment. In addition, when the detection model detects a bounding box and outputs the reliability of the detection, if the reliability of the detection is equal to or less than a predetermined value, the detection may be treated in the same manner as the case where 2D-BB is not detected in the above embodiment.

[0054] In the above embodiment, the area detection program is stored (installed) in the storage unit in advance, but the present invention is not limited to this. The program according to the disclosed technology can also be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory. [Explanation of symbols]

[0055] 10 Area detection device 12 Acquisition Department 14 Detection section 16 Interpolation section 20 Detection Model 301, 302, 303 Camera 401, 402, 403 Images 421, 422 2D-BB 50 Computer 51 CPU 52 Memory 53 Memory section 59 Storage medium 60 Area Detection Program

Claims

1. Acquiring images captured by a plurality of image capturing devices each capturing an image of a person from a different direction; Inputting the acquired images into a machine learning model that has been generated in advance by machine learning so as to detect areas showing people contained in the images, and detecting areas showing people from each of the acquired images; Interpolating an area showing the person in a second image among the acquired images based on an area showing the person detected in a first image among the acquired images and parameters of each of the plurality of image capturing devices. A region detection program for causing a computer to execute a process including the above.

2. the first image is an image in which a region showing the person is detected by the machine learning model, among the acquired images; The second image is an image in which an area showing the person is not detected by the machine learning model among the acquired images. The area detection program according to claim 1 .

3. The plurality of imaging devices are arranged in the same horizontal plane, A width of the area showing the person in the second image is estimated based on a height of the area showing the person detected from the first image, a height of the area showing the person in the second image specified based on parameters of each of the plurality of image capturing devices, and statistical information on the posture of the person. The area detection program according to claim 2 .

4. 4. The area detection program according to claim 3, further comprising: converting an end point of a vertical center line of an area representing the person detected from the first image into coordinates of an end point of a vertical center line of the person in three-dimensional space based on parameters of each of the photographing devices that captured the first image; and converting the converted coordinates in the three-dimensional space into coordinates on a second image based on parameters of the photographing device that captured the second image, thereby identifying a height of the area representing the person in the second image.

5. 5. The area detection program according to claim 4, wherein the statistical information on the person's posture is an average of the sums of heights and widths of a rectangular solid surrounding the person in each of a plurality of different postures taken by the person.

6. 6. The area detection program according to claim 5, wherein the length of a vertical center line of a person in the three-dimensional space is defined as the height of the person in the three-dimensional space, the difference between the average indicated by the statistical information and the height of the person in the three-dimensional space is estimated as the width of the person in the three-dimensional space, and the width of the area indicating the person in the second image is estimated based on the ratio of the height and width of the person in the three-dimensional space and the height of the area indicating the person in the second image.

7. the plurality of image capture devices are arranged in the same vertical plane; A height of the area showing the person in the second image is estimated based on a width of the area showing the person detected from the first image, a width of the area showing the person in the second image specified based on parameters of each of the plurality of image capturing devices, and statistical information on the posture of the person. The area detection program according to claim 2 .

8. an acquisition unit that acquires images captured by a plurality of image capture devices that capture images of a person from different directions; a detection unit that inputs the acquired images into a machine learning model that has been generated in advance by machine learning so as to detect an area showing a person included in the image, and detects an area showing a person from each of the acquired images; an interpolation unit that interpolates an area showing the person in a second image among the acquired images based on an area showing the person detected in a first image among the acquired images and parameters of each of the plurality of image capturing devices; An area detection device comprising:

9. Acquiring images captured by a plurality of image capturing devices each capturing an image of a person from a different direction; Inputting the acquired images into a machine learning model that has been generated in advance by machine learning so as to detect areas showing people contained in the images, and detecting areas showing people from each of the acquired images; Interpolating an area showing the person in a second image among the acquired images based on an area showing the person detected in a first image among the acquired images and parameters of each of the plurality of image capturing devices. The area detection method includes a process performed by a computer.

Images