Image processing device, image processing method and program
The image processing device analyzes panoramic and fisheye partial images separately and integrates their results to accurately estimate human behavior in fisheye images, addressing distortion issues and improving detection and estimation accuracy.
Patent Information
- Application Number
- JP2022551516
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-09-25
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-09-25
AI Technical Summary
Fisheye images suffer from distortion that affects the estimation of human behavior, leading to inaccurate results when input into standard human behavior estimation models, and generating panoramic images from fisheye images can cause further distortion and detection issues.
An image processing device that analyzes both panoramic and fisheye partial images separately and integrates the estimation results to accurately estimate human behavior in fisheye images, using methods like panorama expansion, fisheye partial image analysis, and integration processes.
The method enables high-accuracy estimation of human behavior in fisheye images by minimizing distortion and reducing processing load, improving detection and estimation accuracy.
Smart Images

Figure 0007589741000007 
Figure 0007589741000008 
Figure 0007589741000009
Abstract
Description
[Technical field]
[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]
[0002] Patent Document 1 discloses a technique for performing machine learning using training images and information for identifying business store locations. Patent Document 1 also discloses that panoramic images, images with a field of view greater than 180°, etc. can be used as training images.
[0003] Non-Patent Document 1 discloses a technology for estimating human behavior shown in a video image based on a 3D-CNN (convolutional neural network). [Prior art documents] [Patent documents]
[0004] [Patent Document 1] Special Publication No. 2018-524678 [Non-patent literature]
[0005] [Non-Patent Document 1] Kensho Hara and 2 others, "Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?", [online], Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (pp. 6546-6555), [Retrieved May 28, 2019], Internet <URL: http: / / openaccess.thecvf.com / content_cvpr_2018 / papers / Hara_Can_Spatiotemporal_3D_CVPR_2018_paper.pdf> Summary of the Invention [Problem to be solved by the invention]
[0006] A fisheye lens can capture a wide range. Taking advantage of these characteristics, fisheye lenses are widely used in surveillance cameras and the like. Therefore, the present inventors have investigated a technology for estimating human behavior based on an image generated using a fisheye lens (hereinafter, sometimes referred to as a "fisheye image").
[0007] In fisheye images, distortion occurs, so the direction of gravity may differ for each position in the image. This may result in unnatural situations, such as the direction in which a standing person's body stretches differing for each position in the image. Even if such fisheye images are input into a human behavior estimation model generated by machine learning based on images (learning data) generated using a standard lens (e.g., angle of view of approximately 40° to approximately 60°), sufficient estimation results cannot be obtained.
[0008] As a means for solving this problem, a method is considered in which a fisheye image is panoramic-expanded to generate a panoramic image, and the panoramic image is input to the above-mentioned human behavior estimation model. Here, an overview of panoramic expansion is explained with reference to FIG.
[0009] First, the reference line L s , reference point (x c , y c ), width w, and height h are determined. Reference line L s is the reference point (x c , y c ) and any point on the circumference of the circular image, and is the position where the fisheye image is cut out when the panorama is expanded. This reference line L s The neighboring images are located at the edge of the panoramic image. s There are various ways to define the reference point (x c , y c ) is a point of the image within the circular image circle of the fisheye image, for example the center of the circle. The width w is the width of the panoramic image, and the height h is the height of the panoramic image. These values may be default values or may be arbitrarily set by the user.
[0010] Once these values are determined, the panoramic expansion formula shown in the figure is used to find any target point (x f , y f ) to a point (x p , y p ) for any target point (x f , y f ), the reference point (x c , y c ) and the target point (x f , y f ) and the distance r f Similarly, the reference point (x c , y c ) and the target point (x f , y f ) and the reference line L s The angle θ between the two points can be calculated. As a result, the variables w, θ, h, and r in the "panoramic development" formula shown in the figure are f The values of and r are determined. Note that r is the radius of the image in the image circle. By substituting the values of these variables into the formula, the point (x p , y p ) can be calculated.
[0011] In addition, a panoramic image can be converted into a fisheye image using the "inverse panoramic development" formula shown in the figure.
[0012] It is true that by panorama-expanding a fisheye image to generate a panoramic image, it is possible to reduce unnaturalness such as the direction in which a standing person's body stretches differing depending on the position in the image. However, in the case of the above-mentioned panoramic expansion method, when generating a panoramic image from a fisheye image, the reference point (x c , y c ) is greatly stretched, so the reference point (x c , y c ) may be significantly distorted in the panoramic image. This can lead to problems in estimating human behavior based on the panoramic image, such as not being able to detect the distorted person or reduced estimation accuracy.
[0013] An object of the present invention is to estimate the behavior of a person included in a fisheye image with high accuracy. [Means for solving the problem]
[0014] According to the present invention, A first estimation means for performing image analysis on a panoramic image obtained by panoramic expansion of a fisheye image generated by a fisheye lens camera, and estimating a human behavior shown in the panoramic image; a second estimation means for performing image analysis on a fisheye partial image, which is a partial area of the fisheye image, without panoramic development, and estimating a human behavior indicated by the fisheye partial image; a third estimation means for estimating a human behavior indicated by the fisheye image based on an estimation result based on the panoramic image and an estimation result based on the fisheye partial image; An image processing apparatus is provided having the following:
[0015] Further, according to the present invention, The computer A panoramic image obtained by panoramic expansion of a fisheye image generated by a fisheye lens camera is subjected to image analysis, and human behavior shown in the panoramic image is estimated; A fisheye partial image, which is a part of the fisheye image, is subjected to image analysis without panoramic expansion, and human behavior indicated by the fisheye partial image is estimated; An image processing method is provided for estimating human behavior indicated by the fisheye image based on an estimation result based on the panoramic image and an estimation result based on the fisheye partial image.
[0016] Further, according to the present invention, Computer, a first estimation means for performing image analysis on a panoramic image obtained by panoramic expansion of a fisheye image generated by a fisheye lens camera, and estimating human behavior shown in the panoramic image; a second estimation means for performing image analysis of a fisheye partial image, which is a part of the fisheye image, without panorama development, and estimating a human behavior indicated by the fisheye partial image; a third estimation means for estimating a human behavior indicated by the fisheye image based on an estimation result based on the panoramic image and an estimation result based on the fisheye partial image; A program is provided to function as a Effect of the Invention
[0017] According to the present invention, it is possible to estimate the behavior of a person included in a fisheye image with high accuracy. [Brief description of the drawings]
[0018] The above objects, as well as other objects, features and advantages, will become more apparent from the following preferred embodiments and the accompanying drawings.
[0019] [Figure 1] FIG. 1 is a diagram for explaining a panoramic development technique. [Diagram 2] FIG. 1 is a diagram for explaining an overview of an image processing device according to an embodiment of the present invention. [Diagram 3] 1 is a diagram illustrating an example of an image processing apparatus and a hardware configuration of the processing apparatus according to an embodiment of the present invention. [Figure 4] FIG. 2 is a functional block diagram of an image processing apparatus according to an embodiment of the present invention; [Diagram 5] FIG. 2 is a diagram for explaining the processing of the image processing device of the present embodiment. [Figure 6] FIG. 2 is a diagram for explaining the processing of the image processing device of the present embodiment. [Figure 7] FIG. 2 is a diagram for explaining the processing of the image processing device of the present embodiment. [Figure 8] FIG. 2 is a diagram for explaining the processing of the image processing device of the present embodiment. [Figure 9] FIG. 2 is a diagram for explaining the processing of the image processing device of the present embodiment. [Figure 10] FIG. 2 is a diagram for explaining the processing of the image processing device of the present embodiment. [Figure 11] FIG. 2 is a diagram for explaining the processing of the image processing device of the present embodiment. [Figure 12] 5 is a flowchart showing an example of a processing flow of the image processing device of the present embodiment. [Figure 13]5 is a flowchart showing an example of a processing flow of the image processing device of the present embodiment. [Figure 14] 5 is a flowchart showing an example of a processing flow of the image processing device of the present embodiment. [Figure 15] 5 is a flowchart showing an example of a processing flow of the image processing device of the present embodiment. [Figure 16] FIG. 2 is a diagram for explaining the processing of the image processing device of the present embodiment. [Figure 17] FIG. 2 is a diagram for explaining the processing of the image processing device of the present embodiment. [Figure 18] FIG. 2 is a diagram for explaining the processing of the image processing device of the present embodiment. [Figure 19] FIG. 1 is an example of a block diagram of an image processing apparatus according to an embodiment of the present invention. [Figure 20] 5 is a flowchart showing an example of a processing flow of the image processing device of the present embodiment. [Figure 21] FIG. 2 is a diagram for explaining the processing of the image processing device of the present embodiment. [Figure 22] FIG. 2 is a diagram for explaining the processing of the image processing device of the present embodiment. [Figure 23] FIG. 2 is a diagram for explaining the processing of the image processing device of the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0020] <Summary> First, an overview of the image processing device 10 of this embodiment will be described with reference to FIG.
[0021] As shown in the figure, the image processing device 10 executes a panorama process, a fisheye process, and a merging process.
[0022] In the panorama process, the image processing device 10 performs image analysis on a panorama image obtained by panorama-expanding a fisheye image, and estimates the human behavior indicated by the panorama image. In the fisheye process, the image processing device 10 performs image analysis on a fisheye partial image, which is a partial area of the fisheye image, without panorama-expanding it, and estimates the human behavior indicated by the fisheye partial image. Then, in the integration process, the image processing device 10 estimates the human behavior indicated by the fisheye image based on the estimation result of the human behavior based on the panorama image obtained in the panorama process and the estimation result of the human behavior based on the fisheye partial image obtained in the fisheye process.
[0023] <Hardware configuration> Next, an example of the hardware configuration of the image processing device 10 will be described. Each functional unit of the image processing device 10 is realized by any combination of hardware and software, centering on a CPU (Central Processing Unit) of any computer, memory, programs loaded into the memory, a storage unit such as a hard disk for storing the programs (programs stored beforehand at the stage of shipping the device, as well as programs downloaded from storage media such as CDs (Compact Discs) or servers on the Internet, can be stored), and a network connection interface. Those skilled in the art will understand that there are various variations in the methods and devices for realizing the above.
[0024] FIG. 3 is a block diagram illustrating a hardware configuration of the image processing device 10. As shown in FIG. 3, the image processing device 10 has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The image processing device 10 does not need to have the peripheral circuit 4A. The image processing device 10 may be composed of a plurality of physically and / or logically separated devices, or may be composed of a single physically and / or logically integrated device. When the image processing device 10 is composed of a plurality of physically and / or logically separated devices, each of the plurality of devices can have the above hardware configuration.
[0025] The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to transmit and receive data to each other. The processor 1A is an arithmetic processing device such as a CPU or a GPU (Graphics Processing Unit). The memory 2A is a memory such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The input / output interface 3A includes an interface for acquiring information from an input device, an external device, an external server, an external sensor, a camera, etc., and an interface for outputting information to an output device, an external device, an external server, etc. Examples of the input device include a keyboard, a mouse, a microphone, a physical button, a touch panel, etc. Examples of the output device include a display, a speaker, a printer, a mailer, etc. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.
[0026] <Functional configuration> Next, the functional configuration of the image processing device 10 will be described. Fig. 4 shows an example of a functional block diagram of the image processing device 10. As shown in the figure, the image processing device 10 has a first estimation unit 11, a second estimation unit 12, and a third estimation unit 13. These functional units execute the above-mentioned panorama process, fisheye process, and integration process. Below, the configuration of each functional unit will be described for each process.
[0027] "Panorama Process" The panorama process is executed by the first estimation unit 11. FIG. 5 shows the flow of the panorama process in more detail. As shown in the figure, the first estimation unit 11 acquires a plurality of time-series fisheye images (fisheye image acquisition process), and then panorama-expands each of the images to generate a plurality of time-series panorama images (panorama expansion process). After that, the first estimation unit 11 estimates human behaviors indicated by the plurality of time-series panorama images based on the plurality of time-series panorama images and the first estimation model (first estimation process). Thus, the panorama process includes a fisheye image acquisition process, a panorama expansion process, and a first estimation process. Each of these will be described in detail below.
[0028] (Fisheye image acquisition process) In the fisheye image acquisition process, the first estimation unit 11 acquires a plurality of time-series fisheye images. The fisheye images are images generated using a fisheye lens. The plurality of time-series fisheye images may be, for example, a moving image, or may be a plurality of continuous still images generated by continuously capturing images at a predetermined time interval.
[0029] In this specification, "acquisition" may include "a device going to retrieve data stored in another device or storage medium (active acquisition)" based on user input or program instructions, such as receiving data by making a request or inquiry to another device, or accessing and reading out another device or storage medium. Also, "acquisition" may include "inputting data output from another device to the device (passive acquisition)" based on user input or program instructions, such as receiving data that is distributed (or transmitted, push notification, etc.). Also, "acquisition" may include selecting and acquiring data or information received, and "editing data (converting data to text, rearranging data, extracting part of data, changing file format, etc.) to generate new data and acquiring the new data."
[0030] (Panorama unfolding process) In the panoramic development process, the first estimation unit 11 performs panoramic development on each of a plurality of time-series fisheye images to generate a plurality of time-series panoramic images. An example of a method for panoramic development will be described below, but other methods may be adopted.
[0031] First, the first estimation unit 11 calculates the reference line L s , reference point (x c , y c ), width w, and height h (see Figure 1).
[0032] -Reference point(x c , y c ) Decision- First, the first estimation unit 11 detects a plurality of predetermined points on the body of each of the plurality of people from within the circular image circle of the fisheye image. Then, the first estimation unit 11 specifies the direction of gravity (vertical direction) at the position of each of the plurality of people based on the detected predetermined points.
[0033] For example, the first estimation unit 11 may detect a plurality of points (two points) on the body that are connected by a line parallel to the direction of gravity in an image generated by photographing a standing person from the front. Examples of such combinations of two points include, but are not limited to, (center of both shoulders, center of waist), (top of head, center of waist), (top of head, center of both shoulders), etc. In this example, the first estimation unit 11 specifies the direction from a predetermined one of the two points detected corresponding to each person to the other point as the direction of gravity.
[0034] As another example, the first estimation unit 11 may detect a plurality of body points (two points) whose connecting line is perpendicular to the direction of gravity in an image generated by photographing a standing person from the front. Examples of such a combination of two points include, but are not limited to, (right shoulder, left shoulder) and (right hip, left hip). In this example, the first estimation unit 11 specifies, as the direction of gravity, the direction in which a line extends that passes through the midpoint of the two points detected corresponding to each person and is perpendicular to the line connecting the two points.
[0035] The first estimation unit 11 can detect the above-mentioned multiple body points by using any image analysis technology. For example, the first estimation unit 11 can detect the multiple predetermined body points of each of the multiple people by analyzing a fisheye image with the same algorithm as "an algorithm for detecting multiple predetermined body points of each person present in an image generated using a standard lens (e.g., an angle of view of approximately 40° to approximately 60°)."
[0036] However, in a fisheye image, the direction in which the body of a standing person stretches may vary. 1 Alternatively, the first estimation unit 11 may perform analysis of the image while rotating the fisheye image. That is, the first estimation unit 11 may perform a process of rotating an image in an image circle of the fisheye image, and analyzing the image in the image circle after the rotation to detect a plurality of predetermined points on the person's body.
[0037] An overview of this process will be described with reference to Figures 6 to 9. In the example of Figure 6, five people M1 to M5 exist in an image C1 within an image circle of a fisheye image F. All five people M1 to M5 are standing, but the directions in which their bodies are stretched vary.
[0038] The first estimation unit 11 first analyzes the image in the rotated state shown in Fig. 6, and performs a process of detecting the center P1 of both shoulders and the center P2 of the waist of each person. In this case, the first estimation unit 11 was able to detect points P1 and P2 of persons M1 and M2 whose body stretch directions are close to the up-down direction in the figure, but was unable to detect points P1 and P2 of other persons.
[0039] Next, the first estimation unit 11 rotates the fisheye image F by 90 degrees. This results in the state shown in Fig. 7. The first estimation unit 11 analyzes the image in this rotated state and performs processing to detect the center P1 of both shoulders and the center P2 of the waist of each person. In this case, the first estimation unit 11 was able to detect points P1 and P2 of person M5, whose body stretches in a direction close to the up-down direction in the figure, but was unable to detect points P1 and P2 of the other people.
[0040] Next, the first estimation unit 11 rotates the fisheye image F by another 90°. This results in the state shown in Fig. 8. The first estimation unit 11 analyzes the image in this rotated state and performs processing to detect the center P1 of both shoulders and the center P2 of the waist of each person. In this case, the first estimation unit 11 was able to detect points P1 and P2 of person M4, whose body stretches in a direction close to the up-down direction in the figure, but was unable to detect points P1 and P2 of the other people.
[0041] Next, the first estimation unit 11 rotates the fisheye image F by another 90°. This results in the state shown in Fig. 9. The first estimation unit 11 analyzes the image in this rotated state and performs processing to detect the center P1 of both shoulders and the center P2 of the waist of each person. In this case, the first estimation unit 11 was able to detect points P1 and P2 of person M3, whose body stretches in a direction close to the up-down direction in the figure, but was unable to detect points P1 and P2 of the other people.
[0042] In this way, the first estimation unit 11 can detect a plurality of predetermined points on the body of each of a plurality of people whose body stretching directions are different by analyzing the image while rotating the fisheye image. Note that, although the above example shows the rotation by 90 degrees, this is merely an example and is not limiting.
[0043] Next, the first estimation unit 11 estimates a reference point (x c , y c Then, the first estimation unit 11 determines the determined reference point (x c , y c ) is stored in the storage unit of the image processing device 10.
[0044] When a line that passes through each of the positions of the plurality of people and is extended in the direction of gravity at each of the positions of the plurality of people intersects with another line at one point, the first estimation unit 11 determines the intersection point as a reference point (x c , y c )
[0045] On the other hand, if the straight lines passing through the positions of the plurality of people and extending in the direction of gravity at the positions of the plurality of people do not intersect at one point, the first estimation unit 11 determines a point whose distance from each of the plurality of straight lines satisfies a predetermined condition as a reference point (x c , y c )
[0046] When the first estimation unit 11 detects multiple points (two points) on the body in an image generated by photographing a standing person from the front, the line connecting them is parallel to the direction of gravity, the "straight line passing through the positions of each of the multiple people and extending in the direction of gravity at the positions of each of the multiple people" may be a line connecting the two points detected by the first estimation unit 11.
[0047] When the first estimation unit 11 detects multiple points (two points) on the body whose connecting line is perpendicular to the direction of gravity in an image generated by photographing a standing person from the front, the "straight line passing through the positions of each of the multiple people and extending in the direction of gravity at the positions of each of the multiple people" may be a line that passes through the midpoint of the two points detected by the first estimation unit 11 and is perpendicular to the line connecting the two points.
[0048] FIG. 10 shows the concept of the reference point determination process by the first estimation unit 11. In the illustrated example, the first estimation unit 11 detects the center P1 of both shoulders and the center P2 of the waist of each person. The lines connecting points P1 and P2 are "straight lines L1 to L5 that pass through the positions of each of the multiple people and extend in the direction of gravity at the positions of each of the multiple people." In the illustrated example, the multiple straight lines L1 to L5 do not intersect at a single point. For this reason, the first estimation unit 11 determines points whose distances from each of the multiple straight lines L1 to L5 satisfy a predetermined condition as reference points (x c , y c The predetermined condition is, for example, "the sum of the distances to each of the multiple straight lines is the smallest," but is not limited to this.
[0049] For example, the first estimation unit 11 can calculate a point that satisfies a predetermined condition based on the following expressions (1) to (3).
[0050]
number
[0051]
number
[0052]
number
[0053] First, the straight lines L1 to L5 are expressed by the formula (1). i is the slope of each line, and c i are the intercepts of each line. Using equations (2) and (3), the point at which the sum of the distances from each of the lines L1 to L5 is the smallest is defined as the reference point (x c , y c ) can be calculated as
[0054] In addition, when the installation position and orientation of the camera are fixed, the reference point (x c , y c ) will be in the same position. Therefore, the first estimation unit 11 estimates the reference point (x c , y c ) is calculated, the calculated reference point (x c , y c ) may be associated with the camera that generated the fisheye image and registered. Then, from then on, for fisheye images generated by that camera, the reference point (x c , y c ) is not calculated, and the registered reference point (x c , y c ) may be read and used.
[0055] -Image Complement- The first estimation unit 11 estimates the reference point (x c , y c If the reference point (x) is different from the center of the image in the image circle of the fisheye image, the image is interpolated to the image in the image circle of the fisheye image to generate a complemented circular image.c , y c ) coincides with the center of the image within the image circle of the fisheye image, the first estimation unit 11 does not perform the interpolation of that image.
[0056] The complementary circular image is an image obtained by adding a complementary image to the image in the image circle, and the reference point (x c , y c The complementary circular image is a circular image with the reference point (x c , y c ) to a point on the outer periphery of the image within the image circle becomes the radius, and the image within the image circle may be inscribed. The complementary image added to the image within the image circle may be a single color (e.g., black) image, an arbitrary pattern image, or other images.
[0057] FIG. 11 shows an example of a complementary circular image C2 generated by the first estimation unit 11. The complementary circular image C2 is generated by adding a black-only complementary image to an image C1 within the image circle of the fisheye image F. The complementary circular image C2 is circular as shown in the figure, and has a reference point (x c , y c ) is its center. The radius r of the complementary circular image C2 is the reference point (x c , y c ) to a point on the outer periphery of image C1 within the image circle. Note that image C1 within the image circle is inscribed in complementary circular image C2.
[0058] -Reference line L s Decision- Reference line L s is the reference point (x c , y c ) and any point on the circumference of the circular image (image in the image circle C1, complementary circular image C2, etc.). s The position of the reference line L that does not overlap with a person, for example, is the position at which the circular image is cut out when the image is panoramic-developed. s In this way, the reference line L sBy setting this, the inconvenience of a person being separated into two parts in a panoramic image can be suppressed.
[0059] Reference line L that does not overlap with people s There are various methods for setting the reference line L. For example, the first estimation unit 11 may set the reference line L within a predetermined distance from the multiple points on the body of each person detected in the above process. s Without setting, a reference line L is set at a point that is a certain distance or more away from the multiple points detected above. s You can set stomach.
[0060] -Determining width w and height h- The width w is the width of the panoramic image, and the height h is the height of the panoramic image. These values may be default values, or may be arbitrarily set by the user and registered in the image processing device 10.
[0061] - Panoramic deployment - Reference line L s , reference point (x c , y c After determining the reference point (x), width w, and height h, the first estimation unit 11 performs panoramic expansion of the fisheye image to generate a panoramic image. c , y c If the reference point (x) is different from the center of the image in the image circle of the fisheye image, the first estimation unit 11 panorama-expands the complementary circular image to generate a panoramic image. c , y c ) coincides with the center of the image in the image circle of the fisheye image, the first estimation unit 11 panorama-expands the image in the image circle of the fisheye image to generate a panorama image. The first estimation unit 11 can perform panorama expansion using the method described with reference to FIG.
[0062] Next, an example of the flow of the panoramic development process will be described. Since the details of each process have been described above, the description will be omitted here. First, a reference point (x c , y c An example of the flow of a process for determining
[0063] When the fisheye image is input, the first estimation unit 11 detects a plurality of predetermined points on the body of each of the plurality of people from within the image circle (S10). For example, the first estimation unit 11 detects the center P1 of both shoulders and the center P2 of the waist of each person.
[0064] Here, an example of the process flow of S10 will be described with reference to the flowchart of Fig. 13. First, the first estimation unit 11 analyzes the image within the image circle and detects a plurality of predetermined points on the body of each of a plurality of people (S20). After that, the first estimation unit 11 rotates the image within the image circle by a predetermined angle (S21). The predetermined angle is, for example, 90°, but is not limited to this.
[0065] Then, the first estimation unit 11 analyzes the image in the image circle after rotation and detects a predetermined number of points on the body of each of the people (S22). If the total of the rotation angles does not reach 360° (No in S23), the first estimation unit 11 returns to S21 and repeats the same process. On the other hand, if the total of the rotation angles reaches 360° (Yes in S23), the first estimation unit 11 ends the process.
[0066] 12, after S10, the first estimation unit 11 identifies the direction of gravity at the position of each of the plurality of people based on the predetermined plurality of points detected in S10 (S11). For example, the first estimation unit 11 identifies the direction from the center P1 of both shoulders of each person toward the center P2 of the waist as the direction of gravity at the position of each person.
[0067] Next, the first estimation unit 11 calculates a straight line that passes through the positions of each of the multiple people and extends in the direction of gravity at each position (S12). Then, if the multiple straight lines intersect at one point (Yes in S13), the first estimation unit 11 sets the intersection point as a reference point (x c , y c On the other hand, if the multiple lines do not intersect at one point (No in S13), the first estimation unit 11 finds a point whose distance from each of the multiple lines satisfies a predetermined condition (e.g., the shortest), and sets the point as the reference point (x c , y c) (S15).
[0068] Next, an example of the flow of a panoramic development process will be described with reference to the flowchart in FIG.
[0069] The reference point (x c , y c ) coincides with the center of the image within the image circle of the fisheye image (Yes in S30), the first estimation unit 11 panorama-expands the image within the image circle of the fisheye image by using the method described with reference to Fig. 1 to generate a panorama image (S33). That is, in this case, generation of a complementary circular image and panorama expansion of the complementary circular image are not performed.
[0070] On the other hand, the reference point (x c , y c If the center of the image in the image circle of the fisheye image does not match the center of the image in the image circle of the fisheye image (No in S30), the first estimation unit 11 generates a complementary circular image (S31). The complementary circular image is a circular image obtained by adding a complementary image to the image in the image circle, and the reference point (x c , y c ) is the image at the center of the circle. The complementary circular image is created by dividing the reference point (x c , y c ) to a point on the outer periphery of the image within the image circle becomes the radius, and the image within the image circle may be inscribed. The complementary image added to the image within the image circle may be a single color (e.g., black) image, an arbitrary pattern image, or other images.
[0071] Then, the first estimation unit 11 performs panoramic expansion on the complemented circular image by using the method described with reference to FIG. 1, to generate a panoramic image (S32).
[0072] (First estimation process) In the first estimation process, the first estimation unit 11 estimates human behavior indicated by the plurality of time-series panoramic images based on the generated plurality of time-series panoramic images and a first estimation model.
[0073] First, the first estimation unit 11 generates three-dimensional feature information indicating a change in features at each position in an image over time from a plurality of time-series panoramic images. For example, the first estimation unit 11 can generate the three-dimensional feature information based on a 3D CNN (e.g., a convolutional deep learning network such as 3D Resnet, but is not limited thereto).
[0074] In addition, the first estimation unit 11 generates person position information indicating the position where a person exists in each of the multiple time-series panoramic images. When multiple people exist in an image, the first estimation unit 11 can generate person position information indicating the position where each of the multiple people exists. For example, the first estimation unit 11 extracts a silhouette (whole body) of a person in an image, and generates person position information indicating an area in the image that contains the extracted silhouette. The first estimation unit 11 can generate person position information based on a deep learning technology, more specifically, based on a "deep learning network for object recognition" that recognizes any object (e.g., a person) from a flat image or video with high speed and high accuracy. Examples of deep learning networks for object recognition include Mask-RCNN, RCNN, Fast RCNN, Faster RCNN, etc., but are not limited to these. Note that the first estimation unit 11 may perform a similar person detection process on each of the multiple time-series panoramic images, or may use a person tracking technology to track a person once detected in the image and identify the position.
[0075] Thereafter, the first estimation unit 11 estimates the human behavior indicated by the multiple panoramic images based on the time change of the features indicated by the three-dimensional feature information at the position where the person indicated by the person position information exists. For example, the first estimation unit 11 can correct the three-dimensional feature information to change values at positions other than the position where the person indicated by the person position information exists to a predetermined value (e.g., 0), and then estimate the human behavior indicated by the multiple images based on the corrected three-dimensional feature information. The first estimation unit 11 can estimate the human behavior based on a first estimation model generated in advance by machine learning and the corrected three-dimensional feature information.
[0076] The first estimation model can be a model that estimates human behavior generated by machine learning based on an image (learning data) generated using a standard lens (for example, an angle of view of approximately 40° to approximately 60°). Alternatively, the first estimation model may be a model that estimates human behavior generated by machine learning based on a panoramic image (learning data) generated by panoramic expansion of a fisheye image.
[0077] Here, an example of the processing flow of the first estimation process will be described with reference to the flowchart in FIG.
[0078] First, the first estimation unit 11 executes the panoramic development process to obtain a plurality of time-series panoramic images (S40).
[0079] Then, the first estimation unit 11 generates three-dimensional feature information indicating a change in features at each position in the images over time from the multiple time-series panoramic images (S41). The first estimation unit 11 also generates person position information indicating a position where a person exists in each of the multiple panoramic images (S42).
[0080] Then, the first estimation unit 11 estimates human behaviors indicated by the multiple images based on a time change in the features indicated by the three-dimensional feature information at the position where the person indicated by the person position information exists (S43).
[0081] Next, a specific example of the first estimation process will be described with reference to Fig. 16. Note that this is merely an example and the present invention is not limited to this.
[0082] First, the first estimation unit 11 acquires, for example, 16 frames of time-series panoramic images (16×2451×800). Then, the first estimation unit 11 generates 3D feature information (512×77×25) convolved into 512 channels from the 16 frames of panoramic images based on a 3D CNN (for example, but not limited to, a convolutional deep learning network such as 3D Resnet). In addition, the first estimation unit 11 generates person position information (binary mask in the figure) indicating the position where a person exists in each of the 16 frames of images based on a deep learning network for object recognition such as Mask-RCNN. In the illustrated example, the person position information indicates the position of each of a plurality of rectangular areas containing each person.
[0083] Next, the first estimation unit 11 performs a correction on the three-dimensional feature information to change values at positions other than the positions where people are present, indicated by the person position information, to a predetermined value (e.g., 0). After that, the first estimation unit 11 divides the three-dimensional feature information into N blocks (each having a width of k), and obtains the probability (output value) that each of a plurality of predefined categories (human behaviors) is included for each block through average pooling, flattening, fully-connected layers, etc.
[0084] In the illustrated example, 19 categories are defined and learned. The 19 categories are, but are not limited to, "walking," "running," "waving," "picking up an object," "throwing an object away," "taking off a jacket," "putting on a jacket," "making a phone call," "using a smartphone," "eating a snack," "going up stairs," "going down stairs," "drinking water," "shaking hands," "taking an object from someone else's pocket," "handing an object to someone else," "pushing someone else," "swiping a card to enter a station," and "swiping a card to exit a station ticket gate." For example, the processing device 20 estimates that the image shows a human behavior corresponding to a category whose probability is equal to or exceeds a threshold value.
[0085] In the figure, N instance scores indicate the probability that each of N blocks included in multiple time-series panorama images includes each of the 19 categories. In addition, in the figure, Final scores of the panorama branch for clip 1 indicates the probability that multiple time-series panorama images include each of the 19 categories. Here, the details of the process of calculating Final scores of the panorama branch for clip 1 from N instance scores are not particularly limited, but an example will be described below.
[0086] In this calculation process, a function that returns a statistical value of multiple values can be used. For example, the average function (see formula (4)) that returns the average value, the max function (see formula (5)) that returns the maximum value, and the log-sum-exp function (see formula (6)) that smoothly approximates the max function can be used. These functions are widely known, so a description of them will be omitted here.
[0087]
number
[0088]
number
[0089]
number
[0090] By tracing in the opposite direction to the above flow, it is possible to calculate the position in the image indicating the category (human behavior) whose probability is equal to or greater than the threshold value.
[0091] "Fisheye process" The fisheye process is executed by the second estimation unit 12. As shown in FIG. 5, when the second estimation unit 12 acquires a plurality of time-series fisheye images (fisheye image acquisition process), it cuts out a part of an area from each of the images to generate a plurality of time-series fisheye partial images (first cut-out process). After that, the second estimation unit 12 edits the generated plurality of time-series fisheye partial images to generate a plurality of time-series edited fisheye partial images for each person included in the fisheye partial images (editing process). After that, the second estimation unit 12 estimates the person behavior indicated by the plurality of time-series edited fisheye partial images based on the plurality of time-series edited fisheye partial images and the second estimation model (second estimation process). Thus, the fisheye process includes a fisheye image acquisition process, a first cut-out process, an editing process, and a second estimation process. Each of these will be described in detail below.
[0092] (Fisheye image acquisition process) In the fisheye image acquisition process, the second estimation unit 12 acquires a plurality of time-series fisheye images. The fisheye image acquisition process executed by the second estimation unit 12 is similar to the fisheye image acquisition process executed by the first estimation unit 11 described in the panorama process, and therefore a description thereof will be omitted here.
[0093] (First cutting process) In the first extraction process, the second estimation unit 12 extracts a partial area from each of a plurality of time-series fisheye images to generate a plurality of time-series fisheye partial images. The second estimation unit 12 extracts a partial area from each of a plurality of time-series fisheye images to generate a plurality of time-series fisheye partial images. c , y c ) is used to extract an image within a circular region of radius R centered on the fisheye partial image. The radius R may be a preset fixed value. Alternatively, it may be a variable value determined based on the analysis results of the fisheye image. As an example of the latter, the radius R (size of the fisheye partial image) may be determined based on the detection results (number of people detected) of people existing in a preset central region within the fisheye image. The more people detected, the larger the radius R becomes.
[0094] (Editing Process) In the editing process, the second estimation unit 12 edits the generated multiple time-series fisheye partial images to generate multiple edited fisheye partial images in time series for each person included in the fisheye partial images. This will be described in detail below.
[0095] First, the second estimation unit 12 analyzes the fisheye partial image and detects a person included in the fisheye partial image. The person detection may adopt a method of detecting a person by rotating the fisheye partial image and analyzing the fisheye partial image at each rotation position, similar to the process described in the panorama process (processing in FIG. 13). Alternatively, a person included in the fisheye partial image may be detected based on a person detection model generated by machine learning using the fisheye image as learning data. The second estimation unit 12 may also perform a similar person detection process on each of a plurality of time-series fisheye partial images, or may use a person tracking technology to track a person once detected in a moving image and identify the position.
[0096] After detecting the persons, the second estimation unit 12 performs a rotation process to rotate the fisheye partial image and a second cut-out process to cut out a partial area of a predetermined size for each detected person to generate an edited fisheye partial image.
[0097] In the rotation process, the fisheye partial image is rotated so that the direction of gravity at the position of each person is the up-down direction on the image. The means for identifying the direction of gravity at the position of each person is as described in the panorama process, but other methods may also be used.
[0098] In the second extraction process, an image of a predetermined size including each person is extracted from the fisheye partial image after the rotation process. The shape and size of the extracted image are defined in advance.
[0099] Here, a specific example of the first extraction process and editing process will be described with reference to FIG.
[0100] First, as shown in (A)→(B), the second estimation unit 12 cuts out a partial area of an image C1 within an image circle of a fisheye image F as a fisheye partial image C3 (first cutting process). This process is executed for each fisheye image F.
[0101] Next, as shown in (B)→(C), the second estimation unit 12 detects people from within the fisheye partial image C3. In the illustrated example, two people are detected.
[0102] Next, as shown in (C)→(D), the second estimation unit 12 performs a rotation process on the fisheye partial image C3 for each detected person. As shown in the figure, in the fisheye partial image C3 after rotation, the direction of gravity at the position of each person becomes the up-down direction on the image. This process is performed for each fisheye partial image C3.
[0103] Next, as shown in (D)→(E), the second estimation unit 12 cuts out an image of a predetermined size including each person from the rotated fisheye partial image C3 for each detected person, and generates an edited fisheye partial image C4. This process is performed for each detected person and for each fisheye partial image C3.
[0104] (Second estimation process) In the second estimation process, the second estimation unit 12 estimates human behaviors indicated by the time-series edited fisheye partial images based on the generated time-series edited fisheye partial images and the second estimation model. The human behavior estimation process by the second estimation unit 12 is basically the same as the human behavior estimation process by the first estimation unit 11.
[0105] As shown in Fig. 18, the second estimation unit 12 generates three-dimensional feature information indicating a change in features at each position in an image over time from a plurality of time-series edited fisheye partial images corresponding to a first person. For example, the second estimation unit 12 can generate the three-dimensional feature information based on a 3D CNN (e.g., a convolutional deep learning network such as 3D Resnet, but is not limited thereto). After that, the second estimation unit 12 performs a process of emphasizing the value of the position where the person is detected for the generated three-dimensional feature information.
[0106] The second estimation unit 12 performs the process for each person detected from the fisheye partial image. Then, after linking the "3D feature information in which the value of the position where the person was detected is emphasized" calculated for each person, similar processes such as Average Pooling, flattening, and fully-connected layers are performed to obtain the probability (output value) that each of a plurality of predefined categories (human behaviors) is included in a plurality of time-series edited fisheye partial images corresponding to each person.
[0107] Then, the second estimation unit 12 integrates the probability that each of the multiple categories (human behavior) is included in each of the multiple edited fisheye partial images in a time series corresponding to each person, and performs a calculation to calculate the probability that each of the multiple categories (human behavior) is included in the fisheye partial image.
[0108] In this calculation process, a function that returns a statistical value of multiple values can be used, such as the average function (see formula (4) above) that returns the average value, the max function (see formula (5) above) that returns the maximum value, or the log-sum-exp function (see formula (6) above) that smoothly approximates the max function.
[0109] As is clear from the above description, the second estimation unit 12 performs image analysis on a fisheye partial image, which is a partial region of the fisheye image, without panoramic development, and estimates human behavior indicated by the fisheye partial image.
[0110] "Integration Process" The integration process is executed by the third estimation unit 13. As shown in Fig. 5, the third estimation unit 13 estimates human behavior indicated by the fisheye image based on an estimation result based on a panoramic image obtained in the panorama process and an estimation result based on a fisheye partial image obtained in the fisheye process.
[0111] As described above, both the estimation result based on the panoramic image and the estimation result based on the fisheye partial image indicate the probability that each of the predefined human actions is included. The third estimation unit 13 calculates the probability that the fisheye image includes each of the predefined human actions by a predetermined calculation process based on the estimation result based on the panoramic image and the estimation result based on the fisheye partial image.
[0112] In this calculation process, a function that returns a statistical value of multiple values can be used, such as the average function (see formula (4) above) that returns the average value, the max function (see formula (5) above) that returns the maximum value, or the log-sum-exp function (see formula (6) above) that smoothly approximates the max function.
[0113] <Example> Next, a description will be given of an example of the image processing device 10. Note that the example described here is an example of a case where the image processing device 10 of the present embodiment is implemented, and the invention is not limited to this example.
[0114] 19 is an example of a block diagram of the image processing device 10 of this embodiment. As described above, the basic configuration of the image processing device 10 is composed of a panorama process, a fisheye process, and an integration process. The basic configuration of each process is also as described above.
[0115] FIG. 20 is a flowchart showing the flow of processing in the image processing device 10 of this embodiment.
[0116] In S101, the image processing device 10 divides a plurality of input time-series fisheye images into a plurality of clips, each of which has a predetermined number of images. A specific example is shown in Fig. 21. In the illustrated example, 120 time-series fisheye images are input and divided into eight clips. Each clip contains 16 fisheye images, and only the last clip contains eight fisheye images. After that, the fisheye process (S102 to S108), the panorama process (S109 to S115), and the integration process (S116) are performed for each clip.
[0117] Details of the fisheye process (S102 to S108) are shown in Fig. 17 and Fig. 18. In the fisheye process, the image processing device 10 extracts a partial area of each of a plurality of time-series fisheye images F to generate a plurality of time-series fisheye partial images C3 (S102, Fig. 17 (A) → (B)). After that, the image processing device 10 detects a person from the plurality of time-series fisheye partial images C3 and tracks the person within the video (S103, Fig. 17 (B) → (C)).
[0118] Next, the image processing device 10 executes a rotation process (FIG. 17(C)→(D)) on the fisheye partial image C3 for each detected person, and a process (FIG. 17(D)→(E)) of cutting out an image of a predetermined size including each person from the rotated fisheye partial image C3 (S104). As a result, multiple edited fisheye partial images C4 in chronological order are obtained for each detected person.
[0119] In the subsequent S105, the image processing device 10 inputs each of the multiple edited fisheye partial images in time series for each detected person to a 3D CNN (for example, a convolutional deep learning network such as 3D Resnet, but is not limited to this) to generate three-dimensional feature information, as shown in Fig. 18. In addition, the image processing device 10 performs a process of emphasizing the value of the position where the person is detected for the generated three-dimensional feature information.
[0120] Next, the image processing device 10 connects the 3D feature information obtained for each person (S106). After that, the image processing device 10 obtains the probability (output value) that each of a plurality of predefined categories (human behaviors) is included in a plurality of time-series edited fisheye partial images corresponding to each person through average pooling, flattening, fully-connected layers, etc. (S107).
[0121] After that, the image processing device 10 performs a calculation to integrate the probability that each of the multiple categories (human behavior) is included in each of the multiple edited fisheye partial images in time series corresponding to each person, and calculate the probability that each of the multiple categories (human behavior) is included in the multiple fisheye partial images in time series (S108). In this calculation process, a function that returns a statistical value of multiple values can be used. For example, the average function (see formula (4) above) that returns the average value, the max function (see formula (5) above) that returns the maximum value, the log-sum-exp function (see formula (6) above) that smoothly approximates the max function, and the like can be used.
[0122] The details of the panorama process (S109 to S115) are shown in FIG. 16. In the panorama process, the image processing device 10 performs panoramic expansion on each of a plurality of time-series fisheye images (S109), and then generates three-dimensional feature information (512×77×25) convolved into 512 channels from the plurality of time-series panorama images based on a 3D CNN (for example, but not limited to, a convolutional deep learning network such as 3D Resnet) (S110). In addition, the image processing device 10 generates person position information indicating the position where a person exists in each of a plurality of time-series panorama images based on a deep learning network for object recognition such as Mask-RCNN (S112).
[0123] Next, the image processing device 10 performs a correction on the three-dimensional feature information generated in S110 to change values at positions other than the positions where people are present, which are indicated by the person position information generated in S112, to predetermined values (e.g., 0) (S111).
[0124] Then, the image processing device 10 divides the 3D feature information into N blocks (each having a width of k) (S113), and obtains the probability (output value) that each of a plurality of predefined categories (human behaviors) is included for each block through average pooling, flattening, fully-connected layers, etc. (S114).
[0125] After that, the image processing device 10 performs a calculation to integrate the probability that each of the multiple categories (human behavior) obtained for each block is included, and calculates the probability that each of the multiple categories (human behavior) is included in multiple time-series panoramic images (S115). In this calculation process, a function that returns a statistical value of multiple values can be used. For example, the average function (see formula (4) above) that returns the average value, the max function (see formula (5) above) that returns the maximum value, the log-sum-exp function (see formula (6) above) that smoothly approximates the max function, and the like can be used.
[0126] After that, the image processing device 10 integrates the "probability that each of the multiple categories (human behavior) is included in the multiple time-series fisheye partial images" obtained in the fisheye process and the "probability that each of the multiple categories (human behavior) is included in the multiple time-series panoramic images" obtained in the panoramic process, and performs a calculation to calculate the probability that each of the multiple categories (human behavior) is included in the multiple time-series fisheye images included in each clip (S116, see FIG. 22). In this calculation process, a function that returns a statistical value of multiple values can be used. For example, the average function that returns the average value (see formula (4) above), the max function that returns the maximum value (see formula (5) above), the log-sum-exp function that smoothly approximates the max function (see formula (6) above), etc. can be used.
[0127] By performing the above process for each clip, the "probability that each of the multiple categories (human behavior) is included in the multiple time-series fisheye images included in each clip" is obtained for each clip. In S117, the multiple "probabilities that each of the multiple categories (human behavior) is included in the multiple time-series fisheye images included in each clip" obtained for each clip are integrated, and a calculation is performed to calculate the "probability that each of the multiple categories (human behavior) is included in the 120 input time-series fisheye images" (see FIG. 22). In this calculation process, a function that returns a statistical value of multiple values can be used. For example, the average function that returns the average value (see formula (4) above), the max function that returns the maximum value (see formula (5) above), the log-sum-exp function that smoothly approximates the max function (see formula (6) above), etc. can be used.
[0128] Thereafter, the image processing device 10 outputs the calculation result (S118) and identifies the position of the human behavior predicted to be included (S119).
[0129] In the learning stage, as shown in FIG. 22, the image processing device 10 applies a sigmoid function to convert the probability that each of the multiple categories (human behaviors) is included in the 120 input time-series fisheye images into a value between 0 and 1. S The learning is performed to optimize the value of the number.
[0130] <Modification> "First Variation" Fig. 23 shows the flow of the modified example. As is clear from a comparison with Fig. 5, the modified example differs from the embodiment described above in the configuration of the panorama process. The panorama process of the modified example will be described in detail below.
[0131] First, the first estimation unit 11 performs image analysis to calculate a first estimation result of human behavior shown in a plurality of time-series panoramic images. This processing is the same as the processing of the panoramic process described in the above embodiment.
[0132] The first estimation unit 11 also performs image analysis on the optical flow image generated from the panoramic image to calculate a second estimation result of the human behavior indicated by the panoramic image. The optical flow image is an image of a vector representing the movement of an object in a plurality of time-series panoramic images. In the "process of estimating the human behavior indicated by a plurality of time-series panoramic images" described in the above embodiment, the calculation of the second estimation result is realized by replacing the "plurality of time-series panoramic images" with the "plurality of time-series optical flow images."
[0133] Then, the first estimation unit 11 estimates human behaviors shown in the multiple time-series panoramic images based on the first estimation result and the second estimation result. This estimation result is integrated with the estimation result obtained by the fisheye process.
[0134] In integrating the first and second estimation results, a function that returns a statistical value of multiple values can be used, such as the average function (see formula (4) above) that returns the average value, the max function (see formula (5) above) that returns the maximum value, or the log-sum-exp function (see formula (6) above) that smoothly approximates the max function.
[0135] "Second Variation" In the above embodiment, the image processing device 10 generates a panoramic image, a fisheye partial image, and an edited fisheye partial image, but at least one of these processes may be performed by a device other than the image processing device 10. An image generated by the other device (at least one of the panoramic image, the fisheye partial image, and the edited fisheye partial image) may be input to the image processing device 10. In this case, the image processing device 10 performs the above-mentioned processes using the input image.
[0136] "Third Variation" In the panorama process, a process may be performed on the generated panorama image to eliminate information on a portion corresponding to a partial area extracted in the fisheye process (hereinafter, "that portion") (e.g., making that portion a single color or a predetermined pattern). Then, human behavior may be estimated based on the panorama image after the process and the first estimation model. Since the human behavior included in that portion is estimated in the fisheye process, the information on that portion can be eliminated from the panorama image. However, if there is a person straddling that portion and another portion, a situation may occur in which the estimation accuracy of the human behavior deteriorates. For this reason, it is preferable to execute the process without eliminating the information on that portion from the panorama image, as in the above embodiment.
[0137] "Fourth Variation" In the editing process of the embodiment described above, the second estimation unit 12 analyzes the fisheye partial image to detect a person included in the fisheye partial image. As a modified example of the "process of detecting a person included in a fisheye partial image", the second estimation unit 12 may perform the following process. First, the second estimation unit 12 analyzes the fisheye image to detect a person included in the fisheye image. Then, the second estimation unit 12 detects a person whose detection position (coordinates) in the fisheye image satisfies a predetermined condition (within an area cut out as the fisheye partial image) among the people detected from the fisheye image. The process of detecting a person from a fisheye image is realized by an algorithm similar to the algorithm of the process of detecting a person from the fisheye partial image described above. According to this modified example, the detection accuracy of a person included in a fisheye partial image is improved.
[0138] <Action and effect> As a first comparative example of this embodiment, a process of estimating the human behavior of a person included in a fisheye image by performing only the panorama process without performing the fisheye process and the integration process can be considered.
[0139] However, as mentioned above, when generating a panoramic image from a fisheye image, the reference point (x c , y c ) is greatly stretched, so the reference point (x c , y c) may be significantly distorted in the panoramic image. Therefore, in the case of the first comparative example, problems may occur such as not being able to detect the distorted person or a decrease in estimation accuracy.
[0140] As a second comparative example of this embodiment, a process can be considered in which the panorama process and integration process are not executed, but the entire fisheye image is processed without panoramic expansion, in a manner similar to the fisheye process described above, to estimate the human behavior of a person included in the fisheye image.
[0141] However, when a fisheye image contains many people, the number of images to be generated and processed becomes huge, and the processing load on the computer becomes large. When processing similar to the fisheye process described above is performed, people included in the fisheye image are detected, and the orientation of each person in the image is adjusted to generate multiple images (corresponding to edited fisheye partial images), which are then processed to estimate the human behavior of each of the multiple people. Naturally, the more people are detected, the more images to be generated and processed become huge.
[0142] The image processing device 10 of this embodiment can solve these problems. The image processing device 10 of this embodiment estimates human behavior by analyzing a panoramic image and a reference point (x c , y c The human behavior of a person in a fisheye image is estimated by integrating the human behavior estimated by analyzing a portion of the image around the fisheye image without panoramic expansion.
[0143] Reference point of fisheye image (x c , y c When a part of the image near the reference point (x c , y c ) does not cause the problem of people near the reference point (x c , y c ) and can accurately estimate the behavior of the person. That is, the problem in the first comparative example can be solved.
[0144] In addition, the fisheye image reference point (x c , y c ) is analyzed without panoramic expansion, and the other parts are excluded from the processing. This reduces the number of people detected in the fisheye process. As a result, compared to the second comparative example, the number of images generated and processed in the fisheye process (edited fisheye partial images) is reduced, and the processing burden on the computer can be reduced.
[0145] Although the present invention has been described above with reference to the embodiments (and examples), the present invention is not limited to the above-mentioned embodiments (and examples). Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.
[0146] A part or all of the above-described embodiments can be described as follows, but is not limited to the following. 1. A first estimation means for performing image analysis on a panoramic image obtained by panoramic expansion of a fisheye image generated by a fisheye lens camera, and estimating human behavior shown in the panoramic image; a second estimation means for performing image analysis on a fisheye partial image, which is a partial area of the fisheye image, without panoramic development, and estimating a human behavior indicated by the fisheye partial image; a third estimation means for estimating a human behavior indicated by the fisheye image based on an estimation result based on the panoramic image and an estimation result based on the fisheye partial image; An image processing device comprising: 2. The second estimation means The image processing device described in 1, wherein the fisheye partial image is an image within a circular area centered on a reference point in the fisheye image determined based on the direction of gravity at the position of each of multiple people present in the fisheye image. 3. The image processing device according to claim 2, wherein the direction of gravity at the position of each of a plurality of people present in the fisheye image is identified based on a plurality of predetermined points on the body of each of the plurality of people. 4. The second estimation means 4. The image processing device according to any one of 1 to 3, wherein a size of the fisheye partial image is determined based on a result of detection of a person present in the fisheye image. 5. The second estimation means rotating the fisheye partial image and extracting a partial area of a predetermined size to generate an edited fisheye partial image for each person detected in the fisheye partial image; 5. The image processing device according to any one of 1 to 4, further comprising: analyzing the edited fisheye partial image to estimate a human behavior indicated by the fisheye partial image. 6. The estimation result based on the panoramic image and the estimation result based on the fisheye partial image each indicate a probability of including each of a plurality of predefined human behaviors; The third estimation means calculates the probability that the fisheye image includes each of the predefined human behaviors by a predetermined calculation process based on the estimation result based on the panoramic image and the estimation result based on the fisheye partial image. 7. The first estimation means performing image analysis on the panoramic image to calculate a first estimation result of a human behavior indicated by the panoramic image; performing image analysis on an optical flow image generated from the panoramic image to calculate a second estimation result of a human behavior indicated by the panoramic image; 7. The image processing device according to any one of 1 to 6, wherein a human behavior shown in the panoramic image is estimated based on the first estimation result and the second estimation result. 8. The computer: A panoramic image obtained by panoramic expansion of a fisheye image generated by a fisheye lens camera is subjected to image analysis, and human behavior shown in the panoramic image is estimated; A fisheye partial image, which is a part of the fisheye image, is subjected to image analysis without panoramic expansion, and human behavior indicated by the fisheye partial image is estimated; An image processing method for estimating human behavior indicated by the fisheye image based on an estimation result based on the panoramic image and an estimation result based on the fisheye partial image. 9. Computer a first estimation means for performing image analysis on a panoramic image obtained by panoramic expansion of a fisheye image generated by a fisheye lens camera, and estimating human behavior shown in the panoramic image; a second estimation means for performing image analysis of a fisheye partial image, which is a part of the fisheye image, without panorama development, and estimating a human behavior indicated by the fisheye partial image; a third estimation means for estimating a human behavior indicated by the fisheye image based on an estimation result based on the panoramic image and an estimation result based on the fisheye partial image; A program that functions as a
Claims
1. a first estimation means for inputting a plurality of time-series panoramic images obtained by panoramic expansion of a plurality of time-series fisheye images generated by a fisheye lens camera into a first estimation model that estimates the behavior of a person included in an image that has been generated in advance by machine learning, and obtaining a probability that the panoramic image output from the first estimation model shows each of a plurality of human behaviors; a second estimation means for not panoramic-expanding a fisheye partial image, which is a partial area of the fisheye image, but rotating the fisheye partial image for each person detected in the fisheye partial image to set the orientation of the person to a predetermined reference orientation, then cutting out an area including the person to generate an edited fisheye partial image, inputting the edited fisheye partial image into a second estimation model that estimates the behavior of the person included in an image previously generated by machine learning, and obtaining a probability that the fisheye partial image output from the second estimation model shows each of a plurality of human behaviors; a third estimation means for calculating, for each human behavior, a statistical value of a probability that the human behavior is shown in the panoramic image and a probability that the human behavior is shown in the fish-eye partial image, and estimating the human behavior shown in the fish-eye image based on the statistical value; having The first estimation means includes: determining, when a plurality of straight lines passing through the positions of each of the plurality of persons in the panoramic image and extending in the direction of gravity at the positions of each of the plurality of persons intersect at a single point, the point of intersection as a reference point; and, when the plurality of straight lines do not intersect at a single point, determining, as the reference point, a point at which a sum of distances from each of the plurality of straight lines is minimum; The second estimation means includes: The image within a circular region having the reference point as a center is defined as the fisheye partial image. The reference orientation is an image processing device in which the direction of gravity at the position of each person is the up-down direction on the image.
2. The first estimation means includes: Detecting a plurality of body points that are connected to each other by a line parallel to the direction of gravity when the person is standing, or a plurality of body points that are connected to each other by a line perpendicular to the direction of gravity when the person is standing; When multiple points on the body of a person standing are detected that are connected to each other in a line parallel to the direction of gravity, the direction connecting the multiple points is determined to be the direction of gravity; 2. The image processing device according to claim 1, wherein when a plurality of points on the body of a person standing are detected such that a line connecting the plurality of points is perpendicular to the direction of gravity, the direction perpendicular to the line connecting the plurality of points is determined to be the direction of gravity.
3. The second estimation means includes: The image processing device according to claim 1 , wherein the size of the fish-eye partial image is increased as the number of people detected in a preset central area in the fish-eye image increases.
4. The computer a first estimation step of inputting a plurality of time-series panoramic images obtained by panoramic expansion of a plurality of time-series fisheye images generated by a fisheye lens camera into a first estimation model that estimates the behavior of a person included in the image generated in advance by machine learning, and obtaining a probability that the panoramic image output from the first estimation model shows each of a plurality of human behaviors; a second estimation step of: rotating a fisheye partial image, which is a part of the fisheye image, for each person detected in the fisheye partial image without panoramic expansion, so that the orientation of the person is set to a predetermined reference orientation, and then extracting an area including the person to generate an edited fisheye partial image; inputting the edited fisheye partial image into a second estimation model that estimates the behavior of the person included in an image previously generated by machine learning; and obtaining a probability that the fisheye partial image output from the second estimation model shows each of a plurality of human behaviors; a third estimation step of calculating, for each human behavior, a statistical value of a probability that the human behavior is represented by the panoramic image and a probability that the human behavior is represented by the fish-eye partial image, and estimating the human behavior represented by the fish-eye image based on the statistical value; Run In the first estimation step, determining, when a plurality of straight lines passing through the positions of each of the plurality of persons in the panoramic image and extending in the direction of gravity at the positions of each of the plurality of persons intersect at a single point, the point of intersection as a reference point; and, when the plurality of straight lines do not intersect at a single point, determining, as the reference point, a point at which a sum of distances from each of the plurality of straight lines is minimum; In the second estimation step, The image within a circular region having the reference point as a center is defined as the fisheye partial image. The image processing method according to the present invention, wherein the reference orientation is an orientation in which the direction of gravity at the position of each person is the up-down direction on the image.
5. Computer, a first estimation means for inputting a plurality of time-series panoramic images obtained by panorama-expanding a plurality of time-series fisheye images generated by a fisheye lens camera into a first estimation model that estimates the behavior of a person included in the image that has been generated in advance by machine learning, and obtaining a probability that the panoramic image output from the first estimation model shows each of a plurality of human behaviors; a second estimation means for not panorama-expanding a fisheye partial image, which is a part of the fisheye image, but rotating the fisheye partial image for each person detected in the fisheye partial image to set the orientation of the person to a predetermined reference orientation, then cutting out an area including the person to generate an edited fisheye partial image, inputting the edited fisheye partial image into a second estimation model that estimates the behavior of the person included in an image previously generated by machine learning, and obtaining a probability that the fisheye partial image output from the second estimation model shows each of a plurality of human behaviors; a third estimation means for calculating, for each human behavior, a statistical value of a probability that the human behavior is represented by the panoramic image and a probability that the human behavior is represented by the fish-eye partial image, and estimating the human behavior represented by the fish-eye image based on the statistical value; Function as a The first estimation means includes: determining, when a plurality of straight lines passing through the positions of each of the plurality of persons in the panoramic image and extending in the direction of gravity at the positions of each of the plurality of persons intersect at a single point, the point of intersection as a reference point; and, when the plurality of straight lines do not intersect at a single point, determining, as the reference point, a point at which a sum of distances from each of the plurality of straight lines is minimum; The second estimation means includes: The image within a circular region having the reference point as a center is defined as the fisheye partial image. The program in which the reference orientation is an orientation in which the direction of gravity at the position of each person is the up-down direction on the image.
Citation Information
Patent Citations
Imaging device, recording device and video output control device
JP2016178406A
Image processing system, information processing apparatus, information terminal, and program
JP2017162432A
Business discovery from images
JP2018524678A
Autonomous store tracking system
JP2020053019A
