Image processing device of a human detection system

Abstract

An image processing device of a person detection system suppresses a decrease in detection accuracy of a person. The person detection system is installed in a forklift. The person detection system includes a stereo camera and the image processing device. The image processing device acquires first image data and second image data from the stereo camera. The image processing device detects a rectangular region in which an obstacle is present from the first image data. The image processing device estimates a barycentric position of the obstacle included in the rectangular region in the first image data, and performs barycentric region width adjustment processing of adjusting a width of the rectangular region according to the barycentric position. The image processing device determines whether the obstacle present in the rectangular region is a person by comparing an adjusted region after the barycentric region width adjustment processing is performed with dictionary data. The dictionary data is data of a feature quantity extracted from image data in which a person is present.

Description

Technical Field

[0001] This disclosure relates to an image processing apparatus for a human detection system. Background Technology

[0002] An obstacle detection device is installed on a moving body such as a vehicle to detect obstacles. Patent Document 1 discloses an obstacle detection device comprising a camera and an image processing unit. The image processing unit acquires image data from the camera. Based on the image data acquired from the camera, the image processing unit detects regions in the image data where obstacles are present. The image processing unit performs human detection processing on the regions where obstacles are present. Human detection processing may be performed, for example, using HOG (Histogram of Oriented Gradients) features.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2020-135616 Summary of the Invention

[0006] The problem the invention aims to solve

[0007] The accuracy of person detection may decrease depending on the region detected by the image processing device from the image data and the location of the person contained in that region.

[0008] Solution for solving the problem

[0009] The image processing device for solving the above-mentioned problems in a human detection system is an image processing device installed on a moving body. The image processing device detects areas containing obstacles in image data obtained from a camera, estimates the center of gravity position of the obstacle contained in the area in the image data, performs a center of gravity region width adjustment process based on the center of gravity position, and determines whether the obstacle in the area is a person by comparing the adjusted area after the center of gravity region width adjustment process with dictionary data.

[0010] If the center position of a region in the width direction of the image data deviates significantly from the center position of the obstacle, the accuracy of person detection decreases. By comparing the adjusted region (after adjusting the width of the center region) with the dictionary data, the decrease in person detection accuracy can be suppressed.

[0011] For the image processing device of the aforementioned human detection system, the dictionary data may include upper body comparison data and full-body comparison data. The image processing device determines whether the upper body detection processing condition of the obstacle being free from the ground within a predetermined range from the camera is met for each region. For the region where the upper body detection processing condition is met, the adjusted region after the center-of-gravity region width adjustment processing of the region in the image data is compared with the upper body comparison data to determine whether the obstacle presented in the region is the upper body of a person. For the region where the upper body detection processing condition is not met, the full-body detection processing is performed by comparing the region in the image data with the full-body comparison data to determine whether the obstacle presented in the region is the full-body of a person.

[0012] For the image processing device of the above-mentioned human detection system, the centroid region width adjustment processing can also be a process of setting the adjusted region width as the length obtained by adding the length obtained by making the distance from the centroid position to the end closer to the centroid position in the width direction of the region in the image data twice, plus the value based on the difference between the centroid position and the center position in the width direction of the region.

[0013] For the image processing device of the aforementioned human detection system, the image processing device may also perform width adjustment processing on the adjusted region. The width adjustment processing may be at least one of the following: increasing the width of the adjusted region in the image data when the width of the adjusted region is below a predetermined lower limit value, and decreasing the width of the adjusted region in the image data when the width of the adjusted region is above a predetermined upper limit value.

[0014] For the image processing device of the above-mentioned human detection system, the image processing device may also detect the height of the upper end of the region from the ground, and the upper body detection processing includes processing to correct the height direction dimension of the adjusted region in the image data based on the height of the upper end of the region from the ground.

[0015] Invention Effects

[0016] According to the present invention, it is possible to suppress the decrease in the detection accuracy of people. Attached Figure Description

[0017] Figure 1 This is a side view of the forklift.

[0018] Figure 2 This is a schematic diagram of the forklift and personnel detection system.

[0019] Figure 3 This is a flowchart illustrating the obstacle detection process performed by the image processing device.

[0020] Figure 4 This is a diagram showing an example of the first image data.

[0021] Figure 5 This is a flowchart illustrating the person detection process performed by the image processing device.

[0022] Figure 6 This diagram is used to illustrate the whole-body detection process.

[0023] Figure 7 This is a flowchart illustrating the upper body detection process performed by the image processing device.

[0024] Figure 8 This is a diagram showing a rectangular area.

[0025] Figure 9 This is a graph showing the correspondence between the width of a rectangular region in the world coordinate system and the first correction rate.

[0026] Figure 10 This is a diagram showing the adjusted area.

[0027] Figure 11 This is a graph showing the correspondence between the width of the adjusted region in the world coordinate system and the second correction rate.

[0028] Figure 12 This is a diagram showing the adjusted region in the world coordinate system with the width exceeding the upper limit.

[0029] Figure 13 This is a diagram showing the adjusted region in the world coordinate system where the width is below the lower limit.

[0030] Figure 14 This is a diagram showing the correspondence between the height of the upper end of the rectangular area and the correction amount.

[0031] Figure 15 This diagram is used to illustrate the detection and processing of the upper body.

[0032] Figure 16 This is a flowchart illustrating the character detection process for a change example.

[0033] Explanation of reference numerals in the attached figures

[0034] B1, B2… as rectangular regions, B21… adjusted region, C1… center position, D1… full-body dictionary data, D2… upper-body dictionary data, G1… center of gravity position, IM1… first image data, LL2… lower limit, UL2… upper limit, 10… forklift as a moving body, 30… person detection system, 31… stereo camera as a camera, 41… image processing device. Detailed Implementation

[0035] An embodiment of the image processing apparatus for a human detection system will be described.

[0036] <Forklift>

[0037] like Figure 1 As shown, the forklift 10, as a mobile vehicle, includes a body 11, drive wheels 12, steering wheels 13, and a cargo loading / unloading device 17. The body 11 includes a overhead guard 14 and a counterweight 15. The overhead guard 14 is located above the driver's seat. The counterweight 15 is located at the rear of the body 11. The counterweight 15 is a component used to balance the cargo loaded on the cargo loading / unloading device 17. The forklift 10 can be a forklift operated by a rider, an automatically operating forklift, or a forklift capable of switching between manual and automatic operation. The forklift 10 is an example of an industrial vehicle.

[0038] like Figure 2As shown, the forklift 10 includes a control unit 20, a travel motor M11, a travel control unit 23 for controlling the travel motor M1, and a speed sensor 24. The control unit 20 performs controls related to travel and loading / unloading operations. The control unit 20 includes a processor 21 and a storage unit 22. The processor 21 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), or a digital signal processor (DSP). The storage unit 22 includes random access memory (RAM) and read-only memory (ROM). The storage unit 22 stores programs for operating the forklift 10. In other words, the storage unit 22 stores program code or instructions configured to cause the processor 21 to perform processing. The storage unit 22, i.e., the computer-readable medium, includes all usable media accessible by a general-purpose or special-purpose computer. The control device 20 may also be constructed from hardware circuits such as application-specific integrated circuits (ASICs) and field-programmable gate arrays (FPGAs). As a processing circuit, the control device 20 may include one or more hardware circuits such as processors, ASICs, or FPGAs, or combinations thereof, that operate according to a computer program.

[0039] The control device 20 provides the travel control device 23 with a command regarding the rotational speed of the travel motor M11, so that the forklift 10's speed becomes the target speed. In this embodiment, the travel control device 23 is a motor driver. The speed sensor 24 outputs the rotational speed of the travel motor M11 to the travel control device 23. The travel control device 23 controls the travel motor M11 based on the command from the control device 20, so that the rotational speed of the travel motor M11 matches the command.

[0040] <Human Detection System>

[0041] A person detection system 30 is installed on the forklift 10. The person detection system 30 includes a stereo camera 31 (which functions as a camera) and an image processing device 41. The person detection system 30 detects people present around the forklift 10. In addition to detecting people, the person detection system 30 can also detect obstacles that are different from people. The image processing device 41 is the image processing unit of the person detection system 30.

[0042] <3D Camera>

[0043] like Figure 1 As shown, the stereo camera 31 is configured to view the ground on which the forklift 10 travels from above. The stereo camera 31 is, for example, mounted on the overhead guard 14. The stereo camera 31 captures images of the area behind the forklift 10. Therefore, the person detected by the person detection system 30 is the person behind the forklift 10. The stereo camera 31 captures images within a range determined by the horizontal and vertical viewing angles. The counterweight 15 is included within the vertical viewing angle range. Therefore, the counterweight 15, which is part of the forklift 10, is always included in the image captured by the stereo camera 31.

[0044] like Figure 2 As shown, the stereo camera 31 includes a first camera 32 and a second camera 33. For example, cameras using a CCD image sensor or a CMOS image sensor can be used as the first camera 32 and the second camera 33. The first camera 32 and the second camera 33 are configured with their optical axes parallel to each other. Image data captured by the first camera 32 is designated as first image data, and image data captured by the second camera 33 is designated as second image data.

[0045] <Image Processing Device>

[0046] The image processing apparatus 41 includes a processor 42 and a storage unit 43. The processor 42 may be, for example, a CPU, GPU, or DSP. The storage unit 43 includes RAM and ROM. The storage unit 43 stores various programs for detecting obstacles based on images captured by the stereo camera 31. In essence, the storage unit 43 stores program code or instructions configured to cause the processor 42 to perform processing. The storage unit 43, i.e., the computer-readable medium, includes all usable media accessible by a general-purpose or special-purpose computer. The image processing apparatus 41 may also be constructed from hardware circuits such as ASICs or FPGAs. The image processing apparatus 41, as a processing circuit, may include one or more hardware circuits such as processors, ASICs, or FPGAs, or combinations thereof, that operate according to a computer program.

[0047] <Dictionary Data>

[0048] Storage unit 43 stores full-body dictionary data D1 and upper-body dictionary data D2. Full-body dictionary data D1 is dictionary data used for detecting people. Full-body dictionary data D1 is, for example, data containing feature quantities extracted from multiple known image data presenting a person. In this embodiment, full-body dictionary data D1 is dictionary data obtained from image data presenting a person's entire body. As feature quantities, HOG (Histogram of Oriented Gradients) feature quantities and CoHOG (Co-occurrence HOG) feature quantities can be listed, for example. In this embodiment, HOG feature quantities are used as feature quantities. HOG feature quantities are histograms of the gradient intensity of each gradient direction of the pixel values ​​in a cell of image data. A cell is a local region of a predetermined size. When calculating HOG feature quantities, the image data is divided into multiple cells. Furthermore, a histogram of the gradient direction of the pixel values ​​is calculated for each cell, and normalization is performed within a predetermined range surrounding the cell. Thus, HOG feature quantities can be obtained. Upper body dictionary data D2 is dictionary data obtained by extracting features from image data showing the upper body of a person. Upper body dictionary data D2 can be obtained by performing the same processing on image data showing the upper body of a person as when obtaining full body dictionary data D1. If the image processing device 41 is equipped with an auxiliary storage device, at least one of full body dictionary data D1 and upper body dictionary data D2 may be stored in the auxiliary storage device.

[0049] Obstacle Detection and Handling

[0050] The obstacle detection process performed by the image processing device 41 will be described. The obstacle detection process is performed by executing a program stored in the storage unit 43 by the processor 42. The obstacle detection process is repeated at a predetermined control cycle.

[0051] like Figure 3 As shown, in step S1, the image processing device 41 obtains the first image data and the second image data of the same frame from the image captured by the stereo camera 31.

[0052] Next, in step S2, the image processing device 41 obtains a disparity image by performing stereo processing. A disparity image is an image that maps disparity [px] to pixels. Disparity is obtained by comparing first image data and second image data and calculating the difference in the number of pixels between the first and second image data for the same feature point in each image data. Feature points refer to identifiable boundaries such as the edges of obstacles. Feature points can be detected from information such as brightness.

[0053] The image processing apparatus 41 uses RAM to temporarily store each image data to perform conversion from RGB to YCrCb. The image processing apparatus 41 can also perform distortion correction, edge enhancement processing, etc. The image processing apparatus 41 performs stereo processing to calculate disparity by comparing the similarity of each pixel in the first image data with that in the second image data. Furthermore, as stereo processing, a method for calculating disparity per pixel can be used, or a block matching method can be used to divide each image data into blocks containing multiple pixels and calculate the disparity of each block. The image processing apparatus 41 obtains a disparity image using the first image data as a reference image and the second image data as a comparison image. The image processing apparatus 41 extracts the most similar pixel from the second image data for each pixel of the first image data, and calculates the disparity as the difference in the horizontal pixel count between a pixel in the first image data and the pixel most similar to that pixel. Thus, a disparity image that corresponds to the disparity with each pixel of the first image data, which serves as the reference image, can be obtained. The disparity image does not necessarily need to be displayed; rather, it represents data that corresponds the disparity to each pixel in the disparity image. In addition, the image processing device 41 can also perform processing to remove the parallax of the ground from the parallax image.

[0054] Next, in step S3, the image processing device 41 derives the coordinates of the feature points in the world coordinate system. First, the image processing device 41 derives the coordinates of the feature points in the camera coordinate system. The camera coordinate system is a coordinate system with the stereo camera 31 as the origin. The camera coordinate system is a three-axis orthogonal coordinate system with the optical axis as the Z-axis and the two axes orthogonal to the optical axis as the X-axis and Y-axis, respectively. The coordinates of the feature points in the camera coordinate system can be represented by the Z-coordinate Zc, X-coordinate Xc, and Y-coordinate Yc in the camera coordinate system. The Z-coordinate Zc, X-coordinate Xc, and Y-coordinate Yc can be derived using the following equations (1) to (3), respectively.

[0055]

Mathematical Formula 1

[0056]

[0057]

Mathematical Formula 2

[0058]

[0059]

Mathematical Expression 3

[0060]

[0061] In equations (1) to (3), B is the baseline length [mm], f is the focal length [mm], and d is the parallax [px]. xp is any X coordinate in the parallax image, and x′ is the X coordinate of the center coordinate of the parallax image. yp is any Y coordinate in the parallax image, and y′ is the Y coordinate of the center coordinate of the parallax image.

[0062] By setting xp as the X coordinate of a feature point in the parallax image, yp as the Y coordinate of a feature point in the parallax image, and d as the parallax corresponding to the coordinates of the feature point, the coordinates of the feature point in the camera coordinate system can be derived.

[0063] Here, with the forklift 10 positioned horizontally, a three-axis orthogonal coordinate system is established as the world coordinate system, serving as the coordinate system in actual space. This system comprises the X-axis (extending horizontally along the width of the forklift 10), the Y-axis (extending horizontally in a direction orthogonal to the X-axis), and the Z-axis (extending horizontally in a direction orthogonal to both the X and Y axes). The Y-axis of the world coordinate system can also be described as an axis extending in the forward / backward direction of the forklift 10, which is its direction of travel. The Z-axis of the world coordinate system can also be described as an axis extending vertically. The coordinates of feature points in the world coordinate system can be represented using the X-coordinate (Xw), Y-coordinate (Yw), and Z-coordinate (Zw) of the world coordinate system.

[0064] The image processing device 41 performs a world coordinate transformation to convert camera coordinates to world coordinates using the following equation (4). World coordinates refer to coordinates in the world coordinate system.

[0065]

Mathematical Expression 4

[0066]

[0067] Here, H in equation (4) is the height of the stereo camera 31 in the world coordinate system [mm], and θ is the angle +90° between the optical axes of the first camera 32 and the second camera 33 and the horizontal plane.

[0068] In this embodiment, the origin of the world coordinate system is defined by X-coordinates Xw and Y-coordinates Yw for the position of the stereo camera 31, and Z-coordinates Zw for the coordinates of the ground. The position of the stereo camera 31 is, for example, the midpoint between the lens of the first camera 32 and the lens of the second camera 33.

[0069] The X-coordinate Xw obtained through world coordinate transformation represents the distance from the origin to the feature point relative to the width of the forklift 10. The Y-coordinate Yw represents the distance from the origin to the feature point relative to the travel direction of the forklift 10. The Z-coordinate Zw represents the height from the ground to the feature point. A feature point is a point representing a portion of an obstacle. In the following description, the Y-axis refers to the Y-axis of the world coordinate system. Figure 1 The arrow Y represents the Y-axis of the world coordinate system, and the arrow Z represents the Z-axis of the world coordinate system.

[0070] Next, in step S4, the image processing device 41 extracts obstacles existing in the world coordinate system. The image processing device 41 treats a set of feature points representing a portion of an obstacle, assuming they represent the same obstacle, as a point cluster, and extracts this point cluster as an obstacle. For example, the image processing device 41 performs clustering based on the world coordinates of the feature points derived in step S3, treating feature points within a predetermined range as a point cluster. The image processing device 41 then treats the clustered point cluster as an obstacle. The clustering of feature points in step S4 can be performed using various methods. That is, clustering can be performed using any method as long as it allows treating multiple feature points as a point cluster and considering them as obstacles.

[0071] Next, in step S5, the image processing device 41 extracts the location of the obstacle extracted in step S4. The image processing device 41 can identify the world coordinates of the obstacle based on the world coordinates of the feature points constituting the clustered point group. For example, the X-coordinates (Xw), Y-coordinates (Yw), and Z-coordinates (Zw) of multiple feature points located at the edges of the clustered point group can be set as the X-coordinates (Xw), Y-coordinates (Yw), and Z-coordinates (Zw) of the obstacle. Alternatively, the X-coordinates (Xw), Y-coordinates (Yw), and Z-coordinates (Zw) of the feature point that will become the center of the point group can also be set as the X-coordinates (Xw), Y-coordinates (Yw), and Z-coordinates (Zw) of the obstacle. That is, the coordinates of the obstacle in the world coordinate system can represent the obstacle as a whole or a single point within the obstacle.

[0072] Next, in step S6, the image processing device 41 detects the position of the obstacle in the first image data. The position of the obstacle in the first image data can be detected from the parallax image. The position of the obstacle in the first image data is represented by a region showing the range in the first image data. In this embodiment, the region is a rectangular region. The rectangular region is the region containing the obstacle in the first image data. The first image data is image data acquired from the stereo camera 31, and is image data for detecting the region containing the obstacle. The image processing device 41 performs a mapping between the rectangular region and the world coordinates of the obstacle derived in step S6. For example, the image processing device 41 can map the world coordinates of the obstacle to the rectangular region by converting the world coordinates of the obstacle to camera coordinates, and then converting the camera coordinates to coordinates of the first image data. That is, the image processing device 41 can obtain the world coordinates of the rectangular region. The Z-coordinate Zw of the rectangular region represents the height above the ground. It can be said that the image processing device 41 detects the height of the rectangular region above the ground. The height of the rectangular region above the ground includes the height of the lower end of the rectangular region above the ground and the height of the upper end of the rectangular region above the ground.

[0073] In the following explanation, the position of an obstacle refers to its position in the world coordinate system, i.e., world coordinates. The position of an obstacle can also be described as the position of a rectangular region. The position of an obstacle in the first image data refers to its position in the image coordinate system. The image coordinate system is a coordinate system that represents the pixel position of the first image data, with the horizontal axis as the X-axis and the vertical axis as the Y-axis. The position of an obstacle in the first image data can also be described as the position of a rectangular region in the first image data. In the following explanation, the X-coordinate of the image coordinate system will be appropriately represented as X-coordinate Xi, and the Y-coordinate will be represented as Y-coordinate Yi.

[0074] <Person Detection and Processing>

[0075] The image processing device 41 performs person detection processing. The person detection processing is performed on a per-rectangular region basis. Below, as an example, the obstacle detection processing is described as follows: Figure 4 The situation of rectangular regions B1 and B2 is shown below for explanation. If we can... Figure 4 As understood, the first image data IM1 contains a figure M1 showing its full body and a figure M2 whose lower body is hidden by weight 15. If it is possible to determine... Figure 4 As understood, the Y-coordinate Yi of the lower end of the rectangular regions B1 and B2 in the first image data IM1 represents the lowest point of the obstacle in the first image data IM1. The Y-coordinate Yi of the upper end of the rectangular regions B1 and B2 in the first image data IM1 represents the highest point of the obstacle in the first image data IM1.

[0076] like Figure 5 As shown, in step S11, the image processing device 41 determines whether rectangular regions B1 and B2 meet the upper body detection processing conditions. The upper body detection processing conditions are used to determine whether an obstacle presented in the first image data IM1 is presented in the first image data IM1 while being obscured by a shading object such as the counterweight 15. Specifically, the upper body detection processing conditions are used to classify obstacles that may represent a person whose upper body is presented in the first image data IM1 but whose lower body is hidden by a shading object, and other obstacles. If both the distance condition and the free-floating condition are met, the image processing device 41 determines that rectangular regions B1 and B2 meet the upper body detection processing conditions.

[0077] Distance condition... The distance to the obstacle is within a predetermined distance from the rear end of the forklift 10.

[0078] Freeing conditions... The obstacle has been freed from the ground at a predetermined height.

[0079] like Figure 1As shown, the distance condition can be determined based on the distance to the obstacle in the Y-axis direction. The distance to the obstacle refers to the distance from the rear end of the forklift 10 to the center position of the rectangular areas B1 and B2 in the Y-axis direction. The distance L1 from the stereo camera 31 to the rear end of the forklift 10 in the Y-axis direction is known. By pre-storing the distance L1 from the stereo camera 31 to the rear end of the forklift 10 or the Y coordinate Yw of the rear end of the forklift 10 in the storage unit 43, the image processing device 41 can derive the distance L2 from the rear end of the forklift 10 to the rectangular areas B1 and B2 in the Y-axis direction. Furthermore, if the distance L2 is within a predetermined distance, the image processing device 41 determines that the distance condition is met. The predetermined distance of the distance condition is set, for example, based on the resolution of the stereo camera 31 and dictionary data D1 and D2. The higher the resolution of the stereo camera 31, the more feature data can be obtained, and therefore the longer the predetermined distance can be. The farther the obstacle is from the stereo camera 31, the smaller the size of the obstacle presented in the first image data IM1. Therefore, the distance at which a person can be detected varies depending on the number of pixels in the image data used when obtaining dictionary data D1 and D2. The predetermined distance can also vary based on this distance. The Y-coordinate Yw of the obstacle is its distance from the stereo camera 31. Therefore, it can also be said that the distance condition determines whether the distance from the stereo camera 31 to the obstacle is within the predetermined distance.

[0080] The detachment condition can be determined based on the Z-coordinate Zw of the lower end of the rectangular regions B1 and B2. The origin of the Z-coordinate Zw is the ground. Therefore, it can be said that the Z-coordinate Zw of the lower end of the rectangular regions B1 and B2 represents the height H1 of the obstacle detached from the ground. If we can determine this based on... Figure 1 As understood, when an obstacle is hidden by the counterweight 15, the area below the obstacle becomes a blind spot in the field of view of the stereo camera 31. When the area below the obstacle is hidden by the counterweight 15, the obstacle appears to be free from the ground in the first image data IM1. The predetermined height is determined, for example, by the height of the stereo camera 31 and the position of the counterweight 15. For example, if an obstacle exists within a predetermined distance, the predetermined height can be set to the height of the area obscured by the counterweight 15. Alternatively, the predetermined height can be a value that varies according to the Y-coordinate Yw of the rectangular areas B1 and B2. In this case, the predetermined height can be lowered as the rectangular areas B1 and B2 move further away from the forklift 10.

[0081] The upper body detection and processing conditions can be described as the conditions under which an obstacle is freed from the ground within a predetermined range starting from the stereo camera 31. Figure 4 In the example shown, rectangular region B1 does not meet the upper body detection processing conditions. Rectangular region B2 meets the upper body detection processing conditions.

[0082] If the determination result in step S11 is negative, that is, if the obstacle does not meet the conditions for upper body detection processing, the image processing device 41 performs the processing in step S12. Taking the rectangular region B1 as an example, step S12 will be explained.

[0083] In step S12, the image processing device 41 performs a full-body detection process. For example... Figure 6 As shown, the full-body detection process is performed by comparing a rectangular region B1 corresponding to an obstacle that does not meet the upper-body detection processing conditions with the full-body dictionary data D1. The image processing device 41 extracts features from the rectangular region B1. The feature extraction of the rectangular region B1 is performed using the same method as the feature extraction performed when obtaining the full-body dictionary data D1. In this embodiment, HOG features are extracted from the rectangular region B1. Furthermore, by performing pattern matching and machine learning-based person detection based on the HOG features extracted from the rectangular region B1 and the full-body dictionary data D1, it is determined whether the obstacle in the rectangular region B1 is a person. When the determination in step S12 is completed, the image processing device 41 ends the person detection process. The full-body dictionary data D1 is full-body comparison data.

[0084] like Figure 5 As shown, if the determination result in step S11 is positive, that is, if the obstacle meets the conditions for upper body detection processing, the image processing device 41 performs upper body detection processing S20. Taking the rectangular region B2 as an example, the upper body detection processing S20 will be explained.

[0085] like Figure 7 As shown, in step S21, the image processing device 41 adjusts the width of the rectangular region B2 in the first image data IM1.

[0086] like Figure 8 As shown, the image processing device 41 estimates the centroid position G1 of the obstacle contained in the rectangular region B2. The centroid position G1 of the obstacle can be estimated, for example, from a parallax image. The image processing device 41 can estimate the centroid position G1 from the distribution of parallax. The image processing device 41 estimates the centroid position G1 of the obstacle as a location where the same parallax is concentrated in the Y-axis direction of the image coordinate system. Feature points appear at locations where obstacles exist. Therefore, the centroid position G1 can be estimated by using the distribution of parallax. In the case where the obstacle is a person, the centroid position G1 is likely to be the position of the person's head.

[0087] The image processing apparatus 41 adjusts the width W1 of the rectangular region B2 in the first image data IM1 according to the centroid position G1. The width W1 of the rectangular region B2 in the first image data IM1 refers to the dimension in the X-axis direction of the image coordinate system. In the following description, the X-axis direction of the image coordinate system is sometimes referred to as the width direction, and the Y-axis direction of the image coordinate system is referred to as the height direction. The image processing apparatus 41 changes the width W1 of the rectangular region B2 to the width W11 derived from the following equation (5).

[0088] Width W11 = Distance W2 × 2 + Difference ΔG × First Correction Rate…(5)

[0089] Distance W2 is the distance from the centroid G1 of rectangular region B2 to the end of the region closest to the centroid G1 in the width direction. Figure 8 In the example shown, the distance from the centroid position G1 to the right end of the rectangular region B2 is distance W2. Distance W2 is the difference between the X-coordinate Xi of the centroid position G1 and the X-coordinate Xi of the right end of the rectangular region B2. The difference ΔG is the distance between the center position C1 and the centroid position G1 in the width direction of the rectangular region B2. The difference ΔG is the difference between the X-coordinate Xi of the centroid position G1 and the X-coordinate Xi of the center position C1. The first correction factor is determined by the width of the rectangular region B2 in the world coordinate system. The smaller the width of the rectangular region B2 in the world coordinate system, the smaller the difference ΔG. Therefore, to compensate for the correction effect of the difference ΔG, the smaller the width of the rectangular region B2 in the world coordinate system, the larger the value of the first correction factor.

[0090] like Figure 9 As shown, the width of the rectangular region B2 in the world coordinate system corresponds to the first correction rate. Figure 9 In the example shown, an upper limit width UL1 and a lower limit width LL1 are set for the width of the rectangular region B2 in the world coordinate system. Furthermore, the upper limit width UL1 and the lower limit width LL1 are each associated with a first correction rate. The upper limit width UL1 is 950 [mm]. Values ​​less than 1 are used as the first correction rate and associated with the upper limit width UL1. The first correction rate, as associated with the upper limit width UL1, can be arbitrarily set, for example, from a range of 0.1 to 0.5. Figure 9 In the example shown, 0.2 is used as the first correction rate and corresponds to the upper limit width UL1. The lower limit width LL1 is 450 [mm]. Values ​​greater than 1 are used as the first correction rate and correspond to the lower limit width LL1. The first correction rate corresponding to the lower limit width LL1 can be arbitrarily set, for example, from a range of 1.5 to 2.2. Figure 9 In the example shown, 1.8 is used as the first correction rate and is associated with the lower limit width LL1.

[0091] Image processing device 41 derives a first correction rate based on the width of rectangular region B2 in the world coordinate system. If the width of rectangular region B2 in the world coordinate system is greater than or equal to the upper limit width UL1, image processing device 41 derives the value corresponding to the upper limit width UL1 as the first correction rate. If the width of rectangular region B2 in the world coordinate system is less than or equal to the lower limit width LL1, image processing device 41 derives the value corresponding to the lower limit width LL1 as the first correction rate. If the width of rectangular region B2 in the world coordinate system is between the upper limit width UL1 and the lower limit width LL1, image processing device 41 derives the first correction rate based on the value corresponding to the upper limit width UL1 and the value corresponding to the lower limit width LL1. For example, image processing device 41 can derive the first correction rate based on a linear function represented by the upper limit width UL1, the lower limit width LL1, the value corresponding to the upper limit width UL1, and the value corresponding to the lower limit width LL1.

[0092] like Figure 10 As shown, by adjusting the width through step S21, the adjusted region B21 can be obtained. The adjusted region B21 is the rectangular region B2 after the width adjustment. The width W11 of the adjusted region B21 is the width derived from equation (5). Figure 10 The example shown is the adjusted region B21 with a first correction rate of 1.0. The adjusted region B21 can also be described as the region obtained by extracting a portion of width W11 from one end of the rectangular region B2 closer to the centroid G1 and towards the end farther from the centroid G1. In this embodiment, the region obtained by extracting a portion of width W11 from the right end of the rectangular region B2 towards the left end is the adjusted region B21. Step S21 is to set the width W11 of the adjusted region B21 as the length obtained by adding the length obtained by doubling the distance W2 from the centroid G1 to the end closer to the centroid G1 in the width direction within the rectangular region B2, based on the difference ΔG between the centroid G1 and the center position C1 in the width direction of the rectangular region B2. In this embodiment, the value based on the difference ΔG refers to the value obtained by multiplying the difference ΔG by the first correction rate. The adjustment of the width of the rectangular region B2 in step S21 is performed to reduce the deviation between the center position C1 of the rectangular region B2 in the first image data IM1 and the center of gravity position G1 of the obstacle presented in the rectangular region B2. The process in step S21 is the center of gravity region width adjustment process.

[0093] like Figure 7As shown, next, in step S22, the image processing device 41 adjusts the width of the adjusted region B21 in the first image data IM1. The width of the adjusted region B21 in the first image data IM1 refers to the dimension in the X-axis direction of the image coordinate system. It can be said that the width adjustment in step S22 is performed on the rectangular region B2 after the centroid region width adjustment processing. The adjustment of the width of the adjusted region B21 in step S22 is performed with the aim of reducing the difference between the width of the upper body dictionary data D2 and the width of the adjusted region B21. The upper body dictionary data D2 is dictionary data obtained from image data presenting the upper body of a person. Therefore, the width of the upper body dictionary data D2 is the same as the width of the image data for which feature extraction has been performed. In contrast, the width of the adjusted region B21 can vary depending on the size of the obstacle, the condition of the obstacle, and the positional relationship between the obstacle and the stereo camera 31. The image processing device 41 adjusts the width of the adjusted region B21 when it is too large or too small. Therefore, the difference between the width of the upper body dictionary data D2 and the width of the adjusted region B21 is reduced. If the width of the adjusted region B21 in the world coordinate system is below a predetermined lower limit, the image processing device 41 increases the width of the adjusted region B21. If the width of the adjusted region B21 in the world coordinate system is above a predetermined upper limit, the image processing device 41 decreases the width of the adjusted region B21.

[0094] like Figure 11 As shown, the upper limit UL2 and lower limit LL2 of the width of the adjusted region B21 in the world coordinate system each correspond to a second correction rate. The upper limit UL2 is 950 [mm]. Values ​​less than 1 are used as the second correction rate and associated with the upper limit UL2. The second correction rate associated with the upper limit UL2 can be arbitrarily set, for example, from a range of 0.6 to 0.9. Figure 11 In the example shown, 0.8 is used as the second correction rate, corresponding to the upper limit value UL2. The lower limit value LL2 is 450 [mm]. Values ​​greater than 1 are used as the second correction rate, corresponding to the lower limit value LL2. The second correction rate corresponding to the lower limit value LL2 can be arbitrarily set, for example, from a range of 1.1 to 1.3. Figure 11 In the example shown, 1.2 is used as the second correction rate and mapped to the lower limit value LL2. In another example, 1.0 is used as the second correction rate and mapped to the value between the lower limit value LL2 and the upper limit value UL2.

[0095] The image processing apparatus 41 derives a second correction rate based on the width of the adjusted region B21 in the world coordinate system. When the width of the adjusted region B21 in the world coordinate system is above the upper limit value UL2, the image processing apparatus 41 derives a value corresponding to the upper limit value UL2 as the second correction rate. When the width of the adjusted region B21 in the world coordinate system is below the lower limit value LL2, the image processing apparatus 41 derives a value corresponding to the lower limit value LL2 as the second correction rate. When the width of the adjusted region B21 in the world coordinate system is between the upper limit value UL2 and the lower limit value LL2, the image processing apparatus 41 derives 1.0 as the second correction rate.

[0096] The image processing apparatus 41 sets the value obtained by multiplying the width of the adjusted region B21 by the second correction rate as the new width of the adjusted region B21. When the second correction rate is 1.0, it can be said that the width of the adjusted region B21 is not adjusted. The second correction rate can be multiplied by the width of the adjusted region B21 in the first image data IM1. The second correction rate can also be multiplied by the width of the adjusted region B21 in the world coordinate system. In this case, the width of the adjusted region B21 in the first image data IM1 is adjusted according to the width of the adjusted region B21 in the world coordinate system. That is, as long as the width of the adjusted region B21 in the first image data IM1 can be adjusted, the width can be adjusted through any processing. Figure 10 In the case of the adjusted region B21 shown, the width in the world coordinate system is a value between the upper limit UL2 and the lower limit LL2. Therefore, the width of the adjusted region B21 in the first image data IM1 is not adjusted. The process in step S22 is a width adjustment process.

[0097] Assume that the result obtained through step S21 is Figure 12 The adjusted region B22 is shown. The width of the adjusted region B22 in the world coordinate system is greater than or equal to the upper limit UL2. For example, a person with outstretched arms can be considered as an example where the width of the adjusted region B22 in the world coordinate system is greater than or equal to the upper limit UL2. The image processing device 41 adjusts the width W21 of the adjusted region B22 in the first image data IM1 to a width W22 obtained by multiplying the width W21 by 0.8. The image processing device 41 adjusts the width in a way that the center position C1 of the adjusted region B22 in the width direction does not change before and after the adjustment.

[0098] Assume that the result obtained through step S21 is Figure 13The adjusted region B22 is shown. The width of the adjusted region B22 in the world coordinate system is below the lower limit value LL2. For example, a case where the width of the adjusted region B22 in the world coordinate system is below the lower limit value LL2 can be cited as a case where a person is photographed from the side and the person's torso and arms are aligned. The image processing device 41 adjusts the width W21 of the adjusted region B22 in the first image data IM1 to a width W22 obtained by multiplying the width W21 by 1.2. The image processing device 41 adjusts the width in such a way that the center position C1 of the adjusted region B22 in the width direction does not change before and after the adjustment of the width of the adjusted region B22.

[0099] like Figure 7 As shown, next, in step S23, the image processing device 41 corrects the height of the adjusted region B21 in the first image data IM1. Specifically, the image processing device 41 corrects the position of the lower end of the adjusted region B21 in the first image data IM1, thereby extracting a predetermined length of portion from the upper end of the adjusted region B21 towards the lower end. When the obstacle presented in the adjusted region B21 is a person, the image processing device 41 corrects the adjusted region B21 so that the lower end of the adjusted region B21 in the first image data IM1 is positioned approximately at the chest of the person. When the obstacle presented in the adjusted region B21 is a person, the upper end of the adjusted region B21 in the first image data IM1 is positioned approximately at the top of the person's head. It can be said that the dimension of the adjusted region B21 in the Y-axis direction of the image coordinate system, i.e., the vertical width of the adjusted region B21 in the first image data IM1, is corrected to a range approximately from the chest to the top of the head of the person.

[0100] The distance from a person's chest to the top of their head varies depending on their height. The taller the person, the larger this distance tends to be. For a person 1850 mm tall, the chest is generally 600 mm below the top of the head. For a person 1700 mm tall, the chest is generally 540 mm below the top of the head. For a person 1500 mm tall, the chest is generally 430 mm below the top of the head. This allows for a general assessment of the chest position based on height, enabling adjustments to the height of area B21 based on the position of the top of the head.

[0101] like Figure 14As shown, the height of the upper end of the adjusted region B21 corresponds to the correction amount. The Z-coordinate Zw of the upper end of the adjusted region B21 is the same as the Z-coordinate Zw of the upper end of the rectangular region B2. Therefore, it can also be said that the height of the upper end of the rectangular region B2 corresponds to the correction amount. The correction amount shows the range of the rectangular region B2 in the world coordinate system from the reference to the height of the upper end of the adjusted region B21 downwards. For example, when the height of the upper end of the adjusted region B21 is equivalent to 1850 [mm], the correction amount is 600 [mm]. In this case, 1250 [mm], which is 600 [mm] downwards from 1850 [mm], becomes the height of the lower end of the adjusted region B21. And, the range from 1850 [mm] to 1250 [mm] becomes the range of the adjusted region B21 in the world coordinate system. In this embodiment, three predetermined points P1 to P3 are set for the height of the upper end of the adjusted region B21, and these predetermined points P1 to P3 are mapped to correction values. The three predetermined points P1 to P3 include a first predetermined point P1, a second predetermined point P2, and a third predetermined point P3. The first predetermined point P1 is 1850 mm. The second predetermined point P2 is 1700 mm. The third predetermined point P3 is 1500 mm. 600 mm is used as the correction value and mapped to the first predetermined point P1. 540 mm is used as the correction value and mapped to the second predetermined point P2. 430 mm is used as the correction value and mapped to the third predetermined point P3.

[0102] The image processing device 41 derives a correction amount based on the Z-coordinate Zw of the upper end of the adjusted region B21. If the Z-coordinate Zw of the upper end of the adjusted region B21 corresponds to the height of each specified point P1 to P3, the image processing device 41 derives the value corresponding to each specified point P1 to P3 as the correction amount. As described above, when the Z-coordinate Zw of the upper end of the adjusted region B21 corresponds to 1850 [mm], the image processing device 41 derives 600 [mm] as the correction amount. If the Z-coordinate Zw of the upper end of the adjusted region B21 corresponds to the height between each specified point P1 to P3, the image processing device 41 derives the correction amount based on the value corresponding to each specified point P1 to P3. For example, when the Z-coordinate Zw of the upper end of the adjusted region B21 corresponds to 1600 [mm], the image processing device 41 derives the correction amount based on the value corresponding to the second specified point P2 and the value corresponding to the third specified point P3. For example, the image processing device 41 only needs to derive the correction amount corresponding to 1600 [mm] based on a linear function represented by the second specified point P2, the third specified point P3, the value corresponding to the second specified point P2, and the value corresponding to the third specified point P3. When the Z-coordinate Zw at the upper end of the adjusted region B21 is higher than the first specified point P1, the image processing device 41 derives the value corresponding to the first specified point P1 as the correction amount. When the Z-coordinate Zw at the upper end of the adjusted region B21 is lower than the third specified point P3, the image processing device 41 derives the value corresponding to the third specified point P3 as the correction amount.

[0103] When deriving the correction amount, the image processing device 41 uses the correction amount to correct the height of the lower end of the adjusted region B21. The image processing device 41 sets the position of the lower end of the adjusted region B21 as the height after subtracting the correction amount from the height of the upper end of the adjusted region B21. As mentioned above, when the height of the upper end of the adjusted region B21 is equivalent to 1850 mm, the 1250 mm obtained by subtracting 600 mm from 1850 mm becomes the height of the lower end of the adjusted region B21.

[0104] The image processing device 41 derives the Y-coordinate Yi of the lower end of the adjusted region B21 in the first image data IM1 based on the height of the lower end of the adjusted region B21 obtained by the correction amount. The image processing device 41 uses the Z-coordinate Zw, which corresponds to the height of the lower end of the adjusted region B21 obtained by the correction amount, to derive the camera coordinates of the lower end of the adjusted region B21 obtained by the correction amount. Furthermore, by converting the camera coordinates to the coordinates of the first image data IM1, the Y-coordinate Yi of the lower end of the adjusted region B21 in the first image data IM1 can be derived. Figure 4In the example shown, the Y-coordinate Yi of the lower end of the adjusted region B21 in the first image data IM1 is corrected to the position of a single-dot dashed line. The adjusted region B21, after correcting the position of its lower end, is called the corrected region B23. The corrected region B23 is an example of an adjusted region. The dimension L3 of the Y-axis direction of the image coordinate system of the corrected region B23 is shorter than the dimension L4 of the Y-axis direction of the image coordinate system of the adjusted region B21. The corrected region B23 in the first image data IM1 is the region obtained by extracting a predetermined length from the upper end of the adjusted region B21. The predetermined length is the value after converting the correction amount into the dimension of the image coordinate system. Step S23 can be described as a process of correcting the height dimension of the adjusted region B21 in the first image data IM1 based on the height of the upper end of the rectangular region B2 above the ground.

[0105] like Figure 7 As shown, next, in step S24, the image processing device 41 compares the corrected region B23 with the upper body dictionary data D2. (As shown...) Figure 15 As shown, the upper body dictionary data D2 represents the feature quantity of the same part as the corrected region B23. In this embodiment, the corrected region B23 is the region above the position corresponding to the chest of a person. Therefore, the upper body dictionary data D2 is also set as the feature quantity data above the position corresponding to the chest of a person. Specifically, the upper body dictionary data D2 is dictionary data obtained from image data showing the part above the chest of a person. The image processing device 41 extracts the feature quantity of the corrected region B23. The feature quantity extraction of the corrected region B23 is performed by the same method as the feature quantity extraction performed when obtaining the upper body dictionary data D2. In this embodiment, HOG feature quantity is extracted from the corrected region B23. Furthermore, by performing pattern matching based on the HOG feature quantity extracted from the corrected region B23 and the upper body dictionary data D2, and machine learning-based person detection, it is determined whether the obstacle in the rectangular region B2 is a person. The upper body dictionary data D2 is upper body comparison data. When the determination in step S24 is completed, the image processing device 41 ends the upper body detection processing S20 and the person detection processing.

[0106] <Control device controls based on the character's position>

[0107] As described above, the position of people present around the forklift 10 is detected by a person detection process. The position of the person is output to the control device 20. The control device 20 can control the forklift 10 based on the position of the person. For example, the control device 20 can limit the vehicle speed if a person is present within a predetermined range from the forklift 10. The control device 20 can also issue an alarm via an alarm device if a person is present within a predetermined range from the forklift 10. This alarm can be an alarm for people around the forklift 10 or an alarm for passengers of the forklift 10.

[0108] <Function>

[0109] The function of this embodiment will be explained.

[0110] If the deviation between the center position C1 of the rectangular region B2 in the width direction and the center position G1 of the obstacle is large, the detection accuracy of the person will decrease. The image processing device 41 obtains the adjusted region B21 by adjusting the width of the rectangular region B2. The deviation between the center position of the adjusted region B21 in the width direction and the center position G1 of the obstacle is smaller than the deviation between the center position C1 of the rectangular region B2 in the width direction and the center position G1 of the obstacle.

[0111] <Effect>

[0112] The effects of this implementation method will be explained.

[0113] (1) The image processing device 41 obtains the adjusted region B21 by adjusting the width of the rectangular region B2. Furthermore, the image processing device 41 determines whether the obstacle presented in the rectangular region B2 is a person by comparing the corrected region B23 obtained from the adjusted region B21 with the upper body dictionary data D2. When obtaining the full-body dictionary data D1 and the upper body dictionary data D2, feature values ​​are generally obtained from image data where the person is located at the center position in the width direction of the image data. That is, the full-body dictionary data D1 and the upper body dictionary data D2 represent feature values ​​where the deviation between the person's center of gravity and the center position in the width direction of the image data is small. Therefore, if the deviation between the center position C1 in the width direction of the rectangular region B2 and the center position G1 of the obstacle is large, the detection accuracy of the person will decrease due to the increased difference in feature values ​​between the rectangular region B2 and the upper body dictionary data D2. The deviation between the center position of the width direction of the adjusted region B21 and the center of gravity G1 of the obstacle is smaller than the deviation between the center position C1 of the width direction of the rectangular region B2 and the center of gravity G1 of the obstacle. Therefore, by comparing the corrected region B23 with the upper body dictionary data D2 to determine whether the obstacle in the rectangular region B2 is a person, the reduction in the detection accuracy of people can be suppressed.

[0114] (2) The image processing device 41 performs upper body detection processing on the rectangular region B2 where the upper body detection processing condition is met. When the lower body of a person presented in the first image data IM1 is hidden by an occlusion, the upper body of the person is presented in the first image data IM1. In the first image data IM1, the person whose lower body is hidden by an occlusion is in a state of being free from the ground. When the person whose lower body is hidden by an occlusion is detected as an obstacle, the upper body detection condition is met for the rectangular region B2 containing the obstacle. In the upper body detection processing, the corrected region B23 is compared with the upper body dictionary data D2. If the corrected region B23 is also compared with the full-body dictionary data D1 for the rectangular region B2 where the upper body detection processing condition is met, the image processing device 41 compares the upper body feature value obtained from the corrected region B23 with the full-body feature value obtained from the full-body dictionary data D1. In this case, the detection accuracy of the person decreases because the positions of the compared feature values ​​are different. In contrast, in the upper body detection processing, the feature values ​​of the upper body obtained from the corrected region B23 are compared with the feature values ​​of the upper body obtained from the upper body dictionary data D2. Since feature values ​​at the same location can be compared, the reduction in the detection accuracy of the person can be suppressed. Furthermore, in the embodiment, a counterweight 15 is described as an example of an occlusion, but obstacles other than the counterweight 15 can also act as occlusions. In this case, the same effect can be obtained by performing the same control as in the embodiment. Additionally, even when the lower body of the person is not hidden by an occlusion, when the lower body of the person is located below the shooting range of the stereo camera 31, only the upper body of the person is presented in the first image data IM1. In this case, the same effect can be obtained by performing the same control as in the embodiment.

[0115] (3) The person detection system 30 is installed on the forklift 10. The forklift 10 is often used in environments where people are present. For the person detection system 30 installed on the forklift 10, it is required to detect people over a large area. Therefore, in the person detection system 30 installed on the forklift 10, a wide-angle stereo camera is often used as the stereo camera 31. When using the wide-angle stereo camera 31, the distortion increases closer to the edge of the first image data IM1. This distortion is one reason why the deviation between the center position C1 of the rectangular area B2 in the width direction and the center of gravity position G1 of the obstacle increases. In this regard, by obtaining the adjusted area B21, the reduction in the accuracy of person detection caused by distortion can be suppressed. Therefore, even when using the wide-angle stereo camera 31, the reduction in the accuracy of person detection can be suppressed.

[0116] Furthermore, the person detection system 30 installed on the forklift 10 requires the stereo camera 31 to detect people in the vicinity of the forklift 10. On the other hand, when attempting to photograph the vicinity of the forklift 10 with the stereo camera 31, obstructions such as the counterweight 15 are easily included in the photographic range. Therefore, in the person detection system 30 installed on the forklift 10, the lower body of a person is easily hidden by the forklift 10 itself. By having the image processing device 41 of the person detection system 30 installed on the forklift 10 perform upper body detection processing, the detection accuracy of the person is not easily reduced even when the lower body of the person is hidden by the forklift 10 itself.

[0117] (4) The width of the adjusted region B21 is derived from equation (5). The inventors conducted repeated experiments and simulations and found that by deriving the width of the adjusted region B21 from equation (5), it is easy to include the entire width direction of the person in the adjusted region B21. For example, if the width of the adjusted region B21 is shorter than the width derived from equation (5), depending on the way the person is presented, sometimes a part of the person's width direction, such as the shoulders, will be outside the range of the adjusted region B21. In this case, the detection accuracy of the person may decrease. By setting the width derived from equation (5) as the width of the adjusted region B21, it is easy to include the entire width direction of the person in the adjusted region B21. Thus, it is possible to suppress the decrease in the detection accuracy of the person.

[0118] (5) When the width of the adjusted region B21 in the world coordinate system is below a predetermined lower limit LL2, the width of the adjusted region B21 in the first image data IM1 is increased. When the width of the adjusted region B21 in the world coordinate system is above a predetermined upper limit UL2, the width of the adjusted region B21 in the first image data IM1 is decreased. The width of the adjusted region B21 can be adjusted when it is too large or too small. This reduces the difference between the width of the upper body dictionary data D2 and the width of the adjusted region B21. This helps to suppress the decrease in the detection accuracy of the person.

[0119] (6) The image processing device 41 corrects the length of the adjusted region B21 in the height direction in the first image data IM1 based on the height of the upper end of the rectangular region B2 above the ground. This results in a corrected region B23, whose height has been corrected for the adjusted region B21. In this embodiment, the corrected region B23 is located above a position roughly equivalent to the chest of a person. By using the upper body dictionary data D2 to correct the height of the adjusted region B21, the detection accuracy of the person can be improved.

[0120] (7) To suppress the decrease in the accuracy of person detection, person detection processing using deep learning can also be considered. In this case, high-performance hardware is required. In contrast, in this embodiment, person detection processing is performed using full-body dictionary data D1 and upper-body dictionary data D2. High-performance hardware is not required, thus suppressing the increase in manufacturing costs.

[0121] (8) To extract the rectangular region B2 where the bottom of an obstacle is hidden by an occluder, it is also possible to calculate whether the rectangular region B2 contains a large number of areas with different parallax. For example, when the lower body of a person is hidden by an occluder, the parallax generated by the occluder and the parallax generated by the person are mixed in the rectangular region B2. Therefore, the rectangular region B2 will contain a large number of areas with different parallax. However, in this case, sometimes it is determined that the rectangular region B2 contains a large number of areas with different parallax even when there are only two obstacles in the rectangular region B2. That is, even when the bottom of an obstacle is not hidden by an occluder, sometimes the bottom of the obstacle is extracted as the rectangular region B2 hidden by the occluder. In this regard, in the embodiment, the rectangular region B2 where the bottom of an obstacle is hidden by an occluder is extracted based on whether the obstacle is free from the ground. Thus, the rectangular region B2 where the bottom of an obstacle is hidden by an occluder can be detected appropriately.

[0122] <Example of Change>

[0123] The implementation method can be modified as shown below. The implementation method and the following modifications can be combined with each other to the extent that they are not technically contradictory.

[0124] ○ For example Figure 16As shown, if the determination result of step S11 is negative, the image processing device 41 may also proceed to step S31. In step S31, the image processing device 41 determines whether the rectangular region meets the conditions for full-body detection processing. The rectangular region for determining whether it meets the conditions for full-body detection processing in step S31 is a rectangular region that does not meet the conditions for upper-body detection processing. In this embodiment, the determination of whether the rectangular region B1 meets the conditions for full-body detection processing is performed. A predetermined height set as the freeing condition for upper-body detection processing is set as a first predetermined height. The full-body detection processing condition is that the height at which the obstacle is freed from the ground is less than a second predetermined height. The second predetermined height is a height lower than the first predetermined height. The second predetermined height is, for example, arbitrarily set within the range of 200 mm to 400 mm. The first predetermined height can, for example, be arbitrarily set within the range of 600 mm to 1000 mm. If the determination result of step S31 is positive, that is, if the rectangular region meets the conditions for full-body detection processing, the image processing device 41 proceeds to step S12. If the determination result in step S31 is negative, that is, if the rectangular area does not meet the conditions for full-body detection processing, the image processing device 41 performs the processing in step S12 and the upper-body detection processing S20. Specifically, for a rectangular area containing an obstacle whose height from the ground is less than a first predetermined height and whose height from the ground is greater than a second predetermined height, the processing in step S12 and the upper-body detection processing S20 are performed. In this case, if at least one of the processing in step S12 and the upper-body detection processing S20 determines that the obstacle is a person, then the obstacle present in the rectangular area is determined to be a person.

[0125] The dictionary data stored in storage unit 43 may also be only the full-body dictionary data D1. In this case, the image processing device 41 compares the corrected region B23 with the full-body dictionary data D1 in step S24. The comparison between the corrected region B23 and the full-body dictionary data D1 uses the region in the full-body dictionary data D1 that corresponds to the area above the chest of the person. That is, it uses the region in the full-body dictionary data D1 obtained from image data showing the full body of a person that represents the same part as the corrected region B23 for comparison. For example, as Figure 6 As shown, comparison can be made using only the region D11 from the top of the full-body dictionary data D1 to the area corresponding to the chest of the person. Region D11 is the upper body comparison data. The image processing device 41 extracts the feature values ​​of the corrected region B23. Furthermore, by performing pattern matching based on the feature values ​​extracted from the corrected region B23 and region D11, and through machine learning, it is determined whether the obstacle in the rectangular region B2 is a person.

[0126] Furthermore, when comparing the whole-body dictionary data D1 with the corrected region B23, the image processing device 41 can also perform region ratio processing in step S23 of the upper body detection processing S20 instead of correcting the height of the adjusted region B21. The region ratio refers to the proportion of the adjusted region B21 within the height of the uppermost part of the obstacle. The height of the uppermost part of the obstacle can be determined from the Z-coordinate Zw of the upper end of the adjusted region B21. The region ratio can be obtained by dividing the dimension between the Z-coordinate Zw of the upper end and the Z-coordinate Zw of the lower end of the adjusted region B21 by the height corresponding to the Z-coordinate Zw of the upper end of the adjusted region B21. In other words, the image processing device 41 detects the region ratio based on the height of the lower end of the adjusted region B21 from the ground and the height of the upper end of the adjusted region B21 from the ground. In step S24 of the upper body detection processing S20, the image processing device 41 performs a process that compares the region obtained by applying a region ratio to the full-body dictionary data D1 with the adjusted region B21. The application of the region ratio to the full-body dictionary data D1 is performed by extracting a portion of the region ratio from the area in the full-body dictionary data D1 in the direction from top to bottom. Therefore, if the obstacle presented in the adjusted region B21 is a person, a region of the same part as that person can be obtained. The region obtained by applying the region ratio to the full-body dictionary data D1 is the upper body comparison data.

[0127] ○ The distance condition can also be set to the distance to the obstacle being within a predetermined distance from the stereo camera 31. In this case, the predetermined distance can simply be the length obtained by adding distance L1 to the predetermined distance of the embodiment.

[0128] In step S23, the image processing device 41 may also correct the adjusted region B21 so that the lower end of the adjusted region B21 in the first image data IM1 corresponds to a position different from the chest of the person. For example, the image processing device 41 may also correct the adjusted region B21 so that the lower end of the adjusted region B21 in the first image data IM1 corresponds to the neck or abdomen of the person. In this case, as the upper body dictionary data D2, it is preferable to use image data that presents a portion above the neck of the person or image data that presents a portion above the abdomen of the person.

[0129] In step S22, if the width of the adjusted region B21 in the world coordinate system of the image processing apparatus 41 is below the lower limit value LL2, the width of the adjusted region B21 can be increased by adding a predetermined value to the width of the adjusted region B21 in the first image data IM1. The predetermined value can be a fixed value or a variable value that increases as the width of the adjusted region B21 in the world coordinate system decreases. If the width of the adjusted region B21 in the world coordinate system of the image processing apparatus 41 is above the upper limit value UL2, the width of the adjusted region B21 can be decreased by subtracting a predetermined value from the width of the adjusted region B21 in the first image data IM1. The predetermined value can be a fixed value or a variable value that decreases as the width of the adjusted region B21 in the world coordinate system increases.

[0130] In step S22, the image processing device 41 may also adjust the width of the adjusted region B21 according to the width of the adjusted region B21 in the image coordinate system.

[0131] Step S23 can also be performed before step S21. In this case, the rectangular region B2 is corrected according to its height above the ground. The width adjustment of the rectangular region B2 in step S21 is performed on the rectangular region B2 after the Y-axis dimension of the rectangular region B2 has been corrected. The adjusted region B21 obtained in this case is the adjusted region B21 after the Y-axis dimension of the rectangular region B2 has been corrected. Therefore, even in this case, it can be said that the length of the adjusted region B21 in the height direction in the first image data IM1 has been corrected.

[0132] The image processing device 41 can also derive the width W11 from the formula obtained by omitting the first correction rate from formula (5).

[0133] The image processing device 41 can adjust the width of the rectangular region B2 in any way, as long as the deviation between the center position of the adjusted region B21 in the width direction and the center position G1 of the obstacle is smaller than the deviation between the center position C1 of the rectangular region B2 in the width direction and the center position G1 of the obstacle.

[0134] ○The image processing device 41 may also skip step S23.

[0135] ○The image processing device 41 may also skip step S21.

[0136] In step S22, the image processing device 41 may perform either the process of increasing the width of the adjusted region B21 in the world coordinate system when the width is below a predetermined lower limit LL2, or the process of decreasing the width of the adjusted region B21 in the world coordinate system when the width is above a predetermined upper limit UL2. If, in step S22, the image processing device 41 only performs the process of increasing the width of the adjusted region B21 in the world coordinate system when the width is below the predetermined lower limit LL2, the upper limit UL2 need not be set. In this case, if the width of the adjusted region B21 in the world coordinate system is greater than the lower limit LL2, 1.0 can be derived as the second correction rate. If, in step S22, the image processing device 41 only performs the process of decreasing the width of the adjusted region B21 in the world coordinate system when the width is above the predetermined upper limit UL2, the lower limit LL2 need not be set. In this case, if the width of the adjusted region B21 in the world coordinate system is less than the upper limit UL2, it is sufficient to derive 1.0 as the second correction rate.

[0137] If the determination in step S11 is negative, the image processing device 41 may also proceed to step S21. That is, the image processing device 41 may adjust the width of the rectangular area that does not meet the upper body processing conditions using the center position G1. In this case, the image processing device 41 may also proceed to step S22 for the rectangular area that does not meet the upper body processing conditions. It can be said that the image processing device 41 can compare the rectangular area whose width has been adjusted with the full-body dictionary data D1, or it can compare the rectangular area whose width has not been adjusted with the full-body dictionary data D1. In either case, it can be said that the image processing device 41 determines whether the obstacle presented in the rectangular area is a person by comparing the rectangular area in the first image data IM1 with the full-body dictionary data D1 for the rectangular area where the upper body detection processing conditions are not met.

[0138] Alternatively, if step S21 is performed when the determination in step S11 is negative, steps S21 and S22 may not be performed in the upper body detection process. That is, the center-of-gravity region width adjustment process only needs to be performed on at least one of the rectangular regions that meet the upper body detection processing conditions and the rectangular regions that do not meet the upper body detection processing conditions.

[0139] Alternatively, the image processing device 41 may not determine whether the rectangular region meets the upper body detection processing conditions. In this case, steps S21 and S22 are performed on the entire rectangular region. The image processing device 41 determines whether the obstacle presented in the rectangular region is a person by comparing the adjusted region after processing steps S21 and S22 with the full-body dictionary data D1. In this case, by adjusting the width of the rectangular region, the decrease in the detection accuracy of people can also be suppressed.

[0140] ○The stereo camera 31 can also be set so that the counterweight 15 does not enter the shooting range.

[0141] ○The area with obstacles in the first image data IM1 can also be a circular area or an area other than a rectangular shape.

[0142] The person detection system 30 can also detect people in front of the forklift 10. In this case, the stereo camera 31 is configured to capture images of the front of the forklift 10. Additionally, the person detection system 30 can also detect people on both the front and rear sides of the forklift 10. In this case, the stereo camera 31 is configured with both a stereo camera capturing images of the front of the forklift 10 and a stereo camera capturing images of the rear of the forklift 10.

[0143] Alternatively, conversion from camera coordinates to world coordinates can be performed using tabular data. This tabular data includes data that maps the combination of Y-coordinates Yc and Z-coordinates Zc to Y-coordinate Yw, and data that maps the combination of Y-coordinates Yc and Z-coordinates Zc to Z-coordinate Zw. By pre-storing this tabular data in the storage unit 43 of the image processing apparatus 41, the Y-coordinates Yw and Z-coordinates Zw in the world coordinate system can be calculated from the Y-coordinates Yc and Z-coordinates Zc in the camera coordinate system. Furthermore, in this embodiment, since the X-coordinate Xc in the camera coordinate system is the same as the X-coordinate Xw in the world coordinate system, tabular data for calculating the X-coordinate Xw is not stored.

[0144] ○The world coordinate system is not limited to an orthogonal coordinate system; it can also be set as a polar coordinate system.

[0145] ○A camera is sufficient as long as it can derive the world coordinates of obstacles from the image data obtained by the camera; any camera can be used. For example, a single-lens reflex camera or a ToF (Time of Flight) camera can be used.

[0146] ○ Forklift 10 can also be an engine-driven forklift.

[0147] ○The stereo camera 31 can also be mounted on any location, such as the cargo loading and unloading device 17.

[0148] ○The mobile body can also be an industrial vehicle different from a forklift 10, such as a trailer. The mobile body can also be any mobile body, such as a passenger car, a handling vehicle, construction machinery, or an aircraft.

Claims

1. An image processing apparatus of a person detection system installed on a moving body, the image processing apparatus characterized by, the image processing apparatus detecting, from image data acquired from a camera, a region in which an obstacle is present in the image data, estimating a barycentric position of the obstacle included in the region in the image data, performing barycentric region width adjustment processing of adjusting a width of the region in the image data according to the barycentric position, determining, by comparing an adjusted region after the barycentric region width adjustment processing is performed with dictionary data, whether the obstacle present in the region is a person.

2. The image processing apparatus of the person detection system according to claim 1, wherein, the dictionary data includes upper body comparison data and full body comparison data, the image processing apparatus determines, for each of the regions, whether an upper body detection processing condition that the obstacle is floating from the ground in a predetermined range from the camera is satisfied, for the regions in which the upper body detection processing condition is satisfied, performs upper body detection processing of determining, by comparing the adjusted region after the barycentric region width adjustment processing is performed on the region in the image data with the upper body comparison data, whether the obstacle present in the region is a person, for the regions in which the upper body detection processing condition is not satisfied, performs full body detection processing of determining, by comparing the region in the image data with the full body comparison data, whether the obstacle present in the region is a person.

3. The image processing apparatus of the person detection system according to claim 1 or claim 2, wherein, the barycentric region width adjustment processing is processing of setting, as a width of the adjusted region, a length obtained by adding a value based on a difference between the barycentric position and a center position in a width direction of the region to a length obtained by making a distance from the barycentric position to a side of the region close to the barycentric position in the width direction twice.

4. The image processing apparatus of the person detection system according to any one of claims 1 to 3, wherein, the image processing apparatus performs width adjustment processing on the adjusted region, the width adjustment processing is processing of at least one of increasing the width of the adjusted region in the image data when the width of the adjusted region is below a lower limit value decided in advance and decreasing the width of the adjusted region in the image data when the width of the adjusted region is above an upper limit value decided in advance.

5. The image processing apparatus of the person detection system according to claim 2, wherein, the image processing apparatus detects a height of an upper end of the region from the ground, the upper body detection processing includes processing of correcting a size in a height direction of the adjusted region in the image data according to the height of the upper end of the region from the ground.

Images