MOBILE BODY, MOBILE BODY CONTROL METHOD, AND MOBILE BODY CONTROL SYSTEM

Abstract

To provide a mobile body, a mobile body control method, and a mobile body control system that can properly track an object without sacrificing real-time performance. [Solution] The control device is a mobile body that includes an area identification unit that identifies an object contained in the shooting information as a specific area part, a specific area depth acquisition unit that acquires a depth image by cutting out the specific area part containing depth information from the shooting information, a specific depth image acquisition unit that sets the target cluster range based on the depth information of the specific area part and obtains a specific depth image by multiplying the depth image with a mask image, which is an image from which depth information representing distances other than the object has been removed, a center of gravity identification unit that calculates the center of gravity from the area occupied by the object located in the specific depth image, and a driving control unit that controls the driving device to maintain a distance from the object while directing the vehicle toward the object so that the center of gravity of the object located in the specific depth image is at a predetermined position.

Description

[Technical Field]

[0001] The present invention relates to a moving body that moves following a target using photographic information captured by a photographing means, and a control method and control system. [Background technology]

[0002] For example, Patent Document 1 discloses a moving object that moves by following a person who is the target of tracking. This discloses a technology that uses a ranging sensor that measures the distance to the surroundings, an image sensor that acquires captured images of the surroundings, and the measurement results of the ranging sensor and the captured images to identify the position of the target of tracking in a local coordinate system, and move the moving object by following the target of tracking. The image sensor is typically a monocular camera, and the ranging sensor uses a LIDAR, which is a laser sensor. There is also a description that the image sensor may be a stereo camera. The technology also discloses a technology that can identify the position of the cluster of the target of tracking and track the target of tracking. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-167432 Summary of the Invention [Problem to be solved by the invention]

[0004] To identify the cluster position using a configuration such as that described in Patent Document 1, it is necessary to search the entire range of the ranging scan and identify the target to be tracked. In other words, it is necessary to perform ranging on the entire range of the captured image and then search for and cluster the target to be tracked within it, which imposes a certain level of computational load and also requires a certain level of computing power. It is expected that this will be even more disadvantageous when performing clustering processing on a person who moves irregularly. When installed on a moving object that moves to track the target to be tracked, real-time execution is required, which requires advanced computing and control devices. Furthermore, when the target is a person, the person may not only stand upright, but may also bend over or spread their arms and legs. This can lead to errors in measuring the distance to the target person, and if the vehicle attempts to follow the person while maintaining a constant distance, it may not be able to maintain an appropriate distance. Furthermore, if the captured image contains background other than the target object, it may not be possible to accurately calculate the distance to the target person, whose posture changes. This measurement error becomes more pronounced as the proportion of the target person's area in the captured image and the distance measurement range decreases, which can affect the vehicle's ability to follow the person. In other words, there are challenges in accurately identifying the person to be followed and following them.

[0005] Therefore, the present invention has been made with the above-mentioned problems in mind, and one of its objects is to provide a mobile body, a method for controlling a mobile body, and a control system for a mobile body that can properly track an object without sacrificing real-time performance. [Means for solving the problem]

[0006] In order to achieve the object, one aspect of the present invention is a mobile body comprising: a traveling device supported on a vehicle body; an imaging means attached to the vehicle body and acquiring imaging information including depth information, which is distance information of an object photographed; and a control device capable of controlling the traveling device based on the imaging information, wherein the control device comprises: an area identification unit that identifies the object included in the imaging information as a specific area portion; a specific area depth acquisition unit that acquires a depth image by cutting out the specific area portion including depth information from the imaging information; a specific depth image acquisition unit that sets a target cluster range based on the depth information of the specific area portion and obtains a specific depth image by multiplying the depth image by a mask image, which is an image from which depth information, which is distance other than the object, has been removed; a center of gravity identification unit that calculates the center of gravity from the area occupied by the object located in the specific depth image; and a traveling control unit that controls the traveling device to maintain a distance from the object while directing it toward the object so that the center of gravity of the object located in the specific depth image is at a predetermined position. In addition, in order to achieve the object, another aspect of the present invention is a control method for a mobile body including a control device capable of controlling a traveling device based on shooting information including depth information, which is distance information of a photographed object, and the control method for a mobile body includes an area identification process for identifying an object included in the shooting information as a specific area part, a specific area depth acquisition process for acquiring a depth image by cutting out the specific area part including depth information from the shooting information, a specific depth image acquisition process for setting a target cluster range based on the depth information of the specific area part and obtaining a specific depth image by multiplying the depth image with a mask image, which is an image from which depth information, which is distance other than the object, has been removed, a center of gravity identification process for calculating the center of gravity from the occupied area of ​​the object located in the specific depth image, and a traveling control process for controlling the traveling device to maintain a distance from the object while directing it toward the object so that the center of gravity of the object located in the specific depth image is at a predetermined position. In addition, in order to achieve the object, another aspect of the present invention is a control system for a mobile body that includes a running device supported on the vehicle body, an imaging means that is attached to the vehicle body and acquires imaging information including depth information, which is distance information of the object photographed, and a control device that can control the running device based on the imaging information, wherein the control system includes: identifying the object included in the imaging information as a specific area portion; acquiring a depth image by cutting out the specific area portion including depth information from the imaging information; setting a target cluster range based on the depth information of the specific area portion; obtaining a specific depth image by multiplying the depth image by a mask image, which is an image from which depth information, which is distance other than the object, has been removed; calculating the center of gravity from the area occupied by the object located in the specific depth image; and controlling the running device to maintain a distance from the object while pointing it toward the object so that the center of gravity of the object located in the specific depth image is at a predetermined position. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a mobile body, a method for controlling a mobile body, and a system for controlling a mobile body that can appropriately track an object without sacrificing real-time performance. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a side view showing a configuration example to which an embodiment of the present invention is applied. [Figure 2] FIG. 1 is a plan view showing a configuration example to which an embodiment of the present invention is applied. [Figure 3] 3A and 3B are explanatory diagrams showing photographic information photographed by the photographing means of the embodiment of the present invention. [Figure 4] 10A and 10B are explanatory diagrams showing a specific area extracted from photographic information photographed by the photographing means of the embodiment of the present invention; [Figure 5] FIG. 10 is an explanatory diagram showing depth information included in a specific region according to an embodiment of the present invention. [Figure 6] FIG. 10 is an explanatory diagram showing a state after preprocessing is performed on depth information included in a specific region according to an embodiment of the present invention. [Figure 7] An explanatory diagram showing depth information contained in a specific area in an embodiment of the present invention rearranged in a single column, where (a) shows the depth information in two-dimensional state arranged in the order in which it was extracted, and (b) shows the depth information arranged in the order of the distance contained in the depth information. [Figure 8] FIG. 10 is an explanatory diagram showing the distribution of depth information included in a specific region according to an embodiment of the present invention. [Figure 9] FIG. 10 is a graph showing the distribution of depth information according to an embodiment of the present invention. [Figure 10] FIG. 10 is an explanatory diagram showing a depth image in which depth information is arranged in order of distance according to an embodiment of the present invention. [Figure 11] 10A and 10B are diagrams illustrating an example in which difference values ​​of depth information equal to or greater than a threshold value are shown according to an embodiment of the present invention. [Figure 12] 1A and 1B are diagrams illustrating a mask image according to an embodiment of the present invention. [Figure 13] 10A and 10B are diagrams illustrating a specific depth image and the center of gravity of a subject according to an embodiment of the present invention. [Figure 14] FIG. 1 is a block diagram of an embodiment of the present invention. [Figure 15] FIG. 3 is an explanatory diagram showing a control process according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] Next, an embodiment of the present invention will be described with reference to the drawings. In the following description of the drawings, the same or similar parts may be designated by the same or similar reference numerals. The drawings used for the description are schematic, and the relationship between the dimensions of each part may differ from the actual ones. Furthermore, the technical idea of ​​the present invention can be modified in various ways within the technical scope defined by the claims.

[0010] First, the configuration of the device and the configuration of the system will be described. As shown in FIGS. 1 and 2 , the embodiment of the present invention will be described as being used in a mobile body 1 that has a traveling device 3 and a machine body 2 that supports the traveling device 3 and is configured to be capable of traveling forward and backward and turning left and right. The traveling device 3 is composed of a pair of drive wheels 3a arranged on the left and right sides at the front and a pair of driven wheels 3b arranged on the left and right sides at the rear. Each of the drive wheels 3a is connected to a motor 3c and drives the motor 3c, utilizing the rotation difference between the motors 3c to perform forward and backward straight travel and turning left and right. The traveling device 3 may be of a type other than that described in the embodiment, and the type of traveling device 3 can be freely determined depending on the implementation mode. For example, the drive wheels 3a may be arranged on the rear side and the driven wheels 3b on the front side. Furthermore, the driven wheels 3b may be arranged in front of and behind the drive wheels 3a. Furthermore, the traveling device 3 may be a track-type traveling device, and straight travel and turning may be performed by utilizing the rotation difference between the left and right tracks. The vehicle may also have steering wheels, and turning may be performed by applying a steering angle to the steering wheels. The motor 3c is described as an electric motor 3c driven by power obtained from a battery (not shown), but a hydraulic motor 3c may also be used. Furthermore, the drive source for driving the drive wheels 3a does not have to be the motor 3c, and the drive wheels 3a may be driven by rotational power obtained from a prime mover such as an engine via a transmission mechanism.

[0011] The front of the vehicle 2 is provided with an imaging means 4, which is installed so as to be able to capture images of the forward side in the direction of travel of the vehicle 2. The imaging means 4 is capable of capturing images of objects Ob located in the forward direction, including a human subject P. The imaging means 4 used in the embodiment will be described as capturing images of an angle of approximately 90 to 120 degrees left and right in the forward direction, including the traveling surface S. The viewing angle of the imaging means 4 is not limited, and may be an angle including an angle of 180 degrees left and right in the direction of travel, or may be an imaging device capable of capturing images including an angle of 360 degrees in all directions, front, back, left, right, up and down. It is sufficient if the imaging means 4 can capture images of at least the objects in the forward direction, and adjustments and changes can be made as appropriate depending on the application form.

[0012] The photographing means 4 transmits still images captured intermittently at a fixed interval by the monocular camera 4a as photographic information. In this embodiment, the photographic information is transmitted sequentially by capturing still images approximately every 0.05 seconds. The intermittent time can be changed or adjusted depending on the implementation mode. The image size, which indicates the dimensions and data amount of the still image, can also be changed or adjusted as appropriate. In this embodiment, the calculation load is reduced by performing calculations using still images. However, it is also possible to use so-called moving images with an even shorter intermittent time, and this is applicable within the allowable range of processing power.

[0013] The image captured as the photographic information is a color image, and an RGB image is used, but it may be a monochrome image or may have a different format of color space information.

[0014] Furthermore, the imaging means 4 has a stereo camera 4b, which is a camera with a compound lens. The stereo camera 4b captures an image of an object, and the image parallax obtained from the lenses, which are spaced apart on the left and right sides of the traveling direction, is calculated by the computing device 6, which will be described later, to obtain depth information, which is the distance in the depth direction. Therefore, the imaging information includes information on a still image captured by the monocular camera 4a and depth information captured by the stereo camera 4b, and the distance to the object Ob captured in the still image can be obtained. The image used for image parallax captured by the stereo camera 4b is a monochrome image, but it may also be a color image.

[0015] The aircraft 2 is equipped with a calculation device 6 connected to the imaging means 4. The calculation device 6 is also called an information processing device and is a computer system having at least one CPU (Central Processing Unit) and performs calculations and processing of various information. The calculation device 6 may further include any or all of a ROM (Read Only Memory), a RAM (Random Access Memory), and a storage device. The calculation device 6 is configured to be able to acquire the imaging information obtained by the imaging means 4. The calculation device 6 is able to obtain depth information, which is the distance to the photographed object, from the acquired imaging information. The information processing device 6, which is the calculation device 6, is configured to be able to calculate the distance to the subject P, who is a person included in the imaging information obtained from the imaging means 4.

[0016] The vehicle 2 is equipped with a control device 5 connected to an information processing device 6. The control device 5 uses the calculation results of the information processing device 6 to control the traveling device 3 to operate appropriately. In the embodiment, the information processing device 6 captures an image of the area ahead in the traveling direction, detects a specific person included in the captured image information as a subject P, and controls the traveling device 3 to follow the subject P while maintaining a certain distance from the subject P. Note that in the embodiment, the information processing device 6 is included in the control device 5, and the information processing device 6 and the control device 5 are described as being connected to each other. Alternatively, the information processing device 6 and the control device 5 may be installed in separate locations on the vehicle 2 and connected to each other, or the information processing device 6 may be installed in a location other than the vehicle 2 and connected to the control device 5 installed on the vehicle 2. Note that the connection between the information processing device 6 and the control device 5 may be wireless or wired, and there are no limitations on the communication method or protocol.

[0017] The computing device 6 uses AI (Artificial Intelligence) to identify a person from the information in the shooting information. To identify a person, the AI ​​determines that an object Ob photographed in the shooting information is a person or part of a person when at least part of a human body is included in the object Ob photographed in the shooting information, based on teaching data previously stored in the computing device 6. Hereinafter, a person or part of a person will be referred to as a subject P. In addition, in the following description, the subject P included in the shooting information will be described as including the entire person. Furthermore, the object Ob is any shape with projections and depressions that can be measured by the shooting means, and includes not only people but also road surfaces, wall surfaces, etc.

[0018] The arithmetic unit 6 acquires two-dimensional coordinates of two points, the minimum and maximum coordinates of the points determined to be the subject P from the photographic information. The identified subject P can be represented as being enclosed by a rectangular frame B so that the minimum and maximum coordinate points are included. In other words, the rectangular frame B can be said to be attached so as to surround the entire recognized subject P and minimize the reflection of other objects Ob, such as the background of the subject P. The rectangular frame B is also called a bounding box, and can be added to the photographic image, which is the photographic information, to generate an image. In other words, the arithmetic unit 6 determines the subject P from the acquired photographic information and attaches a rectangular frame B to it, thereby enabling the operator to easily recognize the subject P. In the embodiment, if the subject P cannot be identified from the photographic information, the bounding box B is not attached.

[0019] Based on the depth information included in the photographing information, the calculation device 6 performs various calculations and correction calculations on the specific area B1 determined to be the subject P, thereby enabling accurate and high-speed calculation of the distance to the subject P, who is the target object. The calculation process is as follows.

[0020] (Calculation procedure) First, the arithmetic device 6 performs a photographing information acquisition step. In the photographing information acquisition step, the photographing information acquisition unit 6a of the arithmetic device 6 acquires photographing information, such as a still image and depth information, from the photographing means 4 that has photographed the object. The photographing information acquisition step is executed in step s1 in the calculation step shown in FIG. 15. FIG. 3 is an example showing an entire still image photographed by the photographing means 4. The still image includes multiple objects Ob. Note that, although the still image in this embodiment is output as an RGB image, there are no limitations on the image format or color space correction method.

[0021] After the photographing information acquisition process is completed, the region identification process is performed by the region identification unit 6b of the calculation device 6. The region identification process is executed in step s2 of the calculation process shown in FIG. 15. In the region identification process, a calculation using AI stored in the region identification unit 6b is performed to determine whether the subject P is included in the photographing information and still image. If the region identification unit 6b determines, through AI determination, that a person or part of a person is included in the photographing information and still image, it adds a bounding box B that encloses the entire object identified as the subject P to the two-dimensional still image that is the photographed image. In FIG. 3, the subject P is identified from the identified objects Ob, and a bounding box B is added to the two-dimensional still image so as to surround the subject P.

[0022] After completing the region identification process, the calculation device 6 performs a specific region depth calculation process using the specific region depth acquisition unit 6c. The specific region depth calculation process is performed in step s3 of the calculation process shown in FIG. 15. In the specific region depth calculation process, the specific region depth acquisition unit 6c cuts out a specific region B1, which is a region surrounded by a bounding box B in the still image, including depth information as shown in FIG. 4. The specific region depth acquisition unit 6c then obtains a depth image for the specific region B1, which is information on the distance to the object Ob and the subject P. FIG. 5 shows a depth image that is information on the distance to the object Ob and the subject P for the portion surrounded by the bounding box B shown in FIG. 4. In the illustration, numerical values ​​indicate values ​​indicating the distance to the object, which is the depth information, and values ​​of 0 (zero) indicate points where the distance, which is the depth information, is a measurement error. The depth image at this point is shown as a two-dimensional image. It can be said that the area occupied by the subject P in the cut-out depth image is extremely large compared to the area photographed by the photographing means 4. It can also be said that the area occupied by unnecessary parts other than the subject P, such as the background, can be reduced in the cut-out depth image.

[0023] At this time, the specific region depth acquisition unit 6c can also treat the depth information included in the depth image as depth information that has been subjected to preprocessing such as noise removal and filtering. FIG. 6 shows depth information obtained by preprocessing the depth image shown in FIG. 5. Preprocessing can be used to correct the numerical distances between objects shown in the depth image so as to emphasize them, or to delete or reduce minute information that can be ignored in subsequent calculations. Furthermore, by performing preprocessing, it is possible to efficiently obtain the median (described later) and determine the density of the distance group in which the median is located.

[0024] FIG. 6 is a diagram showing a depth image after preprocessing such as filtering has been performed. Each grid-like section shown in the figure represents one pixel. The numbers displayed in each section indicate the measured distance to each object, with larger numbers indicating greater distances. If a distance cannot be measured in a section, a measurement error is indicated by a "0" (zero) in the section. In the embodiment, measurement errors are displayed as "0," but this is not limited to this. Any display and content that can be interpreted as a measurement error is sufficient. To facilitate understanding, in FIG. 6, the numbers in areas determined to represent subject P are surrounded by thin diagonal lines, and the numbers in areas determined to represent other objects are surrounded by coarse diagonal lines. The depth image contains distance information measured for each pixel.

[0025] After the specific region depth acquisition process by the specific region depth acquisition unit 6c is completed, the depth array conversion unit 6d performs a depth array conversion process. The depth array conversion process is executed in step s4 of the calculation process shown in FIG. 15. In the depth array conversion process, the two-dimensional depth image is transformed into one-dimensional data. That is, each pixel of the image, which appears to be expressed in two dimensions (vertical and horizontal), is rearranged into a simple single column. In the embodiment, as shown in FIG. 7(a), the two-dimensional depth image of FIG. 6 is extracted row by row, pixel by pixel, from top to bottom, starting from the left, and then joined into a single column and arranged from top to bottom. The order and direction of extraction are not limited and can be changed and adjusted as appropriate depending on the implementation mode. Furthermore, although the illustration shows the pixels being rearranged vertically from top to bottom, they may also be rearranged horizontally, and the optimal direction can be adopted as appropriate.

[0026] Even if a two-dimensional image is rearranged into a single column and the order is changed, an array index is attached to each position of the original two-dimensional pixels along with pixel data information so that the original two-dimensional image can be restored. The array index can be considered as an address number assigned to each section in the original two-dimensional space. In the extracted depth images of Figures 5 and 6, where the horizontal direction is the X axis and the vertical direction is the Y axis, the array indexes are assigned in the format of "X0, Y0," "X1, Y0," "X2, Y0," etc. from the top left toward the horizontal direction, and in the format of "X0, Y0," "X0, Y1," "X0, Y2," etc. from the top left toward the vertical direction. Even if a two-dimensional image is transformed into a one-dimensional image, it can be easily restored based on the array index. In this embodiment, the array index, which is an address number, is assigned in the order in which the two-dimensional depth information was extracted. In other words, an array index is assigned in addition to distance information to each pixel in the column rearranged into a single row shown in Figure 7(a). The rules for the numbers and symbols expressed as array indexes can be modified as appropriate depending on the application. Although the array indexes are assigned relative to the depth image extracted from the photographing information, absolute numbers and symbols may be used based on the position of the photographing information. The image of a specific region of the photographing information and the depth image extracted from the photographing information correspond to each other in a convertible manner.

[0027] Then, the depth array conversion unit 6d rearranges (sorts) the columns shown in FIG. 7(a) in order of distance as shown in FIG. 7(b) based on the depth information, which is distance information recorded in the depth image. A position index, which is a number assigned in ascending order of distance, is assigned to each section of each pixel rearranged in order of distance. In the illustration, the columns are rearranged from top to bottom in order of smallest distance, and position indexes are assigned consecutively from 0 to N in order of smallest distance. FIG. 7(b) shows the depth information, i.e., the distance values, sorted from smallest to largest, but they may also be sorted from largest to smallest, and the order can be set arbitrarily. Furthermore, the order in which position indexes are assigned may also be sequentially numbered from largest to smallest. After the columns are rearranged from FIG. 7(a) to FIG. 7(b), position indexes are assigned to the columns of FIG. 7(b) rearranged in order of distance. In other words, in addition to distance information and array indexes, position indexes are also assigned to the columns rearranged in order of distance in the depth array conversion process.

[0028] By the depth array conversion unit 6d assigning the above array index and position index to each section, which is each pixel of the depth image, it is possible to reversibly convert between a depth image shown in two dimensions that can be recognized by normal humans, a depth image that has been transformed one-dimensionally in the order of address numbers, and a depth image in which the one-dimensionally transformed depth image has been further rearranged in the order of distance.

[0029] When the depth array conversion unit 6d finishes assigning position indexes, the median acquisition unit 6e of the calculation device 6 performs a median acquisition step. The median acquisition step is executed in step s5 of the calculation step shown in FIG. 15. The median acquisition unit 6e calculates or searches for the median of the points other than "0" (zero), which is a measurement error, for the depth image sorted in the depth array conversion step, and acquires a position index of the median. In detail, the median acquisition unit 6e searches N pieces of depth information, counting from the 0th point other than the sorted "0" (zero), from the minimum value to the maximum value, and finds the median located at the N / 2th position and the position index indicating the median.

[0030] In this case, it does not necessarily have to be the N / 2th value, but may be the median and position index located in the order of the integer closest to the N / 2th value. Also, if the total number N to be searched is an even number, a position index located in an array expressed by the formula {(N / 2-1)+(N / 2+1)} / 2 may be acquired. In the embodiment, the 0th value is the minimum value, the Nth value is the maximum value, and the search is performed from the minimum value to the maximum value, but the 0th value may also be the maximum value and the Nth value may also be the minimum value. Also, the search may be performed from the maximum value to the minimum value.

[0031] In the median value acquisition step, a frequency distribution table may be generated by the median value acquisition unit 6e. FIG. 8 is a frequency distribution table showing the number of depth information values ​​for a predetermined distance range. The frequency distribution table also shows the median when all values ​​other than "0" (zero), which are measurement error values, are removed. The median is displayed as 650. For example, the numerical values ​​shown in the distance range column in FIG. 8 indicate the number of pixels having a distance value of "0 or more but less than 25," and "50" indicates the number of pixels having a distance value of "25 or more but less than 50." The following expressions follow this pattern.

[0032] In the median value acquisition step, the median value acquisition unit 6e may generate a graph based on the frequency distribution table. The graph shown in FIG. 9 is a graph that is a representation of the frequency distribution table to facilitate visual understanding. The values ​​on the horizontal axis of FIG. 9 indicate depth, and the vertical axis indicates the number corresponding to the depth. The values ​​on the horizontal axis of FIG. 9 are similar to those in FIG. 8, for example, "25" indicates the number of pixels having a distance value of "0 or greater but less than 25," and "50" indicates the number of pixels having a distance value of "25 or greater but less than 50." The following expressions follow this pattern. As shown in the frequency distribution table and median display and graph in FIGS. 8 and 9, distances including the target object Ob are concentrated near the median. In FIGS. 8 and 9, the median value and the object Ob indicated by the median value are detected in the area indicated by "650" to "750." The calculation device 6 determines that the object Ob located near the median value is the subject P. Since the depth information is concentrated near the median, the median acquiring unit 6e provisionally determines that there is an extremely high possibility that the pixel portion assigned a distance near the median indicates the distance to the object, the subject P. In the graph of the embodiment shown in FIG. 9, "0" in the frequency distribution is a measurement error value, and the arithmetic device 6 determines that an object Ob without a median is the floor, which is an object Ob other than the subject P.

[0033] After the median value acquisition step by the median value acquisition unit 6e is completed, the difference value acquisition step is performed by the difference value acquisition unit 6f. The difference value acquisition step is executed in step s6 of the calculation step shown in FIG. 15. In the difference value acquisition step, the sorted array shown in FIG. 7 is sliced ​​so that each position is shifted by one element (column by column) to obtain the difference. In this embodiment, the difference value is obtained by differentiating each of the numerical values ​​of the sorted array. In other words, the difference value is the distance between each adjacent array. The difference value can also be said to be a value that indicates the degree of distance between each adjacent array.

[0034] FIG. 10 is an explanatory diagram that schematically illustrates the array of depth information immediately after sorting in order of distance and the array of depth information to be differentiated. The upper part of FIG. 10 shows depth information from 0 to N, with the portion likely to be the target object indicated by diagonal lines and the position of the median. The diagonal lines in the array in the upper part of FIG. 10 indicate the position tentatively determined by the computing device 6 as the location of subject P based on the frequency distribution table. The lower part of FIG. 10 illustrates the relationship between the difference (differentiation), and the relationship between the array of depth information from 0 to N-1 sorted in order of distance before differentiation and the array of depth information after differentiation using the 1st to Nth depth information, along with the positional relationship of the median.

[0035] The difference value acquisition unit 6f extracts position indexes having values ​​equal to or greater than a threshold value from the array obtained by differentiating the array of depth information from 0th to N-1th before differentiation from 1st to Nth to acquire differences. Figure 11 is a schematic diagram showing the array after differential values ​​have been acquired. The upper part of Figure 11 is a diagram explaining the setting of cluster ranges, and the shaded areas shown in multiple sections indicate position indexes having difference values ​​equal to or greater than a threshold value from the difference values ​​acquired by the difference value acquisition process for information sorted in order of distance. The notations "L" and "R" in the figure indicate the left and right sides when divided at the median. The lower part of Figure 11 is a diagram showing mask arrays and identification information, illustrating an example of identification assigned as cluster ranges to information sorted in order of distance in the mask array setting process.

[0036] After the difference value acquisition step by the difference value acquisition unit 6f is performed, the mask array setting unit 6g performs the mask array setting step. The mask array setting step is executed in step s7 of the calculation step shown in FIG. 15. In the mask array setting step, as shown in the upper part of FIG. 11, among the position indices having values ​​equal to or greater than a threshold after acquiring the difference values ​​of the depth information sorted in order of distance, the position indices are divided into two at the position index indicating the median as a boundary. If the median of the position indices is set as the boundary, the maximum value on the minimum value side, which is located near the boundary when sorted in order of distance, and the minimum value on the maximum value side, which is located on the other side, are set as the cluster range of the measurement target. In FIG. 11, the median is set as the boundary, and the range between the maximum value on the left side and the minimum value on the right side is set as the cluster range of the subject P, who is the measurement target. In other words, the median of the position indices sorted in order of distance is set as the boundary, and the maximum and minimum values ​​of the position indices having difference values ​​equal to or greater than a threshold on both sides are set as the cluster range of the measurement target. The mask array setting unit 6g additionally assigns identification information to the position indices set as the cluster range. As shown in the lower part of FIG. 11, in the identification information in this embodiment, "1" is assigned to the part that is determined to be within the cluster range, and "0" is assigned to the part that is not within the cluster range.

[0037] The method for setting the cluster range by the mask array setting unit 6g may be other than the above. For example, in an array sorted by distance, the position index indicating the median value is used as a boundary. If there is no position index with a difference value equal to or greater than the threshold on either side of the boundary, the array of position indexes immediately adjacent to the boundary position index on the side where there is no difference can be set as the cluster range. For example, in FIG. 11, if there is no position index with a difference value equal to or greater than the threshold on the left side of the position index indicating the median value, the position index immediately to the left of the position index indicating the median value is obtained. Conversely, if there is no position index with a difference value equal to or greater than the threshold on the right side of the position index indicating the median value, the position index immediately to the right of the position index indicating the median value is obtained. In this way, even in an exceptional situation where a difference value equal to or greater than the threshold cannot be obtained on either side of the boundary, the cluster range can be set appropriately for the measurement target.

[0038] As a result, as shown in the lower part of Figure 11, a mask array is generated by assigning identification information, which marks the sorted array determined to be within the cluster range, to the array determined to be within the cluster range. In the illustration, adding a "1" identifies it as belonging to the cluster range, and adding a "0" identifies it as not belonging to the cluster range. The mask array is an array in which cluster range information is assigned to each pixel of the array sorted in order of distance. In other words, the mask array is assigned identification information indicating the cluster range in addition to distance information, array index, and position index. Furthermore, at this point, the mask array has each pixel arranged one-dimensionally in a single row. Each array of the mask array generated in the mask array setting process also has the previously assigned position index information and depth information, unchanged from the specific region depth extraction process and the depth array conversion process. The mask array setting process can be said to more precisely extract only the portion corresponding to subject P from the portion tentatively determined in the median acquisition process to be highly likely to indicate the distance to subject P.

[0039] After the mask array setting process is completed, the mask image conversion process is carried out. The mask image conversion process is executed in step s8 of the calculation process shown in Fig. 15. In the mask image conversion process, the mask image conversion unit 6h reconstructs the generated mask array to the same size and array as the original depth image acquired by the imaging means 4, thereby acquiring a mask image. In other words, the mask image is obtained by converting the mask array expressed one-dimensionally so as to rearrange it into an array expressed two-dimensionally as shown in Fig. 12.

[0040] To explain the conversion process in detail, the mask array, arranged in order of distance contained in the depth information, is rearranged in order of array index before being converted into a mask image. In other words, the depth image, which was represented in one-dimensional distance order, is rearranged in order of address numbers, which is the order of array indexes, and then this array is rearranged to resemble a two-dimensional depth image to generate a mask image. In other words, the depth image, which was represented in one-dimensional distance order, is first rearranged in the order of the two-dimensional depth image, and then rearranged again in two dimensions to resemble the still image obtained from the shooting information. The mask image also contains information on the position index, depth information, and identification information identifying it as belonging to the cluster range. Compared to directly converting a one-dimensional array arranged in order of distance into a two-dimensional depth image, rearranging in order of address numbers allows for the generation of a mask image that resembles a two-dimensional depth image by grouping together a single row. In other words, a two-dimensional mask image can be quickly obtained.

[0041] The mask image shown in Fig. 12 is expressed in the format of "X0, Y0," "X1, Y0," "X2, Y0," etc. from the top left toward the horizontal direction, with the horizontal direction being the X axis and the vertical direction being the Y axis, similar to the extracted depth images of Fig. 5 and Fig. 6, and is arranged in the format of "X0, Y0," "X0, Y1," "X0, Y2," etc. from the top left toward the vertical direction. In other words, even if the arrangement of each pixel block is changed from the depth images of Fig. 5 and Fig. 6 as shown in Fig. 7, Fig. 10, and Fig. 11, it can be converted again into the mask image shown in Fig. 12 based on the information of the pixels assigned with the array indexes.

[0042] After the median value acquisition process and the difference value acquisition process, when a mask array is generated in the mask array setting process, clustering is performed from the depth information of the object Ob and the subject P only on the portion related to the specific region B1 in the fixed region depth acquisition process. This makes it possible to perform calculation processing that focuses more on the subject P. Since the cluster range belonging to the subject P can be selected in a state where the area occupied by the subject P is as large as possible in the specific region B1, a mask image of the subject P can be obtained with high accuracy. It can also be said that the mask image can be divided into the subject P and the portion other than the subject P with even higher accuracy from the portion that is provisionally determined in the median value acquisition process to be highly likely to indicate the distance to the subject P, who is the object.

[0043] After the mask image conversion process is performed, the specific depth image generation unit 6i performs a specific depth image generation process. The specific depth image generation process is executed in step s9 of the calculation process shown in FIG. 15. The specific depth image generation unit 6i performs a bit operation to multiply the depth image and the mask image, thereby obtaining a depth image in which depth information other than the cluster area has been removed. In other words, a specific depth image is obtained in which only the depth information of the part determined to be the subject P remains, compared to the depth image originally obtained from the imaging means 4 shown in . The specific depth image is image information in which the depth information of only the subject P remains, compared to the imaging information obtained from the imaging means 4. It can also be said that the specific depth image is in a state in which the depth information of the area other than the subject P has been removed, compared to the depth image obtained from the imaging means 4. In the shooting information obtained from the shooting means in the shooting information acquisition process, all of the images in Figure 3 had depth information, but by generating a specific depth image, while the still image is identical to that in Figure 3, the depth information attached to the still image that has undergone the specific depth image generation process will only be that of subject P.

[0044] As a result, the calculation is performed based on the median value of the non-zero (0) measurement error portion of the image extracted from the depth image obtained from the imaging means 4, and the calculation device 6 can determine that the portion identified as the cluster range is the subject P. Furthermore, by using the specific depth image, the calculation device 6 can accurately determine the distance to the identified subject P.

[0045] As explained above, the specific depth image is generated through the above steps, so the specific subject P is not determined using all of the imaging information obtained from the imaging means 4. By having the arithmetic device 6 distinguish between the subject P and parts other than the subject P for the part previously identified as the specific region B1, it is possible to perform high-speed calculations on the region occupied by the subject P included in the imaging information of the imaging means 4. Then, the relative positional relationship of the subject P from the imaging means 4 can be calculated at high speed without placing a heavy load on the arithmetic device 6.

[0046] By acquiring the specific depth image, the calculation device 6 can calculate the distance to the subject P based on the depth information attached only to the portion provisionally determined to be the subject P. In other words, since the imaging information obtained from the imaging means 4 only has depth information measuring the distance to the subject P, it becomes possible to accurately and quickly calculate the relative distance between the subject P and the imaging means 4 contained in the specific depth image. Furthermore, even when performing calculations using the relative distance between the subject P and the imaging means 4 in subsequent processes, the specific depth image can be obtained quickly and with a low load, so there is no burden. Furthermore, using the specific depth image makes it possible to simplify various calculations using the positional relationship with the subject P.

[0047] When generating a mask image, the mask image is generated as a two-dimensional image after converting from the position index order to the array index order, so processing delays are unlikely to occur. In other words, it is easy to ensure real-time performance for calculation processing. It is possible to realize an information processing method, information processing system, information processing program, and information processing device that can quickly and easily calculate the distance to an object without sacrificing real-time performance.

[0048] In the embodiment, a specific region B1 is extracted and a clustering calculation is performed focusing on that portion, making it possible to clearly and quickly distinguish between the subject P and other subjects. Furthermore, because a specific region B1 is extracted and a clustering process is performed on this portion, even if the subject P changes his or her posture sequentially, it is possible to clearly distinguish the boundary between the subject P and other subjects and accurately determine the distance of the portion recognized as the subject P.

[0049] In the above calculation method, instead of clustering all of the photographic information obtained from the photographing means 4, a portion of the photographic information is extracted based on the information with the bounding box B attached, and clustering is performed on this portion. Therefore, this is an information processing method that can also reduce the calculation load required to grasp the distance and position of the subject P who is moving successively.

[0050] The algorithm assumes that the object mainly captured within the bounding box B is the target object, performs clustering around the median, and removes depth information from the background, which allows for faster and less demanding calculations than clustering all of the image capture information acquired from the image capture means 4. The information processing device 6 that implements this calculation method can perform clustering calculations from acquired images even if it does not have advanced processing capabilities.

[0051] Furthermore, since clustering is performed within the range reflected within the bounding box B, it is possible to improve the accuracy of clustering for the subject P to be measured. In other words, it is possible to perform calculation processing at high speed, and also to allow the calculation device 6, which is an information processing device, to accurately grasp the distance to the subject P.

[0052] Furthermore, by using the above information processing method as a computer program for operating the arithmetic device 6, it is possible to make the arithmetic device 6 lightweight and operate at high speed. Furthermore, since high-speed processing is possible as a calculation method for operating the arithmetic device 6, it is possible to speed up the response of the entire system. In other words, even when various other operational controls are performed, it is possible to realize an information processing system that is capable of high-speed processing without affecting the arithmetic processing.

[0053] Furthermore, since the method is capable of high-speed processing of various calculations in order to accurately grasp the position of the subject P while distinguishing it from other objects Ob, it is possible to operate the calculation device lightly without placing a load on it. In other words, even when various controls are performed using the position of the subject P in addition to the position of the subject P, it is possible to realize an information processing method capable of high-speed processing without affecting these other calculation processes.

[0054] Furthermore, the response time for reflecting the calculation results can be made faster for the moving body 1 that needs to perform various types of operation control. For example, safety can be improved by having the traveling device 3 of the moving body 1 quickly perform safety operation control to avoid collision with another object Ob.

[0055] In the embodiments, it is possible to realize an information processing method, an information processing system, an information processing program, and an information processing device that can perform calculations at high speed while omitting a search process related to extraction and obtain calculation results. Note that various thresholds used during calculations are freely adjustable values, and can be adjusted and changed as appropriate depending on the implementation form.

[0056] In the embodiment, a clustering method called DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is used as a basis for extracting cluster ranges distributed around the median from the frequency distribution. However, the DBSCAN method does not necessarily have to be the basis. As shown in the embodiment, by performing calculation processing only on the region around the median of the target object in the frequency distribution, the position of the target object and the calculation result of that position can be obtained in real time.

[0057] In the embodiment, the depth information is described as being obtained using a twin-lens stereo camera 4b. However, this is not the only method. For example, a method of adding depth information to a captured image by using a monocular camera in combination with a distance measurement system that uses electromagnetic waves, including visible light and / or radio waves, such as LiDAR or millimeter-wave radar, may also be applied.

[0058] In the embodiment, the specific object to be measured is a person, but it does not necessarily have to be a person. For example, the specific object to be measured may be another moving object, or a marker having a specific pattern attached to a person or another moving object may be the specific object to be measured.

[0059] 14, the embodiment may further include a display device 7. In addition to the shooting information, the display device 7 may display at least one of depth information, a still image, a bounding box B, a still image of a specific region B1, a position index assigned to pixels constituting the still image, an array index, identification information indicating a cluster range, a frequency distribution table, a graph showing the frequency distribution, a mask image, and a specific depth image. Furthermore, the above information may be individually displayed or hidden by selecting whether to display or hide it on the display device 7.

[0060] The embodiment may further include an operation device 8, as shown in FIG. 14 . The operation device 8 may include an operation unit capable of changing various thresholds for the arithmetic device 6, or may include an operation unit for operating the display device 7. The operation device 8 may include an operation unit for operating the moving object 1. The operation device 8 may be included in the display device 7.

[0061] An embodiment will be further described. By obtaining a mask image or a specific depth image, it is possible to identify the center of gravity position for the area occupied by the subject P in the specific region or the captured image. The center of gravity identification unit 6j performs a center of gravity identification process to identify the center of gravity G from the area occupied by the subject P represented on the two-dimensional image. The center of gravity identification process is executed in step s10 of the calculation process shown in FIG. 15. The center of gravity identification process will be described as identifying the center of gravity G from the specific depth image obtained in the specific depth image generation process, but the center of gravity identification process may also be performed based on the mask image obtained in the mask image conversion process. The center of gravity identification process identifies the center of gravity G of the projected subject P based on the identification information identifying the subject P.

[0062] Fig. 13 is a diagram visually showing a specific depth image obtained through the above-mentioned various calculation processes based on the imaging information in Fig. 3. For ease of understanding, Fig. 13 shows the portion identified by the calculation device 6 as the subject P with diagonal lines. Note that in Fig. 13 and Fig. 3 to be described, the subject P is located slightly to the left of the imaging range.

[0063] The center of gravity identification unit 6j identifies the center of gravity G of the area identified as the subject P through the center of gravity identification step and marks it on the specific depth image. That is, the pixel block identified as having the center of gravity G is assigned information about the center of gravity G along with depth information, which is distance information, an array index, a position index, and identification information. In the embodiment, the center of gravity G is calculated near the waist on the left half of the body of the area identified as the subject P. In the case of Figure 13, the center of gravity G is located slightly to the left of the imaging range. That is, when viewed from the imaging means 4 mounted on the moving object 1, the center of gravity G of the subject P is located slightly to the left of the direction of travel. The distance to the center of gravity G can also be obtained from the depth information.

[0064] The depth information is the distance from the imaging means 4 to the subject P, and may be distance information obtained by the imaging means 4 as shown in Fig. 5, or distance information obtained by performing pre-processing such as filtering and / or noise removal on the distance information obtained by the imaging means 4 as shown in Fig. 6. In the embodiment, since the imaging means 4 is located at the front of the moving object 1, the depth information can be said to be the distance from the front of the moving object 1 to the subject P. Note that the positional relationship between the moving object 1 and the imaging means 4 is a relationship that can be known in advance, and therefore even if the imaging means 4 is located at the rear of the moving object 1, the distance between the imaging means 4 and the subject P can be easily converted to the distance between the front of the moving object 1 and the subject P.

[0065] Because the specific depth image generating step accurately obtains the area occupied by the subject P in the still image, the center of gravity G identified in the center of gravity identifying step can accurately obtain the center of gravity G of the area of ​​the subject P. For example, even if the subject P crouches in the direction of travel, bends his or her upper body downward toward the side, or spreads his or her limbs widely, the center of gravity G of the area occupied by the subject P can be grasped in the still image. The center of gravity identifying unit 6j can identify the left-right position of the center of gravity G with respect to the left-right center of the still image obtained from the imaging means 4. Furthermore, the front-to-back distance of the center of gravity G from the imaging means 4 can also be identified. In other words, the center of gravity identifying unit 6j can identify the left-to-right displacement amount, which is a numerical value indicating how far the center of gravity G is laterally away from the left-to-right center of the imaging range, and the front-to-back displacement amount, which is a numerical value indicating the front-to-back distance of the center of gravity G. The center of gravity identifying unit 6j transmits the left-to-right displacement amount and the front-to-back displacement amount to the control device 5.

[0066] The control device 5 mounted on the moving body 1 is capable of controlling the traveling device 3. Therefore, the moving body 1 can move forward and backward in a straight line and turn left and right by adjusting the rotation speed of each motor 3c of the traveling device 3 using the control device 5. The control device 5 has at least one CPU, similar to the information processing device which is the calculation device 6, and performs calculation processing to control the traveling device 3. The control device 5 may further include one or more of a ROM, a RAM, and a storage device. The control device 5 can also perform calculations using one or more of the CPU, ROM, RAM, and storage device which the calculation device 6 has.

[0067] The control device 5, which has received the left-right displacement amount and the front-rear displacement amount from the center of gravity identification unit 6j, performs a travel control step in which it controls the traveling device 3 so that the left-right displacement amount and the front-rear displacement amount become predetermined values. The travel control step is executed in step s11 of the calculation step shown in FIG. 15. The control device 5 controls the operation of the traveling device 3 so that the front-rear displacement amount becomes a first distance, which is a predetermined value, and the left-right displacement amount becomes a second distance, which is a predetermined value. In this embodiment, the second distance is set so that the center of gravity G is located at the left-right center of the captured still image, and the control is performed so that the center of gravity G is separated by a first distance, which indicates a fixed distance from the image capture means 4. In this embodiment, the first distance can be set to any value between 100 cm and 1000 cm, for example.

[0068] In the travel control process, the control device 5 compares the received lateral deviation amount with a predetermined second distance. In the example shown in FIG. 13, the center of gravity G is located slightly left of the lateral center of the still image, so the travel device 3 is controlled so that the lateral deviation amount becomes the second distance. In the embodiment described, the second distance is "0," so the moving body 1 is caused to turn so that it faces left in the traveling direction. For the turning operation, for example, the right motor 3c may be controlled to move the right traveling device 3 of the moving body 1 forward, or the left motor 3c may be controlled to move the left traveling device 3 of the moving body 1 backward. Alternatively, the left and right motors 3c may be controlled simultaneously so that the right traveling device 3 moves forward and the left traveling device 3 moves backward.

[0069] The control device 5 adjusts the traveling devices 3 arranged on the left and right of the moving body 1 so that the left-right deviation amount becomes a second distance. Furthermore, in the traveling control process, the control device 5 adjusts the front-to-rear deviation amount so that it becomes a first distance. If the distance at which the center of gravity G is located is greater than the specified first distance, that is, if the control device 5 determines that the subject P is farther away from the imaging means 4 than the first distance, the control device 5 controls the traveling devices 3 to move the moving body 1 forward until the distance at the center of gravity G reaches the first distance.

[0070] If the distance where the center of gravity G is located is smaller than the specified first distance, that is, if the control device 5 determines that the subject P is closer than the first distance from the imaging means 4, the control device 5 stops the traveling device 3. Then, the moving body 1 remains stopped until the distance to the center of gravity G of the subject P becomes the first distance. Then, when the calculation process and control process are completed up to step s11, the process returns to step s1 again, and the calculation process and control process are repeated in order.

[0071] The range of values ​​designated for the first distance is not limited to the above example. The first distance may be determined and set within the range of distances that are depth information that can be acquired by the imaging means 4. In this case, the moving object 1 can be made to run while maintaining a state in which the subject P is separated from the moving object 1 by an arbitrary distance.

[0072] Furthermore, although the lateral position of the center of gravity G is set to the center of the still image, it is not limited to this position. For example, it may be set to a position offset by an arbitrary second distance to either the left or the right from the center position. In this case, the moving object 1 can follow the target person P while being offset to the left or right in the traveling direction.

[0073] In the travel control step, the travel device 3 is operated so that the left-right deviation is the second distance, and then the travel device 3 is operated so that the front-rear deviation is the first distance, but this order is not limited to this. The control device 5 may operate the travel device 3 to adjust the first distance first, and then operate the travel device 3 so that the left-right deviation is the second distance, or may operate the travel device 3 to adjust the first distance and the second distance simultaneously.

[0074] In the description of the embodiment, the traveling device 3 is kept stopped when the center of gravity G is close to the first distance, which is the forward / backward distance, but this is not limited to this. For example, the traveling device 3 may be controlled to move the moving body 1 backward until the distance to the center of gravity G of the subject P becomes the first distance, and to stop the moving body 1 when the distance to the center of gravity G reaches the first distance.

[0075] In the description of the embodiment, the center of gravity identification step has been described as identifying the center of gravity G based on the specific depth image obtained in the specific depth image generation step. The center of gravity identification step does not have to follow this order, and the center of gravity G may be identified based on the mask image obtained in the mask image conversion step. In this case, a center of gravity identification step can be performed in which the center of gravity G of the subject P is calculated from the area occupied by the subject P obtained from the mask image. The center of gravity identification step identifies the center of gravity G of the imaged subject P based on identification information identifying the subject P. When the center of gravity G is identified in the mask image, the mask image is assigned information about the center of gravity G of the subject P in addition to information about the position index, depth information, and identification information. Then, based on the information about the center of gravity G contained in the mask image, the center of gravity G of the subject P in the specific depth image is converted, thereby making it possible to grasp the positional relationship of the subject P relative to the image capture device 4 and the moving object 1.

[0076] Although the centroid G identified in the centroid identification process has been described as being calculated through a depth array conversion process, a median value acquisition process, a difference value acquisition process, and a mask array setting process, this is not necessarily limited to this. At least, the subject P may be provisionally determined in the region identification process, and after the specific region depth is acquired, the centroid G for the area occupied by the subject P in the provisionally determined state may be calculated. In this case, the process of calculating the precise area occupied by the subject P can be omitted, allowing the centroid G to be calculated more quickly.

[0077] In the above embodiment, the arithmetic device 6 is separate from the control device 5. However, the arithmetic device 6 can be configured to be capable of fulfilling the role of the control device 5 that controls the traveling device 3, and the control device 5 can be configured to be capable of fulfilling the role of the information processing device 6 that processes information from the imaging means 4. This configuration allows the components of the control device 5 and the information processing device 6 to be shared and unified, thereby simplifying the system and device as a whole. When the arithmetic device 6 and the control device 5 are integrated, the role of the control device 5 to control the traveling device 3 is replaced by a traveling control unit.

[0078] The traveling device 5 quickly calculates and receives the relationship between the longitudinal distance and longitudinal deviation amount to the center of gravity G of the subject P and the lateral deviation amount between the lateral positions from the calculation device 6, which is an information processing device, and can therefore operate the traveling device 3 immediately without sacrificing real-time performance. By performing calculations related to information processing by the information processing device 6, which is a calculation device, quickly and lightly, the operation of the mobile object 1 can also respond immediately.

[0079] By setting the center of gravity G of the subject P as the target to be followed by the moving body 1, it is possible to reduce the possibility of losing sight of the target to be followed. The center of gravity G does not fluctuate significantly in a two-dimensional still image in response to the subject P's swaying limbs. In other words, the control device 5 can prevent chattering, which is an overreaction to the subject P's irregular movements, and cause the moving body 1 to properly follow the subject P.

[0080] The control method or calculation method for the moving body shown in the embodiment described above makes it possible to identify the subject P and quickly calculate the relationship between the longitudinal deviation amount of the front-rear distance to the center of gravity G of the subject P and the lateral deviation amount of the lateral position, without placing an unnecessary load on the calculation device 6. Furthermore, since the traveling device 3 is controlled based on each deviation amount, it is possible to provide a control method for operating the moving body 1 without sacrificing real-time performance.

[0081] The embodiment described above is a system using a control device 5 and an arithmetic device 6 that use a computer system, so it is possible to identify the subject P and quickly calculate the relationship between the longitudinal distance and longitudinal deviation amount to the center of gravity G of the subject P and the lateral deviation amount of the lateral position without imposing unnecessary load on the system operation. Then, since the traveling device 3 is controlled based on each deviation amount, it is possible to provide a system that operates the moving body 1 without sacrificing real-time performance.

[0082] The embodiment described above allows the control device 5 and the arithmetic device 6 to perform calculations without imposing a load on them, thereby enabling identification of the subject P and rapid calculation of the relationship between the forward / backward distance and the forward / backward displacement amount to the center of gravity G of the subject P and the lateral displacement amount of the lateral position. In other words, the control device 5 and the arithmetic device 6 can quickly and accurately determine the position of the subject P from the information captured by the imaging means 4, and calculate the forward / backward and lateral displacement amounts from this positional relationship, thereby enabling operation control of the moving body 1 without sacrificing real-time performance.

[0083] In the embodiment, the driving control process is executed based on the imaging information obtained from the imaging means 4. However, the driving control process does not necessarily have to be executed. For example, the calculation process and control process shown in FIG. 15 can be executed from step s1 to step s10, and can be transitioned and repeated. In this case, the subject P is identified, and the position information and center of gravity position information of the subject P are output from an interface provided in the calculation device or stored in a storage device, so that they can be used for control calculations other than driving. Note that the storage device described above can also be a medium that is removable from the calculation device 6 and the control device 5, and the form of the medium can be changed as appropriate depending on the implementation mode.

[0084] As described above, the present invention has been described by the above embodiments, but the descriptions and drawings that form part of this disclosure should not be understood to limit the present invention. It goes without saying that modifications of the embodiments, embodiments, and operating techniques based on this disclosure are possible within the scope of the claims. [Explanation of symbols]

[0085] 1: moving body, 2: machine body, 3: running device, 3a: driving wheel, 3b: driven wheel, 3c: motor, 4: shooting means, 4a: monocular camera, 4b: stereo camera, 5: control device, 6: calculation device (information processing device), 6a: shooting information acquisition unit, 6b: area identification unit, 6c: specific area depth acquisition unit, 6d: depth array conversion unit, 6e: median acquisition unit, 6f: difference value acquisition unit, 6g: mask array setting unit, 6h: mask image conversion unit, 6i: specific depth image generation unit, 7: display device, 8: operation device, B: bounding box (rectangular frame), B1: specific area, Ob: object, P: subject, S: running surface

Claims

1. The vehicle comprises a traveling device supported on a body of the vehicle, a photographing means attached to the body of the vehicle for acquiring photographing information including depth information which is distance information of a photographed object, and a control device capable of controlling the traveling device based on the photographing information, The control device includes: an area specifying unit that specifies an object included in the photographing information as a specific area; a specific region depth acquisition unit that acquires a depth image obtained by cutting out the specific region portion including depth information from the photographing information; a specific depth image acquisition unit that sets a target cluster range based on the depth information of the specific region portion, and obtains a specific depth image by multiplying the depth image by a mask image, which is an image in which depth information that is a distance other than the target has been removed; a centroid specifying unit that calculates a centroid from an occupied area of ​​an object arranged in the specific depth image; a travel control unit that controls the traveling device to maintain a distance from the target while directing the traveling device toward the target so that the center of gravity of the target located in the specific depth image is located at a predetermined position; A moving object comprising:

2. The control device includes a depth array conversion unit that rearranges the depth images into a single array in depth information order; a median value acquiring unit that calculates a median value of the depth information for the depth images rearranged in one row; a difference value acquisition unit that extracts difference values ​​equal to or greater than a threshold value from difference values ​​of distances obtained by differential processing of the depth images rearranged in a single row; a mask array setting unit that sets a cluster range from a difference value equal to or greater than the threshold value based on the median value of the depth information rearranged in one column; 2. The moving body according to claim 1, further comprising:

3. A control device capable of controlling a traveling device based on photographing information including depth information, which is distance information of a photographed object, comprising: The method for controlling a moving body includes: a region specifying step of specifying an object included in the photographing information as a specific region portion; a specific region depth acquisition step of acquiring a depth image obtained by cutting out the specific region portion including depth information from the photographing information; a specific depth image acquisition process for setting a target cluster range based on the depth information of the specific region portion, and obtaining a specific depth image by multiplying the depth image by a mask image, which is an image obtained by removing depth information that is a distance other than the target; a centroid specifying step of calculating a centroid from the area occupied by the object arranged in the specific depth image; a travel control step of controlling the traveling device to maintain a distance from the target while directing the traveling device toward the target so that the center of gravity of the target located in the specific depth image is located at a predetermined position; A method for controlling a moving object, comprising:

4. The method for controlling a moving body includes: rearranging the depth images into a single row in order of depth information; determining a median value of the depth information for the depth images rearranged in one row; extracting a difference value equal to or greater than a threshold value from the difference values ​​of distances obtained by differential processing of the depth images rearranged in a single row; setting a cluster range from difference values ​​equal to or greater than the threshold value based on the median value of the depth information rearranged in one column; The method for controlling a moving body according to claim 3, further comprising:

5. A control system for a moving body including a traveling device supported on a body, an imaging means attached to the body and adapted to acquire imaging information including depth information which is distance information of an object photographed, and a control device capable of controlling the traveling device based on the imaging information, The control system identifies an object included in the photographing information as a specific region portion; acquiring a depth image obtained by cutting out the specific region portion including depth information from the photographing information; A target cluster range is set based on the depth information of the specific region portion, and a specific depth image is obtained by multiplying the depth image by a mask image, which is an image in which depth information that is a distance other than the target is removed; Calculating a center of gravity from an area occupied by an object located in the specific depth image; controlling the traveling device to maintain a distance from the object while directing the traveling device toward the object so that the center of gravity of the object located in the specific depth image is located at a predetermined position; A control system for a moving object comprising:

6. The control system of the moving body rearranges the depth images into a single row in order of depth information; calculating a median value of the depth information for the depth images rearranged in one row; extracting a difference value equal to or greater than a threshold value from the difference values ​​of distances obtained by differential processing of the depth images rearranged in a row; setting a cluster range from difference values ​​equal to or greater than the threshold value based on the median value of the depth information rearranged in one column; The control system for a moving object according to claim 5, further comprising:

Images