Processing system, control system, moving body, photographing position determining method, and photographing position determining program

Abstract

To provide a processing system capable of appropriately determining a photographing position of an image for position estimation, a control system, a moving body, a photographing position determination method and a photographing position determination program.SOLUTION: A processing system includes a photographing position determination part, a movement distance calculation part, and a movement path plan part. The photographing position determination part acquires a first image photographed at a reference position by the moving body, and virtually moves the first image in a plurality of movement directions so as to reduce a deterioration element of the accuracy of estimating the position of the moving body to determine a plurality of photographing positions for estimating the position of the moving body. The movement distance calculation part calculates a movement distance between a plurality of photographing positions. The movement path plan part plans a movement path of the moving body between the plurality of photographing positions on the basis of the movement distance between the plurality of photographing positions.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The embodiments relate to a processing system, a control system, a moving object, a photographing position determining method, and a photographing position determining program. [Background technology]

[0002] As automation in logistics sites progresses, the use of mobile objects such as robots is being considered to automate and streamline the transportation of luggage. The guidance methods of mobile objects are broadly divided into route guidance and autonomous mobile objects. Route guidance mobile objects run along guides such as magnetic tapes and colored tapes installed on the floor. On the other hand, autonomous mobile objects recognize the environmental structure with sensors and run autonomously while estimating their own position. Compared to guidance mobile objects, autonomous mobile objects do not require the construction of a dedicated infrastructure, so that the travel route can be easily set. In addition, the travel route can be easily changed. However, in an environment where the layout of objects changes frequently, such as a logistics warehouse, the accuracy of the position estimation of the mobile object is likely to decrease.

[0003] In order to prevent a decrease in the accuracy of position estimation of an autonomous mobile body, a technology has been proposed that estimates its own position by comparing the current input image with multiple ceiling images registered by prior photographing the position where the mobile body is desired to stop and its vicinity. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-224280 A Summary of the Invention [Problem to be solved by the invention]

[0005] The image of the position where the moving object is to be stopped and the image of the vicinity must correspond to each other. However, depending on how the shooting position is decided, there may be an error in matching when determining the correspondence between the captured images. If such a matching error occurs, the accuracy of the position estimation will decrease.

[0006] The embodiments provide a processing system, a control system, a moving body, a shooting position determination method, and a shooting position determination program that can appropriately determine the shooting position of an image for position estimation. [Means for solving the problem]

[0007] A processing system according to one embodiment includes a photographing position determining unit, a moving distance calculating unit, and a moving path planning unit. The photographing position determining unit obtains a first image photographed by a moving object at a reference position, A plurality of second images having a common area in which any of the light-emitting objects captured in the first image is not captured are obtained. The first image is virtually moved in a plurality of moving directions to obtain an image of a moving object. Self For location estimation Around the reference position A plurality of image capture positions are determined. A movement distance calculation unit calculates a movement distance between the plurality of image capture positions. A movement path planning unit plans a movement path of the moving object between the plurality of image capture positions based on the movement distance between the plurality of image capture positions. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of a configuration of a control system for a moving object according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram showing the configuration of the shooting position determination unit in the first embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a hardware configuration of the photographing position determining device. [Figure 4] FIG. 4 is a diagram showing an example of the arrangement of the stereo cameras. [Diagram 5] FIG. 5 is a flow chart showing the process for creating a reference dictionary. [Figure 6] FIG. 6 is a diagram showing an example of the image capturing positions. [Figure 7]FIG. 7 is a diagram showing an example of feature point matching performed on an image in which deviations in feature points occur. [Figure 8] FIG. 8 is a flowchart showing the travel route planning process in the first embodiment. [Figure 9] FIG. 9 is a diagram showing an overview of the process of determining the shooting position in the first embodiment. [Figure 10] FIG. 10 is a block diagram showing the configuration of a shooting position determination unit in the second embodiment. [Figure 11] FIG. 11 is a flowchart showing a travel route planning process in the second embodiment. [Figure 12] FIG. 12 is a diagram showing an outline of a process for determining a shooting position in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment will be described with reference to the drawings. [First embodiment] 1 is a diagram showing an example of the configuration of a moving body control system according to the first embodiment. The moving body control system includes a photographing position determination device 1 and a moving body 60. The photographing position determination device 1 is configured to be able to communicate with the moving body 60.

[0010] The moving body 60 has a moving body control unit 61 and an imaging unit 62. In addition, although not shown, the moving body 60 has a drive mechanism required for movement.

[0011] The moving body control unit 61 performs control related to the movement of the moving body 60. For example, the moving body control unit 61 controls a drive mechanism for moving the moving body 60 to a specified reference position.

[0012] The photographing unit 62 has an image sensor, and mainly collects images of the external environment of the moving body 60. Then, the photographing unit 62 transmits the collected images to the photographing position determining device 1.

[0013] The photographing position determination device 1 plans a moving route when registering an image used for estimating the position of the moving object 60 in the dictionary storage unit 30. The photographing position determination device 1 also estimates the position of the moving object 60 based on images collected from the photographing unit 62 of the moving object 60, and controls the movement of the moving object 60 according to the result of the position estimation. The photographing position determination device 1 may be a host system of the moving object 60. The photographing position determination device 1 is, for example, a personal computer (PC). The photographing position determination device 1 has a processing unit 10, an input unit 20, a dictionary storage unit 30, a position estimation unit 40, and a position control unit 50.

[0014] The processing unit 10 plans a moving route when an image used for estimating the position of the moving object 60 is registered in the dictionary storage unit 30. Then, the processing unit 10 instructs the moving route to the moving object control unit 61. The processing unit 10 has a photographing position determination unit 11, a moving distance calculation unit 12, and a moving route planning unit 13.

[0015] The photographing position determination unit 11 determines a photographing position where an image used for estimating the position of the moving body 60 should be photographed. The photographing position includes a reference position to which the moving body 60 moves and a position in the vicinity of the reference position. The photographing position is determined so as to reduce factors that reduce the accuracy of position estimation between images collected at each photographing position. FIG. 2 is a block diagram showing the configuration of the photographing position determination unit 11 in the first embodiment. As shown in FIG. 2, the photographing position determination unit 11 includes a light emitter detection unit 111 and a photographing position calculation unit 112.

[0016] The light-emitting object detection unit 111 detects light-emitting objects in an image. A light-emitting object is an area in an image that has a predetermined brightness value or more, for example, a saturated brightness value. As will be described in detail later, the position of the moving object 60 is estimated from an image of the ceiling of the place where the moving object 60 is installed. Usually, such a ceiling is provided with lighting. Therefore, the light-emitting object in the embodiment is mainly lighting provided on the ceiling. As will be described later, a light-emitting object in an image is a factor that causes a decrease in the accuracy of estimating the position of the moving object 60.

[0017] The photographing position calculation unit 112 calculates a photographing position other than the reference position based on the detection result of the light emitting body detection unit 111. The photographing position calculation unit 112 calculates the photographing position as the amount of movement on the image relative to the image at the reference position.

[0018] The movement distance calculation unit 12 calculates the movement distance between each of the image capturing positions calculated by the image capturing position calculation unit 112. The movement distance calculation unit 12 performs a calculation to convert the image capturing position as the amount of movement on the image relative to the image at the reference position into the amount of movement in the coordinate system of the location where the moving body 60 is deployed.

[0019] The movement route planning unit 13 plans a movement route of the moving object 60 based on the shooting positions determined by the shooting position determination unit 11 and the movement distance calculated by the movement distance calculation unit 12. The movement route is a route that indicates the order in which the moving object 60 moves through the shooting positions.

[0020] The input unit 20 is configured to input various parameters required for planning a moving path to the processing unit 10. As will be described in detail later, the parameters include the camera parameters of the photographing unit 62 and the ceiling height of the location where the moving body 60 is deployed.

[0021] The dictionary storage unit 30 stores a dictionary that holds an image of the moving body 60 at a reference position, images collected at multiple shooting positions near the reference position, feature points and feature amounts extracted from each image, the correspondence between each image, and information on the collection positions of each image. The image of the moving body 60 at the reference position, images collected at multiple shooting positions near the reference position, feature points and feature amounts extracted from each image, and information on the correspondence between each image may be stored in the dictionary storage unit 30 by the movement path planning unit 13.

[0022] The position estimation unit 40 estimates the position of the moving object 60 based on an image of the outside of the moving object 60 acquired from the photographing unit 62 and an image registered in the dictionary storage unit 30 .

[0023] The position control unit 50 controls the operation of the moving object 60 in accordance with the position of the moving object 60 estimated by the position estimation unit 40 .

[0024] 3 is a diagram showing an example of the hardware configuration of the photographing position determination device 1. The photographing position determination device 1 has, for example, a CPU 101, an input device 102, a display device 103, a communication device 104, and a storage device 105 as hardware. The CPU 101, the input device 102, the display device 103, the communication device 104, and the storage device 105 are connected to a bus 106.

[0025] The CPU 101 is a processor that controls the overall operation of the photographing position determination device 1. The CPU 101 operates as the processing unit 10, the position estimation unit 40, and the position control unit 50 by executing a program stored in, for example, the storage device 105. The CPU 101 may be replaced with an MPU, a GPU, an ASIC, an FPGA, or the like. The CPU 101 may be a single CPU or the like, or may be multiple CPUs or the like.

[0026] The input device 102 is an input device such as a joystick, a touch panel, a keyboard, a mouse, etc. The input device 102 operates as the input unit 20. When the input device 102 is operated, a signal corresponding to the operation content is input to the CPU 101 via the bus 106. The CPU 101 performs various processes in response to this signal.

[0027] The display device 103 is a display device such as a liquid crystal display, an organic EL display, etc. The display device 103 displays various images.

[0028] The communication device 104 is, for example, a communication device for wireless LAN communication. The communication device 104 communicates with the mobile unit 60. The communication device 104 does not necessarily have to be a communication device for wireless LAN communication.

[0029] The storage device 105 is, for example, a storage device such as a hard disk drive or a solid state drive. The storage device 105 operates as the dictionary storage unit 30. The storage device 105 may also store various programs executed by the CPU 101.

[0030] The bus 106 is a data transfer path for transmitting data between the CPU 101 , the input device 102 , the display device 103 , the communication device 104 , and the storage device 105 .

[0031] Each of the components shown in FIG. 1 will be described in more detail below. The moving body 60 shown in FIG. 1 is, for example, an automated guided vehicle (AGV). The AGV is a trackless vehicle that automatically travels to a designated loading location and transports luggage loaded by a person or an automatic robot at the loading location to a designated unloading location. The moving body 60 can have any driving mechanism. For example, the moving body 60 may have a two-wheel drive mechanism, a four-wheel drive mechanism, or a caterpillar. The moving body 60 may also be a moving body that walks on two or more legs, or may be an air vehicle. The moving body 60 may not be trackless, but may be a line-trace type moving body that moves on a designated track. In the embodiment, the moving body 60 is assumed to be an AGV unless otherwise specified.

[0032] The moving body control unit 61 controls the drive mechanism for moving the moving body 60 to the designated reference position. At this time, the moving body control unit 61 may control the drive mechanism by determining the direction and distance required to move toward the designated reference position while recognizing its own position based on the position estimated by the photographing position determination device 1. The reference position of the moving body 60 may be provided from, for example, the photographing position determination device 1 as a higher-level system of the moving body 60, may be provided by calling up a previously set position, or may be provided by a person directly setting it. The reference position may include not only the spatial position of the moving body 60 but also the attitude of the moving body 60 at the reference position. In the embodiment, unless otherwise specified, the position and attitude of the image sensor mounted on the moving body 60 are the position and attitude of the moving body.

[0033] The photographing unit 62 collects images of the environment outside the moving body 60. In addition to the images of the outside of the moving body 60, the photographing unit 62 can preferably collect information that can measure or calculate the distance around the moving body 60. It is more preferable that the photographing unit 62 can collect information that can measure or calculate the distance around the moving body 60 in a plane. For example, a depth camera and a 3D-LADAR can collect planar distance images. A distance image is an image generated by converting a distance value into a brightness value. The depth camera may be of any type, such as a ToF (Time of Flight) type or a pattern irradiation type. In addition, even if it is a line measurement type laser range finder, it is possible to collect planar distance images by mechanically changing the measurement direction or changing the measurement direction of multiple units. Furthermore, as a method that does not directly measure, it is also possible to use a stereo camera or a monocular camera. If it is a stereo camera, the image can be converted into distance information by a stereo matching method. Even if it is a monocular camera, it is possible to realize stereo shooting similar to that of a stereo camera by changing the position and posture of the moving body, and it is possible to calculate distance information. The photographing unit 62 may be other than these. In the embodiment, unless otherwise specified, the photographing unit 62 is a stereo camera.

[0034] FIG. 4 shows an example of the arrangement of the stereo camera. In FIG. 4, the stereo camera 621 is installed in the center of the upper surface of the moving body 60. The stereo camera 621 is installed so that its optical center 622 is at a height h from the ground 72. The optical axis of the stereo camera 621 is tilted by an installation angle φ with respect to the ground 72. The angle of view of the stereo camera 621 is θ. At this time, the imaging range 80 of the stereo camera 621 is an imaging range 81 that mainly includes the ceiling 70 that is at a height H from the ground 72. As shown in FIG. 4, when the moving body 60 is approaching the wall 71, the imaging range 80 of the stereo camera 621 is imaging ranges 81 and 82 that include the ceiling 70 and the wall 71. The imaging range 80 of the stereo camera 621 may be changed depending on the use of the image collected by the stereo camera 621. For example, when the purpose is to estimate the position of the moving object 60, it is preferable that the subject in the images collected by the stereo camera 621 varies depending only on the position of the moving object 60 and does not vary over time. As shown in Fig. 4, the imaging range 80 of the stereo camera 621 is set to imaging ranges 81 and 82 that may include the ceiling 70 and the wall 71, so that images with little temporal variation of the subject and suitable for estimating the position of the moving object 60 can be collected. Of course, the installation position and orientation of the stereo camera 621 are not limited to those shown in Fig. 4.

[0035] Next, a description will be given of the dictionary storage unit 30. The dictionary storage unit 30 stores a reference dictionary in which information on the reference position required by the position estimation unit 40 is registered. Examples of registered information in the reference dictionary include the following. -Images of the reference position and its vicinity -Feature points and features of the image at and near the reference position Image matching results - Reference position and image positions nearby There are two methods for registering a reference dictionary when there are multiple reference positions: 1) a method in which registration information for all reference positions is collectively registered in one dictionary, and 2) a method in which information is divided into dictionaries for each reference position. In the method 1), it is not necessary to select a dictionary, but it takes time to estimate the position because it is necessary to match feature points for all images registered in the reference dictionary. In the method 2), the process for estimating the position is short because it is only necessary to process the reference dictionary in which the required reference position is registered. On the other hand, in the method 2), it is necessary to specify a reference dictionary to be used by a higher-level system and manually. Thus, the methods 1) and 2) for registering a reference dictionary each have advantages and disadvantages. Therefore, it is preferable to use the methods 1) and 2) for registering a reference dictionary separately as necessary.

[0036] 5 is a flow chart showing the process of creating a reference dictionary. The process of FIG. 5 is performed prior to estimating a position.

[0037] In step S101, the position control unit 50 of the photographing position determining device 1 starts moving the moving body 60 to one photographing position. The photographing position is a photographing position in the vicinity of a reference position, and is determined by a moving route planning process described later and registered in the dictionary storage unit 30. The position control unit 50 sequentially selects photographing positions according to the moving route and instructs the moving body 60 to move. In response to this, the moving body 60 starts moving autonomously.

[0038] In step S102, the position control unit 50 judges whether the moving body 60 has stopped based on the image of the outside world of the moving body 60 acquired by the image capturing unit 62. For example, the position control unit 50 calculates the speed of the moving body 60 based on the time change of the image acquired by the image capturing unit 62, and judges that the moving body 60 has stopped when the calculated speed is equal to or less than a threshold. Here, the moving body 60 does not necessarily stop when it arrives at the image capturing position. For example, the moving body 60 may be configured to stop at a corner or the like on the way to the image capturing position. In this case, too, when the speed of the moving body 60 is equal to or less than a threshold in the judgment of step S102, it is judged that the moving body 60 has stopped. The process is held until it is judged that the moving body 60 has stopped in step S102. If it is judged that the moving body 60 has stopped in step S102, the process proceeds to step S103.

[0039] In step S103, the position control unit 50 instructs the moving object 60 to capture an image and acquires an image from the moving object 60.

[0040] In step S104, the position control unit 50 determines whether or not the specified number of images have been acquired. For example, the position control unit 50 determines that the specified number of images have been acquired when images at all shooting positions have been acquired. In step S104, if the specified number of images have not been acquired, that is, if there are shooting positions at which images have not been acquired, the process returns to step S101. In this case, the position control unit 50 specifies a new shooting position and starts moving the moving body 60. In step S104, if the specified number of images have been acquired, the process proceeds to step S105.

[0041] In step S105, the position estimation unit 40 detects feature points from the acquired image. The position estimation unit 40 may detect feature points using Scale Invariant Feature Transform (SIFT), Accelerated KAZE (AKAZE), or the like.

[0042] In step S106, the position estimation unit 40 extracts a feature amount from each of the detected feature points. The position estimation unit 40 may extract the feature amount according to the method used for detecting the feature points.

[0043] In step S107, the position estimation unit 40 performs feature point matching between the image at the reference position and the images at each shooting position. Specifically, the position estimation unit 40 associates the feature points of each image so as to minimize the difference in the feature amount. The position estimation unit 40 may perform feature point matching using a method such as NN (Nearest Neighbor), k-NN, kd-tree, Hamming distance, etc.

[0044] In step S108, the position estimation unit 40 estimates the correspondence between the image at the reference position and the images at each of the shooting positions. For example, the position estimation unit 40 estimates the relative position and relative orientation of the moving object 60 at each of the shooting positions with respect to the reference position from the correspondence between the feature points of the image at the reference position and the image at the shooting position. Then, the position estimation unit 40 generates three-dimensional information for each image according to the principle of triangulation using the estimated relative position and relative orientation.

[0045] In step S109, the position estimation unit 40 registers, in a reference dictionary, the feature points and feature amounts for each image, the correspondence between the image at each shooting position and the reference position, and the coordinates of the reference position and the shooting position.

[0046] Here, when a plurality of images at reference positions are registered in the dictionary storage unit 30, the process of FIG. 5 may be repeated the same number of times as the number of images at reference positions.

[0047] Fig. 6 is a diagram showing an example of image capturing positions. In Fig. 6, a certain position of the moving body 60 is set as a reference position P0 as a reference. The reference position P0 is, for example, a position immediately in front of a cart. Positions P1, P2, P3, and P4 in the vicinity of the reference position P0 are set as capturing positions. At the capturing positions P1, P2, P3, and P4, the stereo camera 621 is adjusted and capturing is performed so that an image equivalent to a virtual image described later is captured.

[0048] In the embodiment, the position of the moving object 60 is estimated based on an image of the ceiling where the moving object 60 is placed. Here, the ceiling is provided with high-brightness light sources as light emitters, such as fluorescent lamps and light bulbs. Images of these high-brightness light sources may include saturated areas. A saturated area is an area where the brightness value of the target exceeds the dynamic range of an image sensor. Here, when the same position of the ceiling is photographed from different shooting positions, the position of the saturated area of ​​the image obtained at each shooting position changes slightly. Since the saturated area is easily detected as a feature point in an image, if the position of the saturated area changes slightly depending on the shooting position, the feature point that should be the same may be detected at a shifted position. This shift in the feature point may cause the feature point matching between images to be incorrectly performed. FIG. 7 is a diagram showing an example when feature point matching is performed on images in which a shift in the feature point occurs. Image I0 in FIG. 7 is an image obtained when the ceiling is photographed from a certain reference position registered in a reference dictionary, and image I1 is an image obtained when the ceiling is photographed from a shooting position near the reference position registered in a reference dictionary. In this case, as a result of matching between the image I0 and the image I1, there is a possibility that matching will occur at a position different from the position of the feature point that should be matched. FIG. 7 shows a state in which matching has occurred between the feature point RF1 and the feature point SF11, between the feature point RF2 and the feature point SF21, between the feature point RF3 and the feature point SF31, and between the feature point RF4 and the feature point SF41, but instead, matching has occurred between the feature point RF1 and the feature point SF12, between the feature point RF2 and the feature point SF22, between the feature point RF3 and the feature point SF32, and between the feature point RF4 and the feature point SF42. If an image with such erroneous feature point matching is registered in the reference dictionary, the accuracy of the position estimation of the moving body 60 by the position estimation unit 40 will decrease. In the embodiment, in order to reduce the decrease in the accuracy of the position estimation due to a light-emitting body, when a light-emitting body that is a factor of the decrease in the accuracy of the position estimation is captured in an image captured at a reference position, the remaining shooting positions are determined so that at least one light-emitting body is removed from the image to be captured.

[0049] Fig. 8 is a flowchart showing the planning process of a travel route in the first embodiment. The process of Fig. 8 is performed prior to the creation process of the reference dictionary in Fig. 5. Prior to the process of Fig. 8, the mobile object 60 moves to a reference position and acquires an image of the reference position. The movement of the mobile object 60 to the reference position may be performed autonomously based on an instruction from the position control unit 50, or may be performed manually by operating a joystick or the like.

[0050] In step S201, the light-emitting body detection unit 111 of the photographing position determination unit 11 detects a light-emitting body from an image of a reference position collected from the photographing unit 62 of the moving body 60. Then, the light-emitting body detection unit 111 clusters the area of ​​the light-emitting body in the image. For example, the light-emitting body detection unit 111 detects pixels whose luminance values ​​are saturated while scanning the image. Then, the light-emitting body detection unit 111 clusters the areas of pixels whose luminance values ​​are saturated, and attaches a label to each clustered area. Here, the detection of the light-emitting body may be performed using a learned model in which possible light-emitting bodies have been learned in advance by a method using deep learning disclosed in J. Redmon, S. Divvala, R. Girshick and A. Farhadi, "You Only Look Once: Unified, Real-Time Object Detection," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779-788, 2016. For example, as shown in FIG. 9, assume that four light-emitting objects are detected in an image I0 at the reference position, and labels L1, L2, L3, and L4 are attached to the respective light-emitting objects.

[0051] In step S202, the photographing position calculation unit 112 judges whether the process of determining the photographing positions is completed. For example, when the photographing positions are determined as many as the number of light-emitting bodies, the process of determining the photographing positions is judged to be completed. If the process is not completed in step S202, the process proceeds to step S203. If the process is completed in step S202, the process proceeds to step S205.

[0052] In step S203, the photographing position calculation unit 112 executes a process of determining the photographing position for the label Li (i=1, 2, ...). In the process of determining the photographing position, the photographing position calculation unit 112 calculates a virtual image movement amount where the light-emitting object with the label Li is not captured while virtually moving the image at the reference position so that the light-emitting object with the label Li is not captured. Specifically, the photographing position calculation unit 112 virtually moves the image I0 at the reference position in the up, down, left and right directions based on the center of the image so that the light-emitting object with the label Li is not captured. For example, the image I1 in FIG. 9 is an image obtained when the image I0 is virtually moved to the left. Similarly, the image I2 is an image obtained when the image I0 is virtually moved upward, the image I3 is an image obtained when the image I0 is virtually moved to the right, and the image I4 is an image obtained when the image I0 is virtually moved downward. Here, the determination of whether or not a light-emitting object is captured may be performed in the common area R1 between the images I1 and I0, the common area R2 between the images I2 and I0, the common area R3 between the images I3 and I0, and the common area R4 between the images I4 and I0. The greater the amount of movement of the virtual image, the fewer the light-emitting objects are captured. On the other hand, the greater the amount of movement of the virtual image, the fewer the common areas between the images captured at the respective capture positions. The reduction in the common areas between the images means the reduction in the corresponding feature points between the images. The reduction in the corresponding feature points between the images may be a cause of matching errors. Therefore, it is desirable to limit the amount of movement of the images to a range in which a certain degree of common area is secured. For example, the amount of movement of the images may be limited so that the area ratio of the common areas R1, R2, R3, and R4 to the entire image is 60% or more. The direction of movement of the images is not limited to the up, down, left, and right directions based on the center of the images, but may also include diagonal directions.

[0053] In step S204, the photographing position calculation unit 112 calculates the photographing position for the label Li. Specifically, the photographing position calculation unit 112 calculates the movement direction and movement amount of the image that is virtually moved in each of the up, down, left and right directions. Then, the photographing position calculation unit 112 selects the movement direction and movement amount that are the shortest distance from the image center among the calculated movement directions and movement amounts. On the other hand, when the selected movement direction and movement amount have already been selected in the process for another label, the photographing position calculation unit 112 selects the movement direction and movement amount that are the next shortest distance from the image center. After that, the process returns to step S202. For example, in FIG. 9, for the light-emitting body of the label L1, the movement direction and movement amount that obtain the image I1 are selected. This movement direction and movement amount correspond to the photographing position where the light-emitting body of the label L1 is not captured.

[0054] The process of steps S202-S204 is repeated until the number of shooting positions is calculated for the number of light-emitting objects. Then, in step S205 after the number of shooting positions is calculated for the number of light-emitting objects, the movement distance calculation unit 12 calculates the movement distance between each of the shooting positions. Below, a method of calculating the movement distance from the movement amount as the shooting position when the ceiling is assumed to be flat will be described.

[0055] First, it is assumed that the camera coordinate system is set based on the optical axis of the stereo camera 621. For example, the Z axis in the camera coordinate system is the direction of the optical axis of the stereo camera 621, and the XY axes are plane directions perpendicular to the optical axis of the stereo camera 621. Based on this definition, the coordinates (X c , Y c , Z c The projection equation from the image to the coordinates (x, y) on the image can be expressed as follows (Equation 1). Here, α is a constant. Also, f x , f y is the focal length of the stereo camera 621 expressed in pixels. x , c y are the coordinates of the optical center of the stereo camera 621 expressed in pixels. x , f y , cx , c y is information input in advance to the photographing position calculation unit 112 as a camera parameter.

number

[0056] Furthermore, when the ceiling is represented as a plane, the equation representing the plane of the ceiling in the camera coordinate system is expressed by the following (Equation 2).

number

[0057] Here, if the stereo camera 621 is installed so that only two parameters, the mounting angle φ of the stereo camera 621 and the height h of the stereo camera 621, can be changed, the coefficients a, b, c, and d in the plane equation of (Equation 2) can be expressed as (Equation 3).

number

number

[0058] In order to calculate the shooting position from the amount of movement of the image, it is necessary to convert the pixel positional relationship into the actual distance. Therefore, (Equation 1) is converted into the coordinates (X c, Y c , Z c ) is a solution, we obtain the following relationship (Equation 5).

number

number

[0059] In this way, the coordinates in the camera coordinate system (X c , Y c , Z c ) can be expressed as coordinates (x, y) in the image. Here, the camera coordinate system is inclined by an installation angle φ with respect to a coordinate system based on the plane on which the moving object 60 is placed. Therefore, the coordinates of the camera coordinate system expressed by (Equation 6) need to be converted to coordinates of a coordinate system based on the plane on which the moving object 60 is placed, i.e., the ground 72. The coordinates (X c, Y c , Z c ) to the coordinates (X hp , Y hp , Z hp ) is expressed by a rotation of the mounting angle φ from the camera coordinate system and a translation of the height (Hh), so it can be expressed as the following (Equation 7).

number

[0060] Returning now to the description of FIG. 8, in step S206, the movement path planning unit 13 plans the movement path of the moving body 60 based on the image capture position determined by the image capture position determination unit 11 and the movement distance calculated by the movement distance calculation unit 12. After that, the processing of FIG. 8 ends. The movement path may be planned so that the total movement distance is the shortest. As a method for planning a movement path that minimizes the total movement distance, Dijkstra's algorithm, A* (A-star) search algorithm, Bellman-Ford algorithm, Warshall-Floyd algorithm, etc. may be used. In these methods, the movement distance is treated as a cost, and a movement path is searched for so as to minimize the cost. In addition, when there are restrictions due to the drive mechanism of the moving body 60, restrictions due to the location where the moving body 60 is deployed, etc., these restrictions may also be treated as costs.

[0061] As described above, according to the first embodiment, in a system in which the position of a moving object is estimated by including images at a shooting position near the reference position in addition to an image at a reference position, when a light-emitting object is detected in an image at the reference position, another shooting position is automatically determined so that at least one light-emitting object is not captured. This suppresses the shift in the position of a feature point caused by the saturated region of the light-emitting object changing depending on the shooting position. In this way, images at the reference position and the shooting positions near it are appropriately collected, so that the deterioration of the accuracy of the position estimation of the moving object 60 is suppressed. In addition, there is no need to re-shoot at a shooting position near the reference position.

[0062] In the first embodiment, the movement path is planned so that the total movement distance between the image capturing positions is the shortest, thereby enabling a plurality of images to be captured efficiently in the vicinity of the reference position.

[0063] In this way, in the first embodiment, the determination of the image capturing positions, the planning of the travel route, and the capturing of images at each image capturing position can all be performed automatically.

[0064] Here, in the first embodiment, the moving body 60 and the photographing position determining device 1 are described as separate systems. However, the moving body 60 and the photographing position determining device 1 may be integrated. Also, a part of the photographing position determining device 1 may be separate from the photographing position determining device 1. For example, the dictionary storage unit 30, the position estimation unit 40, and the position control unit 50 may be provided in a server or the like outside the photographing position determining device 1.

[0065] [Second embodiment] A second embodiment will be described. FIG. 10 is a block diagram showing the configuration of the shooting position determination unit 11 in the second embodiment. As shown in FIG. 10, the shooting position determination unit 11 has a common area calculation unit 111a and a shooting position calculation unit 112a. In the second embodiment, the configuration other than the shooting position determination unit 11 is the same as in the first embodiment. Therefore, a description of the elements other than the shooting position determination unit 11 will be omitted.

[0066] The common area calculation unit 111a evaluates a common area between an image at a reference position and an image virtually moved from the image at the reference position. The evaluation criteria include values ​​such as the number of pixels in the common area, the area ratio of the common area to the entire image, the number of feature points in the common area, and the ratio of the number of feature points in the common area to the number of feature points in the entire image. When the evaluation criteria is the number of feature points, the common area calculation unit 111a may detect the feature points using SIFT (Scale Invariant Feature Transform), AKAZE (Accelerated KAZE), or the like. In addition, the common area calculation unit 111a may perform feature point matching using a method such as (Nearest Neighbor), k-NN, kd-tree, or Hamming distance.

[0067] The photographing position calculation unit 112a calculates a common area that satisfies the evaluation criteria calculated by the common area calculation unit 111a, and calculates the moving direction and moving amount of the virtual image in which this common area is obtained as the photographing position on the image. The moving direction of the image is, for example, up, down, left and right directions based on the image center at the reference position. The number of movements does not have to be one.

[0068] Fig. 11 is a flowchart showing a process for planning a travel route in the second embodiment. As in the first embodiment, prior to the process in Fig. 11, the moving object 60 moves to a reference position and acquires an image of the reference position.

[0069] In step S301, the common area calculation unit 111a determines a virtual image movement direction relative to the image at the reference position. For example, the common area calculation unit 111a determines a virtual movement direction in the up, down, left, and right directions based on the image center of the image at the reference position.

[0070] In step S302, the common area calculation unit 111a calculates a common area that satisfies the evaluation criteria in each moving direction in order to determine the shooting position. FIG. 12 is a diagram showing an outline of the process of determining the shooting position in the second embodiment. For example, when the evaluation criteria is the area of ​​the common area, the common area calculation unit 111a calculates each common area in FIG. 12, for example, a common area R1 with the image I0 in the image I1 when the image I0 at the reference position is virtually moved to the left, a common area R2 with the image I0 in the image I2 when the image I0 is virtually moved upward, a common area R3 with the image I0 in the image I3 when the image I0 is virtually moved to the right, and a common area R4 with the image I0 in the image I4 when the image I0 is virtually moved downward, each of which has an area ratio of, for example, 60% of the entire image. As described above, the common area may be calculated according to other evaluation criteria.

[0071] In step S303, the photographing position calculation unit 112a calculates a photographing position where the common area for each moving direction calculated by the common area calculation unit 111a can be obtained. Specifically, the photographing position calculation unit 112a calculates the movement amount of the virtual image where the common area calculated for each moving direction can be obtained. If the photographing position is such that the image I1 to image I4 shown in FIG. 12 can be photographed, a certain degree of common area is secured, so that the deterioration of the accuracy of the position estimation can be suppressed.

[0072] In step S304, the movement distance calculation unit 12 calculates the movement distance between the respective shooting positions. The movement distance calculation can be performed in the same manner as in the first embodiment. Therefore, the description will be omitted.

[0073] In step S305, the movement path planning unit 13 plans a movement path of the moving object 60 based on the image capture position determined by the image capture position determination unit 11 and the movement distance calculated by the movement distance calculation unit 12. Then, the process of FIG. 12 ends. The movement path can be planned in the same manner as in the first embodiment. Therefore, the description will be omitted.

[0074] As described above, according to the second embodiment, another photographing position is automatically determined so that a common area that satisfies a certain evaluation criterion is secured between the image of the reference position. This prevents the common area from being reduced, resulting in insufficient corresponding points being obtained. In this way, a decrease in the accuracy of the position estimation of the moving body 60 is prevented. Furthermore, in the second embodiment, no light-emitting body is detected. Therefore, the photographing position determination device of the second embodiment can be configured more simply than the photographing position determination device of the first embodiment.

[0075] [Other variations] Here, in the above-described embodiment, the subject of the image is assumed to be the ceiling of the location where the moving body 60 is deployed. This is because if it is a ceiling, the subject in the image changes only depending on the position of the moving body 60 and does not change over time. However, if the condition that the subject in the image changes only depending on the position of the moving body 60 and does not change over time is satisfied, the subject of the image is not limited to the ceiling of the location where the moving body 60 is deployed. For example, the subject of the image may be a wall, floor, etc. of the location where the moving body 60 is deployed.

[0076] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents described in the claims, as well as in the scope and spirit of the invention. [Explanation of symbols]

[0077] 1 Shooting position determination device, 10 Processing unit, 11 Shooting position determination unit, 12 Travel distance calculation unit, 13 Travel path planning unit, 20 Input unit, 30 Dictionary storage unit, 40 Position estimation unit, 50 Position control unit, 60 Moving object, 61 Moving object control unit, 62 Shooting unit, 101 CPU, 102 Input device, 103 Display device, 104 Communication device, 105 Storage device, 106 Bus, 111 Light-emitting object detection unit, 111a Common area calculation unit, 112 Shooting position calculation unit, 112a Shooting position calculation unit.

Claims

1. an image capturing position determination unit that acquires a first image captured by a moving object at a reference position, and virtually moves the first image in a plurality of moving directions to obtain a plurality of second images having a common area in which any of the light-emitting objects captured in the first image are not captured, thereby determining a plurality of image capturing positions around the reference position for estimating the own position of the moving object; a movement distance calculation unit that calculates movement distances between the plurality of shooting positions; a movement path planning unit that plans a movement path of the moving object between the plurality of image capture positions based on movement distances between the plurality of image capture positions; A processing system comprising:

2. The processing system according to claim 1 , wherein the photographing position determination unit calculates the photographing position of each of the second images based on camera parameters of a camera mounted on the moving body and an installation angle of the camera.

3. The processing system according to claim 1 , wherein the movement path planning unit plans the movement path so that a total movement distance of the moving object between the plurality of image capture positions is minimized.

4. The processing system according to claim 1 , wherein the first image is an image of a ceiling at the reference position.

5. a dictionary storage unit that stores a dictionary that holds an image of the moving object at a reference position, images collected at a plurality of photographing positions in the vicinity of the reference position, feature points and feature amounts extracted from each image, correspondences between each image, and information on the collection positions of each image; a position estimation unit that estimates a position of the moving object based on an image of the outside of the moving object acquired from an imaging unit and an image stored in the dictionary storage unit; The processing system according to claim 1 , further comprising:

6. A control system for a mobile body comprising: a processing system as described in any one of claims 1 to 5; and a mobile body moved by a drive mechanism, which takes photographs at the multiple shooting positions according to the movement path planned by the movement path planning unit of the processing system.

7. 7. A moving body comprising the processing system according to claim 1, the moving body being moved by a drive mechanism that captures images at the plurality of imaging positions according to the movement path planned by the movement path planning unit of the processing system.

8. acquiring a first image captured by a moving body at a reference position, and virtually moving the first image in a plurality of moving directions so as to obtain a plurality of second images having a common area in which any of the light-emitting objects captured in the first image are not captured, thereby determining a plurality of capturing positions around the reference position for estimating the own position of the moving body; Calculating a moving distance between the plurality of photographing positions; planning a movement path of the moving object between the plurality of image capture positions based on movement distances between the plurality of image capture positions; A method for determining a photographing position comprising the steps of:

9. acquiring a first image captured by a moving body at a reference position, and virtually moving the first image in a plurality of moving directions so as to obtain a plurality of second images having a common area in which any of the light-emitting objects captured in the first image are not captured, thereby determining a plurality of capturing positions around the reference position for estimating the own position of the moving body; Calculating a moving distance between the plurality of photographing positions; planning a movement path of the moving object between the plurality of image capture positions based on movement distances between the plurality of image capture positions; A shooting position determination program for causing a computer to execute the above.

Images