Self-location estimation system and self-location estimation method

Abstract

According to the present invention, a work information projection system estimates a position in a work site. This work information projection system comprises a camera (13), a stereo camera (14), and a control device. The camera (13) is disposed facing a first direction and acquires first environmental information that is information about objects disposed in the first direction and therearound. The stereo camera (14) is disposed facing a second direction and acquires second environmental information that is information about objects disposed in a second direction and therearound. The control device estimates a position in a first coordinate system on the basis of the first environmental information when it has been determined that the second environmental information is no longer acquired.

Description

[Technical field]

[0001] The present invention primarily relates to a self-location estimation system that estimates its own location in a workplace. [Background technology]

[0002] Patent Document 1 discloses a system including a worker terminal and a control device. The worker terminal is worn on the worker's head. The worker terminal includes a stereo camera and a projector. The control device creates map information indicating the shape and position of objects in the workplace based on images captured by the stereo camera. The control device further estimates the position of the stereo camera (i.e., the worker terminal) in the map information. The control device creates an image corresponding to the position of the worker terminal and transmits it to the projector. The projector projects the image created by the control device onto the workplace. This makes it possible to project an image corresponding to the position of the worker terminal to assist with work. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-98451 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the system of Patent Document 1, if the stereo camera is no longer able to acquire the shape and position of an object in the workplace, it becomes impossible to properly estimate the position of the worker terminal.

[0005] The present invention has been made in consideration of the above circumstances, and its main objective is to provide a system that can continue to estimate its own position even if one sensor is no longer able to capture the environment around the workplace. [Means for solving the problem]

[0006] The problem to be solved by the present invention has been described above. Next, the means for solving this problem and the effects thereof will be described.

[0007] According to a first aspect of the present invention, there is provided a position estimation system having the following configuration. That is, the position estimation system estimates a position in a workplace. The position estimation system includes a first sensor, a second sensor, and a control device. The first sensor is arranged facing a first direction and acquires first environmental information which is information on objects arranged in the first direction and in the surrounding area. The second sensor is arranged facing a second direction and acquires second environmental information which is information on objects arranged in the second direction and in the surrounding area. The control device acquires the first environmental information. from The self-position in the first coordinate system can be calculated, and the second environmental information from The control device is capable of calculating a self-position in a second coordinate system. When the control device is unable to acquire the second environmental information, the control device estimates the self-position based on the first environmental information.

[0008] According to a second aspect of the present invention, there is provided a position estimation method as follows. That is, the position estimation method includes a first acquisition step, a second acquisition step, and a position estimation step. In the first acquisition step, a first sensor acquires first environmental information, which is information on objects arranged in a work area facing a first direction and arranged in the first direction and its surroundings. In the second acquisition step, a second sensor acquires second environmental information, which is information on objects arranged in the work area facing a second direction and arranged in the second direction and its surroundings. In the position estimation step, when the second environmental information cannot be acquired, the first environmental information is acquired. from Estimate your location.

[0009] This makes it possible to continue estimating the position in the workplace even if the second sensor is no longer able to acquire environmental information. Effect of the Invention

[0010] According to the present invention, it is possible to provide a system that can continue estimating its own position even when one sensor is no longer able to acquire information about the environment around the workplace. [Brief description of the drawings]

[0011] [Figure 1] 1 is a schematic diagram of a work information projection system according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a block diagram of a work information projection system. [Diagram 3] 11 is a flowchart showing a calibration process. [Figure 4] FIG. 13 is a perspective view showing a state in which a calibration image is projected onto the calibration jig. [Diagram 5] FIG. 13 is a perspective view showing how a reference marker is placed on a workpiece and measurement is performed. [Figure 6] 11 is a flowchart showing a process during work. [Figure 7] 1A to 1C are diagrams for explaining a situation in which SLAM measurement is performed and a situation in which conversion information is calculated while the SLAM measurement is being performed. [Figure 8] 1A to 1C are diagrams for explaining a situation in which marker measurement is performed and a situation in which the process returns to SLAM measurement. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Next, an embodiment of the present invention will be described with reference to the drawings. First, an overview of a task information projection system 1 and a position recognition method will be described with reference to Figs.

[0013] The work information projection system (position recognition system) 1 of this embodiment is installed in a workplace. A workplace is a place where work is performed, such as a factory, an office, or a facility. Work is something that a worker does to an object by hand, using a tool, or operating a machine, such as assembling parts, painting, cleaning, transporting, etc. In this embodiment, a worker performs the task of assembling parts to a workpiece 31 arranged in a factory.

[0014] The work information projection system 1 recognizes a position in the workplace, and projects an auxiliary image 101 onto the workplace based on the recognized position. The auxiliary image 101 is an image that assists with work, such as an image for communicating the work content, work position, or work sequence to a worker. As shown in FIG. 1, the auxiliary image 101 of this embodiment is projected onto the workpiece 31, and indicates the name of the part to be assembled and the assembly position. The work information projection system 1 recognizes the position of the workpiece 31 (details will be described later), and projects the auxiliary image 101 onto an appropriate position on the workpiece 31.

[0015] 1 and 2, the work information projection system 1 includes a dolly 11, a projector 12, a camera 13, a stereo camera 14, and a control device 20. In the following description, unless otherwise specified, "position" includes not only the location where an object exists but also the direction in which the object faces. Thus, for example, the term positional relationship includes not only the relative position of two objects but also the relative orientation.

[0016] The dolly 11 has wheels and a base. The dolly 11 of this embodiment does not have a drive source, and moves within the workplace by being pushed by a worker. The dolly 11 may have a drive source and be capable of autonomous travel. The base supports the projector 12, the camera (first sensor) 13, the stereo camera (second sensor) 14, and the like. The dolly 11 may be capable of moving along rails provided on the floor, ceiling, or the like. The dolly 11 is not an essential component and may be omitted.

[0017] The projector 12 is placed on a dolly 11. The projector 12 projects an auxiliary image 101 under the control of the control device 20.

[0018] The camera 13 and the stereo camera 14 are fixed to the upper surface of the projector 12. As a result, the relative positions of the projector 12, the camera 13, and the stereo camera 14 do not change. In other words, the projector 12, the camera 13, and the stereo camera 14 move integrally. Note that the mounting method of the projector 12, the camera 13, and the stereo camera 14 may be different from that of this embodiment. For example, the projector 12, the camera 13, and the stereo camera 14 may be mounted on the dolly 11. Alternatively, a support member may be mounted on the dolly 11, and the projector 12, the camera 13, and the stereo camera 14 may be mounted on the support member.

[0019] In the following description, the directions in which the projector 12, camera 13, and stereo camera 14 face are the directions in which the optical axes extend from the respective devices. The optical axis is a straight line extending in the axial direction from a point passing through an optical element (imaging element, light-emitting element).

[0020] Camera 13 is a monocular camera with one image sensor. The facing direction of camera 13 (first direction, see FIG. 1) is substantially the same as the facing direction of projector 12. Substantially the same direction means, for example, that the difference between the two optical axes in a plan view is 15 degrees or less or 10 degrees or less, and the difference in the elevation angle between the two optical axes is 15 degrees or less or 10 degrees or less. From another perspective, the optical axis of camera 13 overlaps with the range in which projector 12 can project the projection light. In other words, the projection light forms a space that spreads from projector 12 as the origin, and this space intersects with the optical axis of camera 13. In this way, the facing direction of camera 13 is substantially the same as the facing direction of projector 12, so camera 13 can capture the image projected by projector 12.

[0021] Furthermore, a reference marker 51 and an interpolation marker 52 for position measurement are provided at appropriate positions in the factory (for example, on the surface of the workpiece 31). The image captured by the camera 13 may include the reference marker 51 or the interpolation marker 52.

[0022] The direction in which the stereo camera 14 faces (second direction, FIG. 1) is different from the first direction and also different from the direction in which the projector 12 faces. The different directions refer to, for example, a difference between the two optical axes being 30 degrees or more, 60 degrees or more, or 90 degrees or more in a plan view (the difference between the optical axes being 180 degrees in a plan view in this embodiment). Alternatively, the different directions may refer to a difference between the elevation angles of the two optical axes being 30 degrees or more or 60 degrees or more. Therefore, the stereo camera 14 cannot capture an image projected by the projector 12. The stereo camera 14 captures objects such as equipment, devices, tools, and workpieces 31 installed in a factory. In FIG. 1, a shelf 53 is shown as an example of an object captured by the stereo camera 14.

[0023] The stereo camera 14 includes two imaging elements, each of which captures an image of the workplace separately. The two imaging elements are disposed at an appropriate distance from each other. Each imaging element is, for example, a CCD (Charge Coupled Device). The two imaging elements operate in synchronization with each other and simultaneously capture the workplace to create a pair of image data. In this embodiment, it is assumed that information detected in real time will be projected as an auxiliary image, so that the stereo camera 14 preferably captures images multiple times per second, for example.

[0024] The stereo camera 14 also includes an image processing unit that processes the pair of image data. The image processing unit performs known stereo matching processing on the pair of image data obtained by the stereo camera 14 to determine the misalignment (parallax) between the corresponding positions of the images. The parallax increases in inverse proportion to the distance as the object is closer. Based on this parallax, the image processing unit creates a distance image in which distance information is associated with each pixel of the image data.

[0025] The stereo camera 14 has two imaging elements arranged in one housing. Alternatively, the stereo camera may be configured by combining two separate cameras. The image processing unit may be provided in a device (e.g., the control device 20) separate from the stereo camera 14.

[0026] The control device 20 is a computer equipped with a CPU, a ROM, a RAM, etc. The control device 20 in this embodiment is placed on the dolly 11. The control device 20 can communicate with the projector 12, the camera 13, and the stereo camera 14 via signal lines not shown. The control device 20 may be placed outside the dolly 11. In this case, the control device 20 communicates with the projector 12, the camera 13, and the stereo camera 14, for example, wirelessly.

[0027] The control device 20 acquires an image (first environmental information) captured by the camera 13 (first acquisition step) and acquires a distance image (second environmental information) captured by the stereo camera 14 (second acquisition step). The control device 20 creates an auxiliary image 101 based on these pieces of information and other information, and transmits it to the projector 12. As shown in FIG. 1, the control device 20 includes a communication device 21, an analysis unit 22, an image creation unit 23, and a projection control unit 24. The individual units included in the control device 20 are conceptually divided into units for each process performed by the control device 20 (for each function possessed by the control device 20). The control device 20 in this embodiment is realized by one computer, but the control device 20 may be composed of multiple computers. In this case, these multiple computers are connected via a network.

[0028] The communication device 21 is a communication module for communicating with the projector 12, the camera 13, and the stereo camera 14, and includes, for example, a connector for connecting a signal line, an antenna for wireless communication, etc. The communication device 21 receives images captured by the camera 13, receives images captured by the stereo camera 14, and transmits the auxiliary image 101 created by the image creation unit 23 to the projector 12.

[0029] When the image captured by the camera 13 includes the reference marker 51 or the interpolation marker 52, the analysis unit 22 performs known analysis processing based on the position, size, degree of distortion, etc. of the reference marker 51 or the interpolation marker 52 to calculate the relative position (self-position) of the camera 13 with respect to the reference marker 51 or the interpolation marker 52. Note that the self-position is the position of the device itself performing the measurement, and when the position is calculated based on the image captured by the camera 13, it indicates the position of the camera 13 (or the work information projection system 1).

[0030] Furthermore, the analysis unit 22 performs SLAM (Simultaneous Localization and Mapping) processing on the distance image captured by the stereo camera 14. The analysis unit 22 analyzes the distance image to generate map information (environmental map) indicating the shapes and positions of objects in the workplace, and estimates the position of the stereo camera 14 (self-position).

[0031] Since the SLAM process is well known, it will be briefly described below. That is, the analysis unit 22 sets appropriate feature points by analyzing the distance image and acquires the movement of the feature points. The analysis unit 22 extracts and tracks a plurality of feature points from the distance image to obtain data expressing the movement of the feature points in a plane corresponding to the image as vectors. The analysis unit 22 creates map information based on this data. As described above, the map information is data indicating the shape and position of an object in the work area, and more specifically, data indicating the three-dimensional positions of the extracted plurality of feature points (point cloud). The analysis unit 22 also estimates a change in the position of the stereo camera 14 based on the change in the position and distance of the input feature points and the position of the feature points in the map information. The SLAM process can also be performed based on an image captured by a monocular camera with one image sensor. Therefore, a monocular camera may be provided instead of the stereo camera 14.

[0032] The image creation unit 23 creates the auxiliary image 101. The control device 20 stores work information, which is information related to a work. The work information in this embodiment is the name of a part to be attached to the workpiece 31 and the attachment position of the part. The image creation unit 23 creates the auxiliary image 101 to be projected by the projector 12, based on the work information and a position estimated based on an image captured by the camera 13 or the stereo camera 14.

[0033] The projection control unit 24 transmits the auxiliary image 101 created by the image creation unit 23 to the projector 12, and causes the projector 12 to project the auxiliary image 101. In this manner, the auxiliary image 101 can be projected onto the workplace.

[0034] A detailed description will be given below regarding the estimation of the position of the projector 12. First, a calibration process performed before the operation will be described with reference to Figs.

[0035] First, the calibration jig 32 is placed in front of the projector 12. The calibration jig 32 is a member for calibrating the projector 12 and the camera 13. When the control device 20 receives an instruction to start a calibration process, it transmits a calibration image 102 to the projector 12. As a result, as shown in Fig. 4, the projector 12 projects the calibration image 102 onto the calibration jig 32 (S101). The camera 13 captures the calibration image 102 projected onto the calibration jig 32 (S102).

[0036] Next, the control device 20 calculates the relative positional relationship between the projector 12 and the camera 13 by performing a known analysis method based on the position, orientation, size, degree of distortion, etc. of the calibration image 102 contained in the image captured by the camera 13 (S103).

[0037] Next, the reference marker 51 is placed. The position where the reference marker 51 is placed becomes the origin of the reference coordinate system (marker coordinate system). In this embodiment, a coordinate system with a marker (reference marker 51 or interpolation marker 52) as the origin is called a marker coordinate system, and a coordinate system with the reference marker 51 as the origin is particularly called a reference coordinate system. The reference coordinate system is a coordinate system used for describing the above-mentioned work instructions, etc. The position of the origin of the reference coordinate system is arbitrary, but when performing work on the workpiece 31, for example, it is preferable to place the origin on the workpiece 31. This makes it difficult for the position where the auxiliary image 101 is projected to shift even if the position of the workpiece 31 changes slightly.

[0038] 5, after the reference marker 51 is set, the camera 13 captures an image of an area including the reference marker 51, and the stereo camera 14 captures an image of the surrounding work area (S104). Next, the control device 20 calculates conversion information between the reference coordinate system (marker coordinate system) and the SLAM coordinate system (S105).

[0039] The conversion information is information for performing coordinate conversion between the reference coordinate system and the SLAM coordinate system. In other words, the conversion information is information indicating the positional relationship between the camera 13 and the stereo camera 14. The control device 20 calculates the position of the camera 13 based on the reference marker 51 (i.e., in the reference coordinate system) based on the position, size, degree of distortion, etc. of the reference marker 51 included in the image captured by the camera 13. The control device 20 calculates the position of the stereo camera 14 in the SLAM coordinate system by performing the above-mentioned SLAM processing based on the distance image captured by the stereo camera 14. Then, the control device 20 calculates multiple sets of position information, with the position of the camera 13 in the reference coordinate system and the position of the stereo camera 14 in the SLAM coordinate system, which are captured at the same time, being regarded as one set.

[0040] Based on the multiple sets of position information calculated in this manner, conversion information is created. Specifically, the conversion information is calculated based on equation (1) shown in FIG. 3. The left side of equation (1) indicates the inner product of the "vector from the position of camera 13 to the position of stereo camera 14" and the "vector of the orientation of stereo camera 14". Since camera 13 and stereo camera 14 do not move relative to each other, the value of the inner product, λ, is constant. Therefore, by substituting multiple sets of position information into equation (1), it is possible to calculate the marker origin coordinate (t) in the SLAM coordinate system and the rotation coordinate (R) from the SLAM coordinate system to the marker coordinate system. These values ​​correspond to the conversion information.

[0041] By using the conversion information, the position and orientation of the stereo camera 14 in the SLAM coordinate system estimated by the SLAM process can be converted to the reference coordinate system. Specifically, the position of the stereo camera 14 in the reference coordinate system is expressed as R(P S -t), and the orientation of the stereo camera 14 in the reference coordinate system is Rd s By performing a similar calculation, it is also possible to calculate transformation information for transforming the marker origin coordinate system into the SLAM coordinate system.

[0042] Next, the process during work will be described with reference to Fig. 6 to Fig. 8. In the following description, estimating a position by performing SLAM processing based on a distance image captured by the stereo camera 14 is referred to as SLAM measurement. Estimating a position based on the interpolation markers 52 included in the image captured by the camera 13 is referred to as marker measurement.

[0043] In this embodiment, SLAM measurement is used as a base, and marker measurement is performed when SLAM measurement cannot be performed appropriately. When SLAM measurement cannot be performed appropriately, the stereo camera 14 cannot obtain appropriate information, specifically, when the stereo camera 14 captures only a flat wall and the number of feature points is small. This will be described in detail below.

[0044] The control device 20 estimates the position in the SLAM coordinate system from the distance image captured by the stereo camera 14, and converts it to a position in the reference coordinate system by applying the conversion information calculated in the calibration process (S201, position estimation step). Next, the control device 20 determines whether the interpolation marker 52 is included in the image captured by the camera 13 (S202).

[0045] 7, the image captured by the camera 13 does not include the interpolation marker 52. In such a case, the control device 20 creates an auxiliary image 101 based on the position of the reference coordinate system obtained by the stereo camera, and projects it from the projector 12 (S206).

[0046] In the situation described as "2. SLAM measurement, conversion information calculation" in FIG. 7, the image captured by the camera 13 includes the interpolation marker 52. In such a case, the control device 20 calculates conversion information for converting the marker coordinate system of the interpolation marker 52 into the reference coordinate system (S203). The process of calculating this conversion information is similar to step S105 of the calibration process. That is, since both the position in the marker coordinate system with the interpolation marker 52 as the origin and the position in the SLAM coordinate system are obtained, they are paired and substituted into formula (1). This makes it possible to calculate conversion information for converting the marker coordinate system of the interpolation marker 52 into the SLAM coordinate system. Also, the conversion information for converting the SLAM coordinate system into the reference coordinate system has already been calculated in step S105. Therefore, by combining the two pieces of conversion information, it is possible to calculate conversion information for converting the marker coordinate system of the interpolation marker 52 into the reference coordinate system.

[0047] Next, the control device 20 judges whether the feature points of the SLAM processing are equal to or smaller than a threshold value (S204). In other words, the control device 20 judges whether the second environmental information has been acquired. In the situation described as "2. SLAM measurement, conversion information calculation" in FIG. 7, there are a sufficient number of feature points because the stereo camera 14 detects the shelf 53, etc. In such a case, position measurement based on the SLAM measurement is used, without using the position measurement using the interpolation marker 52. In other words, the control device 20 creates an auxiliary image 101 based on the position of the reference coordinate system obtained by the stereo camera 14, and projects it from the projector 12 (S206).

[0048] In other words, the processing of step S204 has a judgment condition for judging whether the information acquired from the stereo camera 14 or the information calculated based on it is appropriate, and the control device 20 judges whether the judgment condition is satisfied.

[0049] In the situation described as "3. Marker + camera measurement" in FIG. 8, since no object such as shelf 53 exists within the range captured by stereo camera 14, the feature points of SLAM processing are below the threshold. In such a case, position measurement based on SLAM measurement is not used, but position measurement using interpolation marker 52 is used. Therefore, control device 20 converts the position of the marker coordinate system obtained from the image captured by camera 13 (image including interpolation marker 52) into the position of the reference coordinate system (S205, position estimation step). This conversion is performed using the conversion information calculated in step S203. After that, control device 20 creates auxiliary image 101 based on the position of the reference coordinate system obtained by camera 13, and projects it from projector 12 (S207).

[0050] It is possible to predict in advance the situation in which the feature points obtained by the SLAM processing are equal to or smaller than the threshold value. Therefore, in this embodiment, the interpolation marker 52 is placed near the camera 13 in the situation in which the feature points obtained by the SLAM processing are equal to or smaller than the threshold value. This makes it possible to perform at least one of the SLAM measurement and the marker measurement.

[0051] The flowchart shown in Fig. 6 is executed repeatedly. Therefore, when the situation described as "4. SLAM measurement" in Fig. 8 occurs after the situation described as "3. Marker + camera measurement" in Fig. 8, that is, when the feature points of the SLAM processing exceed the threshold value, the position measurement using the interpolation marker 52 is stopped, and the position measurement based on the SLAM measurement is used.

[0052] As a result, by using SLAM measurement as a base while performing marker measurement in situations where the position estimation accuracy using SLAM measurement decreases, it is possible to maintain high position estimation accuracy in the workplace.

[0053] As described above, the task information projection system 1 of this embodiment performs a position estimation method for estimating a position in a work area. The task information projection system 1 includes a camera 13, a stereo camera 14, and a control device 20. The camera 13 is arranged facing a first direction and acquires first environmental information, which is information on objects arranged in the first direction and in the surroundings (in the first direction). The stereo camera 14 is arranged facing a second direction and acquires second environmental information, which is information on objects arranged in the second direction and in the surroundings (in the second direction). The control device 20 can calculate its own position in a first coordinate system based on the first environmental information, and can calculate its own position in a second coordinate system based on the second environmental information. When the control device 20 is unable to acquire the second environmental information, it estimates its position in the first coordinate system based on the first environmental information.

[0054] This makes it possible to continue estimating the position in the workplace even when the stereo camera 14 is no longer able to acquire environmental information.

[0055] In the position estimation system of this embodiment, the stereo camera 14 is disposed so as to move integrally with the camera 13.

[0056] This allows the positional relationship between the camera 13 and the stereo camera 14 to remain unchanged, making it possible to more appropriately estimate the position of the workplace.

[0057] In the position estimation system of this embodiment, the first direction and the second direction are different directions.

[0058] This allows one sensor to complement the other sensor since the detection ranges of camera 13 and stereo camera 14 are significantly different.

[0059] In the working information projection system 1 of this embodiment, when the control device 20 determines that the first environmental information and the second environmental information can be acquired, it calculates conversion information for converting between the first coordinate system and the second coordinate system.

[0060] This makes it possible to continue using the same coordinate system even when the sensor used for position measurement is switched.

[0061] The work information projection system 1 of this embodiment includes a projector 12 that projects an auxiliary image 101 that assists in the work onto a work site. The control device 20 creates the auxiliary image 101 according to a position in the work site, and transmits the auxiliary image 101 to the projector 12.

[0062] In particular, the work information projection system 1 of this embodiment is unlikely to lose track of its own position, and therefore can achieve high reliability.

[0063] In the working information projection system 1 of this embodiment, the optical axis of the camera 13 overlaps with an area where the projector 12 can project the auxiliary image 101. The camera 13 is a camera 13 that captures an image of an area including a marker provided in a work area. The stereo camera 14 is a stereo camera 14 that captures an image of an object placed in the second direction and around the second direction.

[0064] As a result, the auxiliary image 101 projected by the projector 12 is not captured by the stereo camera 14, and therefore the auxiliary image 101 can be prevented from being recognized as a feature point.

[0065] In the work information projection system 1 of this embodiment, when the control device 20 determines that the first environmental information and the second environmental information can be acquired while the projector 12 is projecting the auxiliary image 101 onto the work area, the control device 20 calculates conversion information for converting between the first coordinate system and the second coordinate system.

[0066] This allows the necessary conversion information to be calculated during work, reducing the amount of preparation required in advance.

[0067] In the position estimation system of this embodiment, conversion information is calculated based on a formula for calculating the inner product of a vector from the position of camera 13 to the position of stereo camera 14 and a vector indicating the orientation of stereo camera 14.

[0068] This allows the conversion information to be calculated through simple processing.

[0069] Although the preferred embodiment of the present invention has been described above, the above configuration can be modified, for example, as follows.

[0070] In the above embodiment, the first sensor facing substantially the same direction as the projector 12 is the camera 13, and the second sensor on the opposite side is the stereo camera 14. Alternatively, the first sensor may be the stereo camera 14, and the second sensor may be the camera 13. In this case, the interpolation marker 52 is installed at an appropriate position in the workplace, instead of on the workpiece 31.

[0071] In the above embodiment, the first sensor and the second sensor (i.e., the camera 13 and the stereo camera 14) face in different directions. Alternatively, if the detection ranges of the first sensor and the second sensor are different (for example, the detection distances are different, the horizontal widths of the detection ranges are different, the elevation and depression widths of the detection ranges are different, etc.), the first sensor and the second sensor may face in the same direction. In other words, the first direction and the second direction may be the same direction. In this case, the second sensor can detect a range that cannot be detected by the first sensor, so the same processing as in the above embodiment can be applied.

[0072] In the above embodiment, the stereo camera 14 has been described as an example of a three-dimensional measuring device, but other devices, such as LiDAR (Light Detection and Ranging), may be used. LIDAR is a technology that acquires the positions and shapes of surrounding objects by emitting radio waves in various directions and measuring the time it takes to receive the reflected waves of the radio waves.

[0073] The flowcharts shown in the above embodiment are merely examples, and some processes may be omitted, some processes may be changed, or new processes may be added. For example, in the above embodiment, SLAM measurement is performed with priority over SLAM measurement and marker measurement as shown in FIG. 6, but instead, marker measurement may be performed with priority. In other words, when the image captured by the camera 13 includes the interpolation marker 52, the marker measurement is always performed, and when the image captured by the camera 13 does not include the interpolation marker 52, the SLAM measurement is performed.

[0074] In the above embodiment, the estimated position is converted into a reference coordinate system, but this process may be omitted.

[0075] The position estimation system of the present invention is not limited to a work information projection system, but can be applied to various systems that estimate their own position. For example, the present invention can be applied to a traveling system that moves autonomously. [Explanation of symbols]

[0076] 1. Work information projection system (position recognition system) 11 Trolley 12 Projector 13 Camera (first sensor) 14 Stereo camera (second sensor, 3D measurement device) 20 Control device

Claims

1. A position estimation system for estimating a position in a workplace, a first sensor arranged to face a first direction and to acquire first environmental information which is information on objects arranged in the first direction and around the first direction; a second sensor arranged to face a second direction and to acquire second environmental information which is information on objects arranged in the second direction and around the second direction; a control device capable of calculating a self-position in a first coordinate system from the first environmental information, and capable of calculating a self-position in a second coordinate system from the second environmental information, and estimating a self-position from the first environmental information when the second environmental information cannot be acquired; A position estimation system comprising:

2. 2. The position estimation system of claim 1, A position estimation system, wherein the second sensor is arranged to move integrally with the first sensor.

3. 3. A position estimation system according to claim 1, further comprising: A position estimation system, wherein the first direction is different from the second direction.

4. A position estimation system according to any one of claims 1 to 3, A position estimation system characterized in that the control device calculates conversion information for converting between the first coordinate system and the second coordinate system when the first environmental information and the second environmental information can be acquired.

5. A position estimation system for estimating a position in a workplace, comprising: a first sensor arranged to face a first direction and to acquire first environmental information which is information on objects arranged in the first direction and around the first direction; a second sensor arranged to face a second direction and to acquire second environmental information which is information on objects arranged in the second direction and around the second direction; a control device capable of calculating a self-position in a first coordinate system based on the first environmental information, and capable of calculating a self-position in a second coordinate system based on the second environmental information, and estimating a self-position based on the first environmental information when the second environmental information cannot be acquired; a projector that projects an auxiliary image that assists in a task onto the work area; Equipped with A position estimation system, characterized in that the control device creates the auxiliary image according to a position in the workplace and transmits the auxiliary image to the projector.

6. 6. The position estimation system of claim 5, an optical axis of the first sensor overlaps with an area onto which the projector can project the auxiliary image; the first sensor is a camera that captures an image of an area including a marker provided in the work area, A position estimation system characterized in that the second sensor is a three-dimensional measuring device for acquiring the shape and position of objects arranged in the second direction and around the second direction.

7. 7. The position estimation system of claim 6, a control device that calculates conversion information for converting between the first coordinate system and the second coordinate system when the first environmental information and the second environmental information can be acquired while the projector is projecting the auxiliary image onto the workplace, said control device calculating conversion information for converting between the first coordinate system and the second coordinate system.

8. A position estimation system for estimating a position in a workplace, comprising: a first sensor arranged to face a first direction and to acquire first environmental information which is information on objects arranged in the first direction and around the first direction; a second sensor arranged to face a second direction and to acquire second environmental information which is information on objects arranged in the second direction and around the second direction; a control device capable of calculating a self-position in a first coordinate system based on the first environmental information, and capable of calculating a self-position in a second coordinate system based on the second environmental information, and estimating a self-position based on the first environmental information when the second environmental information cannot be acquired; Equipped with the control device calculates conversion information for converting between the first coordinate system and the second coordinate system when the first environmental information and the second environmental information can be acquired; A position estimation system comprising: a position estimation unit that calculates the conversion information based on an equation that calculates an inner product of a vector from the position of the first sensor to the position of the second sensor and a vector indicating an orientation of the second sensor.

9. a first acquisition step of acquiring, by a first sensor, first environmental information that is information on objects arranged in the first direction and around the first sensor, the first environmental information being arranged in the first direction; a second acquisition step of acquiring, by a second sensor, second environmental information that is information on objects arranged in the second direction and around the second sensor, the second environmental information being arranged in the second direction; a position estimation step of estimating a self-position from the first environmental information when a transition is made to a state in which the second environmental information cannot be acquired after a state in which the self-position is estimated from the second environmental information; A position estimation method comprising:

Images