Self-position estimation system and self-position estimation method

Abstract

A work information projection system for estimating a position in a work site. The work information projection system includes a camera (13), a stereo camera (14), and a control device. The camera (13) is disposed toward a first direction, and acquires first environment information that is information of an object disposed in the first direction and around. The stereo camera (14) is disposed toward a second direction, and acquires second environment information that is information of an object disposed in the second direction and around. In a case where it is determined that the second environment information cannot be acquired, the control device estimates a position in a first coordinate system based on the first environment information.

Description

Technical Field

[0001] This invention relates to a self-position estimation system, which is used to estimate the self-position at a work site. Background Technology

[0002] Patent Document 1 discloses a system comprising an operator terminal and a control device. The operator terminal is mounted on the operator's head. The operator terminal includes a stereo camera and a projector. The control device generates map information based on images captured by the stereo camera, the map information displaying the shape and position of an object at the work site. The control device further estimates the position of the stereo camera (i.e., the operator terminal) within the map information. The control device generates an image corresponding to the position of the operator terminal and transmits it to the projector. The projector projects the image generated by the control device onto the work site. Thus, it is possible to project an image corresponding to the position of the operator terminal to assist in the operation.

[0003] [Existing Technical Documents]

[0004] [Patent Literature]

[0005] Patent Document 1: Japanese Patent Application Publication No. 2020-98451 Summary of the Invention

[0006] The technical problem that the invention aims to solve

[0007] In the system described in Patent Document 1, the position of the operator's terminal cannot be properly estimated when the stereo camera cannot obtain the shape and position of an object in the work site.

[0008] The present invention was developed in view of the above-mentioned situation, and its main objective is to provide a system that can continue to estimate its own position even when a sensor is unable to obtain information about the environment around the work site.

[0009] Technical means used to solve the problem

[0010] The problem that this invention aims to solve has been described above. The means used to solve this problem and their effects are described below.

[0011] According to a first aspect of the present invention, a position estimation system with the following structure is provided. That is, the position estimation system is used to estimate the position of a work site. The position estimation system includes a first sensor, a second sensor, and a control device. The first sensor is configured facing a first direction and acquires first environmental information, which is information about objects disposed in the first direction and in the surrounding environment. The second sensor is configured facing a second direction and acquires second environmental information, which is information about objects disposed in the second direction and in the surrounding environment. The control device is capable of calculating its own position in a first coordinate system based on the first environmental information, and is capable of calculating its own position in a second coordinate system based on the second environmental information. If the second environmental information is unavailable, the control device estimates its own position based on the first environmental information.

[0012] According to a second aspect of the present invention, a position estimation method is provided below. That is, the position estimation method includes a first acquisition step, a second acquisition step, and a position estimation step. In the first acquisition step, a first sensor is positioned at the work site facing a first direction to acquire information about objects positioned in that first direction and in the surrounding area, i.e., first environmental information. In the second acquisition step, a second sensor is positioned at the work site facing a second direction to acquire information about objects positioned in that second direction and in the surrounding area, i.e., second environmental information. In the position estimation step, if the second environmental information cannot be acquired, the position is estimated based on the first environmental information.

[0013] Therefore, even if the second sensor cannot obtain environmental information, the location of the work site can still be estimated.

[0014] The benefits of invention

[0015] According to the present invention, a system can be provided that can continue to estimate its own position even when a sensor cannot obtain information about the environment around the work site. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of a job information projection system according to an embodiment of the present invention;

[0017] Figure 2 This is a block diagram of the job information projection system;

[0018] Figure 3 This is a flowchart showing the calibration process;

[0019] Figure 4 It is a three-dimensional view showing the projection of the calibration image onto the calibration guide plate;

[0020] Figure 5It is a three-dimensional view showing the measurement process performed with reference marks set on the workpiece;

[0021] Figure 6 This is a flowchart showing the processing of the task;

[0022] Figure 7 This is a diagram illustrating the SLAM measurement process and the simultaneous calculation of conversion information during SLAM measurement; and

[0023] Figure 8 This is a diagram illustrating the status of the marked measurement and the status of returning to SLAM measurement. Detailed Implementation

[0024] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. First, referring to... Figure 1 as well as Figure 2 Briefly describe the job information projection system 1 and the location recognition method.

[0025] The work information projection system (location recognition system) 1 in this embodiment is located at the work site. The work site refers to a place used for performing work, such as a factory, office, or facility. A work operation refers to an operator performing certain actions on an object using manual methods, tools, or operating machinery, such as assembling parts, applying paint, cleaning, or transporting. In this embodiment, the operator performs the work of assembling parts onto workpieces 31 located within the factory.

[0026] The work information projection system 1 is used to identify the location of the work site and project an auxiliary image 101 onto the work site based on the identified location. The auxiliary image 101 refers to an image that assists in the work, such as an image used to convey the work content, work location, or work sequence to the operator. Figure 1 As shown, in this embodiment, the auxiliary image 101 is projected onto the workpiece 31, displaying the names of the assembled parts and their assembly positions. The work information projection system 1 identifies the position of the workpiece 31 (details to be described later) and projects the auxiliary image 101 onto the appropriate position of the workpiece 31.

[0027] like Figure 1 as well as Figure 2 As shown, the work information projection system 1 includes a trolley 11, a projector 12, a camera 13, a stereo camera 14, and a control device 20. In the following description, unless otherwise stated, "position" includes not only the location where an object exists, but also the direction in which the object is facing. Thus, for example, the term "positional relationship" includes not only the relative positions of two objects, but also the relative orientations of these objects.

[0028] The trolley 11 includes wheels and a base. In this embodiment, the trolley 11 does not have a drive source and moves within the work area by being pushed by an operator. Alternatively, the trolley 11 may have a drive source and be able to move independently. The base supports the projector 12, the camera (first sensor) 13, and the stereo camera (second sensor) 14, etc. The trolley 11 can also move along tracks provided on the floor or ceiling. Furthermore, the trolley 11 is not an essential component and can be omitted.

[0029] The projector 12 is mounted on the trolley 11. The projector 12 projects auxiliary images 101 under the control of the control device 20.

[0030] Camera 13 and stereo camera 14 are fixed on top of projector 12. Therefore, the relative positions of projector 12, camera 13, and stereo camera 14 do not change. In other words, projector 12, camera 13, and stereo camera 14 move as a unit. Furthermore, the mounting method of projector 12, camera 13, and stereo camera 14 can also differ from this embodiment. For example, projector 12, camera 13, and stereo camera 14 can be mounted on trolley 11. Alternatively, a support member can be mounted on trolley 11, and projector 12, camera 13, and stereo camera 14 can be mounted on the support member.

[0031] In the following description, the direction in which the projector 12, camera 13, and stereo camera 14 face refers to the direction in which the optical axis extends from each of the devices. The optical axis is a straight line formed by extending axially from a point passing through an optical element (photographic element, light-emitting element).

[0032] Camera 13 is a single-lens reflex camera with a single photographic element. The direction in which camera 13 faces (first direction, reference direction) Figure 1 The direction in which the camera 13 faces is substantially the same as that of the projector 12. "Substantially the same" means, for example, that the difference between the two optical axes when viewed from above is equal to or less than 15 degrees or equal to or less than 10 degrees, and the difference in the elevation angle of the two optical axes is equal to or less than 15 degrees or equal to or less than 10 degrees. To put it another way, the optical axis of the camera 13 overlaps with the range of projection light that the projector 12 can project. That is, the projection light forms a space extending from the projector 12, but this space intersects with the optical axis of the camera 13. Thus, since the direction in which the camera 13 faces is substantially the same as that of the projector 12, the camera 13 is able to capture the image projected by the projector 12.

[0033] Additionally, reference marks 51 and interpolation marks 52 for position measurement are provided at appropriate locations within the factory (e.g., on the surface of workpiece 31). Sometimes, reference marks 51 or interpolation marks 52 can also be included in images captured by camera 13.

[0034] The direction in which the stereo camera 14 is facing (the second direction, Figure 1 The direction is different from the first direction and also different from the direction in which the projector 12 is facing. A different direction means, for example, that the difference between the two optical axes is equal to or greater than 30 degrees, equal to or greater than 60 degrees, or equal to or greater than 90 degrees when viewed from above (in this embodiment, the difference between the optical axes when viewed from above is 180 degrees). Alternatively, a different direction could also mean that the difference in elevation angle between the two optical axes is equal to or greater than 30 degrees or equal to or greater than 60 degrees. Therefore, the stereo camera 14 cannot capture the image projected by the projector 12. The stereo camera 14 captures images of equipment, machines, tools, and workpieces 31, etc., located within the factory. Figure 1 In the example shown, cabinet 53 is shown as an object photographed by stereo camera 14.

[0035] The stereo camera 14 includes two imaging elements, each capturing an image of the work site. The two imaging elements are positioned at an appropriate distance from each other. Each imaging element is, for example, a CCD (charge-coupled device). The two imaging elements simultaneously capture images of the work site by operating synchronously, thus creating a pair of image data. In this embodiment, since it is assumed that the information detected in real time will be projected as auxiliary images, it is preferable that the stereo camera 14 takes multiple images, for example, within one second.

[0036] Furthermore, the stereo camera 14 includes an image processing unit that processes the pair of image data. The image processing unit performs known stereo matching processing on the pair of image data acquired by the stereo camera 14 to determine the positional deviation (parallax) corresponding to each image. The closer the object being projected, the greater the parallax becomes, inversely proportional to the distance. Based on this parallax, the image processing unit creates a distance image that establishes a correspondence between distance information and each pixel of the image data.

[0037] The stereo camera 14 has a structure in which two photographic elements are arranged in a single housing. Alternatively, two separate cameras can be combined to form a stereo camera. Furthermore, the image processing unit can be located in a different device than the stereo camera 14 (e.g., the control device 20).

[0038] The control device 20 is a computer, which includes a CPU, ROM, and RAM. In this embodiment, the control device 20 is disposed on the trolley 11. The control device 20 can communicate with the projector 12, camera 13, and stereoscopic camera 14 via signal lines (not shown). Alternatively, the control device 20 can be disposed outside the trolley 11. In this case, the control device 20 communicates with the projector 12, camera 13, and stereoscopic camera 14, for example, wirelessly.

[0039] The control device 20 acquires the image captured by the camera 13 (first environmental information) (first acquisition step) and the distance image captured by the stereo camera 14 (second environmental information) (second acquisition step). Based on this information and other information, the control device 20 creates an auxiliary image 101 and transmits it to the projector 12. Figure 1 As shown, the control device 20 includes a communication device 21, an analysis unit 22, an image production unit 23, and a projection control unit 24. The various parts of the control device 20 are conceptually divided according to each process performed by the control device 20 (according to each function possessed by the control device 20). In this embodiment, the control device 20 is implemented using a single computer, but the control device 20 can also be composed of multiple computers. In this case, these multiple computers are connected via a network.

[0040] The communication device 21 is a communication module used to communicate with the projector 12, the camera 13, and the stereo camera 14. For example, it includes a connector for connecting signal lines or an antenna for wireless communication. The communication device 21 receives images captured by the camera 13, or receives images captured by the stereo camera 14, or transmits auxiliary images 101 created by the image creation unit 23 to the projector 12.

[0041] When the image captured by camera 13 includes reference marker 51 or interpolation marker 52, the analysis unit 22 performs known analysis processing based on the position, size, distortion, etc., of the reference marker 51 or interpolation marker 52, thereby calculating the relative position (self-position) of camera 13 with respect to the reference marker 51 or interpolation marker 52. Furthermore, self-position refers to the position of the measuring machine itself, and when calculating the position based on the image captured by camera 13, it is the position of the displayed camera 13 (or the work information projection system 1).

[0042] In addition, the analysis unit 22 performs SLAM (Simultaneous Localization and Mapping) processing on the distance images captured by the stereo camera 14. By analyzing the distance images, the analysis unit 22 creates a map (environmental map) showing the shape and position of an object in the work site, and estimates the position (self-position) of the stereo camera 14.

[0043] Since SLAM processing is well-known, it will be briefly explained below. Specifically, the analysis unit 22 sets appropriate feature points by analyzing a distance image and obtains their motion. Then, the analysis unit 22 extracts multiple feature points from the distance image for tracking, and calculates data representing the motion of the feature points in a plane corresponding to the image, expressed as vectors. The analysis unit 22 generates map information based on this data. As described above, the map information is data displaying the shape and position of an object at the work site; more specifically, it is data displaying the three-dimensional positions of the extracted multiple feature points (groups of points). Furthermore, the analysis unit 22 estimates the positional change of the stereo camera 14 based on the position of the input feature points, changes in distance, and the position of the feature points in the map information. Moreover, SLAM processing can also be performed based on images captured by a single-lens reflex camera. Therefore, a single-lens reflex camera can be used instead of the stereo camera 14.

[0044] The image processing unit 23 creates an auxiliary image 101. The control device 20 stores operation information, which is information related to the operation. In this embodiment, the operation information is the name of the part mounted on the workpiece 31 and the installation position of the part. Based on the operation information and the position estimated based on the image taken by the camera 13 or the stereo camera 14, the image processing unit 23 creates an auxiliary image 101 to be projected by the projector 12.

[0045] The projection control unit 24 transmits the auxiliary image 101 produced by the image production unit 23 to the projector 12, so that the auxiliary image 101 is projected. As a result, the auxiliary image 101 can be projected at the work site.

[0046] The following details the estimation of the position of projector 12. First, refer to... Figures 3 to 5 This describes the calibration process performed before the operation.

[0047] First, the calibration guide plate 32 is positioned in front of the projector 12. The calibration guide plate 32 is a component used for calibrating the projector 12 and the camera 13. When an instruction to begin calibration processing is received, the control device 20 transmits the calibration image 102 to the projector 12. Thus, as... Figure 4 As shown, the projector 12 projects the calibration image 102 onto the calibration guide plate 32 (S101). The camera 13 captures the calibration image 102 projected onto the calibration guide plate 32 (S102).

[0048] Next, the control device 20 calculates the relative positional relationship between the projector 12 and the camera 13 by performing known analysis methods based on the position, orientation, size, and degree of distortion of the calibration image 102 included in the image captured by the camera 13 (S103).

[0049] Then, reference mark 51 is set. The position where reference mark 51 is set becomes the origin of the reference coordinate system (marker coordinate system). In this embodiment, the coordinate system with the mark (reference mark 51 or interpolation mark 52) as its origin is called the mark coordinate system, and the coordinate system with reference mark 51 as its origin is specifically called the reference coordinate system. The reference coordinate system refers to the coordinate system used to describe the operation instructions, etc. The position of the origin of the reference coordinate system can be arbitrary; for example, when working on the workpiece 31, it is preferable to set the origin on the workpiece 31. Thus, even assuming that the position of the workpiece 31 changes slightly, the position of the projection auxiliary image 101 is unlikely to shift.

[0050] like Figure 5 As shown, after setting the reference marker 51, the camera 13 captures the area including the reference marker 51, and the stereo camera 14 captures the surrounding work site (S104). Next, the control device 20 calculates the transformation information between the reference coordinate system (marker coordinate system) and the SLAM coordinate system (S105).

[0051] The transformation information is used for coordinate transformation between the reference coordinate system and the SLAM coordinate system. In other words, the transformation information displays the positional relationship between camera 13 and stereo camera 14. The control device 20 calculates the position of camera 13 based on the position, size, distortion, etc., of reference marker 51 included in the image captured by camera 13, using reference marker 51 as a reference (i.e., in the reference coordinate system). The control device 20 performs the SLAM processing based on the distance image captured by stereo camera 14, calculating the position of stereo camera 14 in the SLAM coordinate system. Then, the control device 20 calculates multiple sets of position information, grouping the positions of camera 13 in the reference coordinate system captured at the same time and the positions of stereo camera 14 in the SLAM coordinate system as one set.

[0052] Based on the multiple sets of location information calculated in this way, conversion information is created. Specifically, based on... Figure 3 Equation (1) is shown to calculate the transformation information. The left side of Equation (1) shows the dot product of the vector from the position of camera 13 toward the position of stereo camera 14 and the vector pointing to the stereo camera 14. Since camera 13 and stereo camera 14 do not move relative to each other, the value of the dot product, λ, is a constant. Thus, by substituting multiple sets of position information into Equation (1), it is possible to calculate the coordinates (t) of the marker origin in the SLAM coordinate system and the rotation coordinates (R) from the SLAM coordinate system toward the marker coordinate system. These values ​​are equivalent to the transformation information.

[0053] By using transformation information, the position and orientation of the stereo camera 14 in the SLAM coordinate system estimated during SLAM processing can be converted into reference coordinates. Specifically, the position of the stereo camera 14 in the reference coordinate system is displayed as R(P) S -t), which displays the orientation of the stereo camera 14 in the reference coordinate system as Rd. s Furthermore, by performing the same calculations, it is also possible to calculate the transformation information from the marker origin coordinate system to the SLAM coordinate system.

[0054] Next, refer to Figures 6 to 8 The following explanation describes the processing methods used during the operation. In the following description, the method of estimating position by performing SLAM processing on the distance image captured by stereo camera 14 is referred to as SLAM measurement. The method of estimating position based on the interpolation marks 52 included in the image captured by camera 13 is referred to as mark measurement.

[0055] In this embodiment, SLAM measurement is used as the basis, and marker measurement is performed only when SLAM measurement cannot be performed appropriately. The situation where SLAM measurement cannot be performed appropriately refers to a situation where the stereo camera 14 cannot obtain adequate information; specifically, it is a situation where the stereo camera 14 only photographs a flat wall, or where the number of feature points is small. This will be explained in detail below.

[0056] The control device 20 estimates the position of the SLAM coordinate system based on the distance image captured by the stereo camera 14, and converts it to the position of the reference coordinate system by applying the transformation information calculated in the calibration process (S201, position estimation step). Next, the control device 20 determines whether the image captured by the camera 13 includes the interpolation mark 52 (S202).

[0057] exist Figure 7 In the case described as "1. SLAM measurement", the interpolation mark 52 is not included in the image captured by the camera 13. In this case, the control device 20 generates an auxiliary image 101 based on the position of the reference coordinate system obtained by the stereo camera and projects it from the projector 12 (S206).

[0058] exist Figure 7In the case described as "2. SLAM measurement and calculation of conversion information", the interpolated marker 52 is included in the image captured by the camera 13. In this case, the control device 20 calculates the conversion information (S203) to convert the marker coordinate system of the interpolated marker 52 to the reference coordinate system. The process of calculating this conversion information is the same as the calibration process step S105. That is, since both the position in the marker coordinate system with the interpolated marker 52 as the origin and the position in the SLAM coordinate system are obtained, these are substituted into equation (1) as a set. Thus, the conversion information to convert the marker coordinate system of the interpolated marker 52 to the SLAM coordinate system can be calculated. In addition, the conversion information to convert the SLAM coordinate system to the reference coordinate system has been calculated in step S105. Thus, by combining the two conversion information, the conversion information to convert the marker coordinate system of the interpolated marker 52 to the reference coordinate system can be calculated.

[0059] Next, the control device 20 determines whether the feature points processed by SLAM are equal to or less than a threshold (S204). In other words, the control device 20 determines whether second environmental information can be obtained. Figure 7 In the case described as "2. SLAM measurement, calculation and conversion information", a sufficient number of feature points are obtained based on the detection of cabinet 53 by stereo camera 14. In this case, position measurement based on SLAM measurement is used instead of position measurement using interpolation marker 52. That is, the control device 20 creates an auxiliary image 101 based on the position of the reference coordinate system obtained by stereo camera 14 and projects it from projector 12 (S206).

[0060] If we rephrase the process of step S204, we can say that the process of step S204 has a determination condition for determining whether the information obtained from the stereo camera 14 or the information calculated based on the information is appropriate, and the control device 20 determines whether the determination condition is met.

[0061] exist Figure 8 In the case described as "3. Marker + Camera Measurement", since there are no objects such as cabinet 53 within the range captured by stereo camera 14, the feature points processed by SLAM become equal to or less than the threshold. In this case, position measurement based on SLAM measurement is not used; instead, position measurement using interpolated markers 52 is used. Therefore, the control device 20 converts the position in the marker coordinate system obtained from the image captured by camera 13 (including the image of interpolated markers 52) into the position in the reference coordinate system (S205, position estimation step). This conversion is performed using the conversion information calculated in step S203. Then, the control device 20 creates an auxiliary image 101 based on the position in the reference coordinate system obtained by camera 13 and projects it from projector 12 (S207).

[0062] Furthermore, the situation where the feature points processed by SLAM become equal to or less than the threshold can be predicted in advance. Therefore, in this embodiment, when the feature points processed by SLAM are equal to or less than the threshold, an interpolation marker 52 is placed near the camera 13. Thus, at least one of SLAM measurement and marker measurement can be performed.

[0063] Repeat execution Figure 6 The process is shown. Therefore, in Figure 8 After the situation was recorded as "3. Marking + Camera Measurement", in Figure 8 In the case described as "4. SLAM measurement", that is, when the feature points processed by SLAM exceed the threshold, the position measurement using interpolation marker 52 is stopped, and position measurement based on SLAM measurement is adopted instead.

[0064] As mentioned above, by using SLAM measurement as a basis and performing marker measurement when the position estimation accuracy of SLAM measurement decreases, the position estimation accuracy at the work site can be maintained at a relatively high level.

[0065] As explained above, the work information projection system 1 of this embodiment is used for a position estimation method to estimate the position in a work site. The work information projection system 1 includes a camera 13, a stereo camera 14, and a control device 20. The camera 13 is positioned facing a first direction to acquire first environmental information, which is information about objects positioned in and around the first direction. The stereo camera 14 is positioned facing a second direction to acquire second environmental information, which is information about objects positioned in and around the second direction. The control device 20 can calculate its own position in a first coordinate system based on the first environmental information, and can also calculate its own position in a second coordinate system based on the second environmental information. If the second environmental information cannot be acquired, the control device 20 estimates its position in the first coordinate system based on the first environmental information.

[0066] Therefore, even when the stereo camera 14 cannot obtain environmental information, it is still possible to estimate the location of the work site.

[0067] In the position estimation system of this embodiment, the stereo camera 14 is configured to move integrally with the camera 13.

[0068] Therefore, since the positional relationship between camera 13 and stereo camera 14 remains unchanged, the location of the work site can be estimated more appropriately.

[0069] In the position estimation system of this embodiment, the first direction and the second direction are different directions.

[0070] Therefore, since the detection ranges of camera 13 and stereo camera 14 are very different, one sensor can complement the other.

[0071] In the work information projection system 1 of this embodiment, when it is determined that the first environmental information and the second environmental information can be obtained, the control device 20 calculates the transformation information for converting between the first coordinate system and the second coordinate system.

[0072] Therefore, even when switching to a different sensor for position measurement, the same coordinate system can still be used.

[0073] The work information projection system 1 of this embodiment includes a projector 12, which projects an auxiliary image 101 to the work site. The control device 20 generates an auxiliary image 101 corresponding to the location in the work site and transmits the auxiliary image 101 to the projector 12.

[0074] Therefore, it can assist the operator in their work. In particular, since the work information projection system 1 of this embodiment is not prone to losing its own position, it can achieve high reliability.

[0075] In the work information projection system 1 of this embodiment, the optical axis of the camera 13 overlaps with the area of ​​the projector 12 capable of projecting the auxiliary image 101. The camera 13 is a camera that captures a range including the markings set at the work site. The stereo camera 14 is a stereo camera used to capture objects arranged in and around the second direction.

[0076] Therefore, since the stereo camera 14 does not capture the auxiliary image 101 projected by the projector 12, it is possible to prevent the auxiliary image 101 from being identified as a feature point.

[0077] In the work information projection system 1 of this embodiment, when the projector 12 projects the auxiliary image 101 onto the work site, and the control device 20 determines that the first environmental information and the second environmental information can be obtained, it calculates the conversion information for converting between the first coordinate system and the second coordinate system.

[0078] Therefore, it is possible to calculate the conversion information required in the operation, thus reducing the trouble of preparation in advance.

[0079] In the position estimation system of this embodiment, conversion information is calculated according to a formula that calculates the dot product of the vector from the position of camera 13 toward the position of stereo camera 14 and the vector of the direction in which stereo camera 14 is displayed.

[0080] Therefore, information can be transformed through simple processing and calculation.

[0081] The preferred embodiments of the present invention have been described above, but the structure can be modified, for example, as follows.

[0082] In the described embodiment, the first sensor, oriented in substantially the same direction as the projector 12, is the camera 13, and the second sensor, on the opposite side, is the stereo camera 14. Alternatively, the first sensor may be the stereo camera 14, and the second sensor may be the camera 13. In this case, the interpolation mark 52 is placed at an appropriate location on the work site rather than on the workpiece 31.

[0083] In the described embodiment, the first sensor and the second sensor (i.e., camera 13 and stereo camera 14) face different directions. Alternatively, as long as the detection ranges of the first sensor and the second sensor are different (e.g., different detection distances, different horizontal widths of the detection ranges, different horizontal widths of the detection ranges, etc.), the first sensor and the second sensor can also face the same direction. In other words, the first direction and the second direction can also be the same. In this case, since the second sensor can detect areas that cannot be detected by the first sensor, the same processing as in the described embodiment can be applied.

[0084] In the described embodiment, a stereo camera 14 was used as an example of a three-dimensional measuring device, but other devices, such as LiDAR (laser detection and ranging device), can also be used. LiDAR is a technique that obtains the position and shape of surrounding objects by irradiating radio waves in various directions and measuring the time until the reflected wave of the radio waves is received.

[0085] The process described in the embodiment is an example, and it is also possible to omit some processes, change the content of some processes, or add new processes. For example, in the embodiment described, such as Figure 6 As shown, SLAM measurement is prioritized over marker measurement in both SLAM measurement and marker measurement, but marker measurement can also be prioritized instead. That is, marker measurement is usually performed when the image captured by camera 13 includes interpolated markers 52, and SLAM measurement is performed only when the image captured by camera 13 does not include interpolated markers 52.

[0086] In the described embodiment, the estimated position is converted into a reference coordinate system, but this process can be omitted.

[0087] The position estimation system of this invention is not limited to job information projection systems, but can be applied to various systems that estimate their own position. For example, this invention can be applied to driving systems that perform autonomous movement.

[0088] Explanation of reference numerals in the attached figures

[0089] 1. Job Information Projection System (Location Recognition System)

[0090] 11 strollers

[0091] 12 projectors

[0092] 13 cameras (first sensor)

[0093] 14. Stereo camera (second sensor, three-dimensional measuring device)

[0094] 20 control devices

Claims

1. A location estimation system for estimating location in a work site, characterized in that... include: A first sensor, configured to face a first direction, acquires first environmental information, which is information about objects configured in the first direction and in the surrounding environment. The second sensor, which is configured to face the second direction, acquires second environmental information, which is information about objects configured in the second direction and in the surrounding environment; as well as The control device is capable of calculating its own position in a first coordinate system by performing marker measurements based on the first environmental information, and capable of calculating its own position in a second coordinate system by performing SLAM measurements based on the second environmental information. If the second environmental information is unavailable, the control device estimates its own position based on the first environmental information. The SLAM measurement is a method for estimating location by performing SLAM processing based on the second environmental information. The marker measurement is a method of estimating position based on markers included in the first environmental information that serve as the origin of the first coordinate system. If the first environmental information and the second environmental information are available, the control device calculates transformation information for converting between the first coordinate system and the second coordinate system. When generating the transformation information, the marker measurement is not performed; instead, the SLAM measurement is performed. If the second environmental information cannot be obtained, the SLAM measurement is not performed; instead, the position is estimated based on the marker measurement and the transformation information.

2. The location estimation system according to claim 1, wherein, The second sensor is configured to move integrally with the first sensor.

3. The location estimation system according to claim 1 or 2, wherein, The first direction is a different direction from the second direction.

4. The location estimation system according to claim 1 or 2, wherein, It includes a projector that projects auxiliary images used for assisting the work onto the work site. The control device generates an auxiliary image corresponding to the location in the work site and transmits the auxiliary image to the projector.

5. The location estimation system according to claim 4, wherein, The optical axis of the first sensor overlaps with the area of ​​the projector capable of projecting the auxiliary image. The first sensor is a camera that captures images of the area including the markings located at the work site. The second sensor is a three-dimensional measuring device used to obtain the shape and position of objects disposed in and around the second direction.

6. The location estimation system according to claim 5, wherein, During the operation where the projector projects the auxiliary image onto the work site, the control device, having acquired the first environmental information and the second environmental information, calculates the transformation information for converting between the first coordinate system and the second coordinate system.

7. The location estimation system according to claim 1, wherein, The conversion information is calculated using a formula that calculates the dot product of the vector from the position of the first sensor toward the position of the second sensor and the vector showing the direction of the second sensor.

8. A location estimation method, characterized in that... The process includes the following steps: The first acquisition step involves acquiring information about objects positioned in and around the work site, i.e., first environmental information, using a first sensor positioned in a first direction at the work site. The second acquisition step involves acquiring information about objects positioned in and around the work site, i.e., second environmental information, using a second sensor positioned in a second direction at the work site. as well as The position estimation process includes calculating the user's position in a first coordinate system by performing marker measurements based on the first environmental information, and calculating the user's position in a second coordinate system by performing SLAM measurements based on the second environmental information. If the second environmental information is unavailable, the user's position is estimated based on the first environmental information. The SLAM measurement is a method for estimating location by performing SLAM processing based on the second environmental information. The marker measurement is a method of estimating position based on markers included in the first environmental information that serve as the origin of the first coordinate system. If the first environmental information and the second environmental information are available, calculate the transformation information for converting between the first coordinate system and the second coordinate system. When generating the transformation information, the marker measurement is not performed; instead, the SLAM measurement is performed. If the second environmental information cannot be obtained, the SLAM measurement is not performed; instead, the position is estimated based on the marker measurement and the transformation information.

Images