Object detection device and survey system

Abstract

The present invention automatically detects a specific object without preparing many training data items but only by registering, in advance, one reference image and any pixel of a target object that should be detected. An object detection device (90A) comprises: a control calculation unit (91) provided with at least one processor and at least one memory; and a storage unit (92) provided with a computer-readable storage medium. The storage unit (92) stores a reference feature amount, which is a feature amount of a detection target object in a reference image (81), as reference data (80). The control calculation unit (61) comprises a detection target object detection unit (912) which: generates a feature image from a measurement image (70) by changing each pixel into a feature amount; compares the feature amount of each pixel with the reference feature amount to generate a similarity map; identifies, as a feature point, a pixel that is most similar in the similarity map; segments the measurement image by using a segmentation model trained through machine learning with a lot of images and mask images; automatically designates the pixel that is the feature point in a segmentation image; and detects a segment including the designated pixel as the detection target object.

Description

Object detection device and surveying system

[0001] The present disclosure relates to an object detection device and a surveying system, and more particularly to an object detection device for detecting an object from an image and a surveying system using the same.

[0002] Conventionally, object detection techniques have been known as techniques for detecting the presence or absence of an object and its position in an image. Known object detection algorithms include R-CNN, Faster R-CNN, and SSD. However, conventional object detection techniques have had the problem that even for a specific object, it is necessary to prepare a large number of data sets that pair an image containing the object with a mask image of the object, making the annotation process cumbersome.

[0003] In recent years, a large number of image data and basic models for segmentation have been proposed. One such model is Meta's Segment Anything Model (see Non-Patent Document 1). The Segment Anything Model makes it possible to perform segmentation tasks in a so-called zero-shot fashion without additional learning for individual objects.

[0004] JP 2016-138802 A JP 2005-214854 A

[0005] Kirillov A, Mintun E, Ravi N, Mao HZ, Rolland C, Gustafson L, Xiao TT, Whitehead S, Berg AC, Lo WY, Dollar P, Girshick R. “Segment anything”, [onlne], arXiv preprint arXiv:2304.02643, 2023, [Retrieved March 22, 2020], Internet <URL: https: / / arxiv.org / abs / 2304.02643>

[0006] However, this model is for segmentation, i.e., for classifying each pixel by associating it with the object to which it belongs. Therefore, in order to detect a specific object, it is necessary to manually select the pixels of the object to be detected for each output image and extract the detection target portion, which is a time-consuming process. Furthermore, in recent years, there has been a demand in the field of surveying for a technology that uses object detection technology to improve the efficiency of surveying work.

[0007] The present invention has been made in consideration of the above circumstances, and aims to provide an object detection technology that can be applied to surveying systems, which can automatically detect specific objects by simply registering a single reference image and any one pixel of the object to be detected in advance, without having to prepare a large amount of training data.

[0008] In order to achieve the above object, an object detection device according to one aspect of the present invention has the following configuration.

[0009] 1. An object detection device includes a control and calculation unit including at least one processor and at least one memory, and a storage unit including a computer-readable storage medium, the storage unit storing, as reference data, reference feature amounts that are feature amounts of a detection object in a reference image including the detection object, the control and calculation unit generating a feature image by converting each pixel from a measurement image into a feature amount, comparing the feature amount of each pixel with the reference feature amount to generate a similarity map, identifying the most similar pixel in the similarity map as a feature point, segmenting the measurement image using a segmentation model based on machine learning using a large number of images and a mask image, automatically designating pixels of the feature points in the segmentation image, and detecting a segment including the designated pixel as the detection object.

[0010] 2. In the above aspect, it is also preferable that the reference feature amount is obtained by segmenting a reference image including the detection object using the segmentation model, and averaging feature amounts of segments selected from the segmentation image by specifying pixels included in the detection object.

[0011] A surveying system according to another aspect of the present invention has the following configuration.

[0012] 3. a distance measuring device housed in a telescope, which transmits distance measuring light to a prism and receives reflected light from the prism to measure the distance to the center of the prism; an angle detector which detects the angle of the collimation optical axis of the telescope; a rotation drive unit which drives and rotates the telescope; and a camera which has an angle of view wider than the field of view of the telescope and captures an image in front of the telescope as the measurement image; wherein a general target for roughly turning the prism is set, the general target is the object to be detected, and the reference image is an image of the general target, and the control and calculation unit detects the general target from the measurement image, calculates a general rotation angle for roughly turning the collimation optical axis of the telescope toward the general target based on the position of the general target, sets a target general target to be turned around from the general target in the measurement image, and drives the rotation drive unit to rotate the telescope by the general rotation angle.

[0013] 4. In the above aspect 3, it is also preferable that the outline targets include a first outline target and a second outline target having dimensions larger than the first outline target, and the target outline target is set according to the approximate distance to the first outline target and the approximate distance to the second outline target.

[0014] 5. In the above aspects 3 and 4, it is also preferable that the first outline target is the prism, and the second outline target is clothing worn by a worker holding the prism.

[0015] 6. In the above aspects 3 to 5, it is also preferable that the outline target is a prism.

[0016] 7. In the above aspects 3 to 6, it is also preferable that the reference image is an image of the detection object viewed from a plurality of directions.

[0017] 8. In the above aspects 3 to 7, it is also preferable that the reference images are images of the general target taken at a plurality of distances from the camera.

[0018] 9. In addition, in the above aspects 3 to 8, it is also preferable that the detection target detection unit includes a color determination unit that determines the color of the outline target in the image, the memory unit stores a predetermined color as the color of the outline target, and the control and calculation unit detects the outline target of the predetermined color as the outline target.

[0019] 10. In the above aspects 3 to 9, it is also preferable that the outline target is clothing worn by a worker, the detection object detection unit includes a pose estimation unit that estimates a pose of the worker, and the control calculation unit detects the outline target as the outline target when the outline target exhibits a predetermined pose.

[0020] According to the above configuration, it is possible to provide an object detection technology that can be applied to surveying systems, which can automatically detect specific objects by simply registering a single reference image and any one pixel of the object to be detected in advance, without having to prepare a large amount of training data.

[0021] 1A and 1B are block diagrams of the object detection device according to the first embodiment. FIG. 1B is a flowchart illustrating an example of a reference data registration process in the object detection device. FIG. 1C is a flowchart illustrating an example of a mask image generation process in the object detection device. FIG. 1D is a diagram illustrating a reference data registration process in the object detection device. FIG. 1D is a flowchart illustrating an example of a detection target detection process in the object detection device. FIG. 1E is a diagram illustrating a detection target detection process in the object detection device. FIG. 1F is a schematic external view showing a usage state of a surveying system according to a second embodiment. FIG. 1F is a block diagram of the surveying system. FIG. 1C is a flowchart illustrating an example of a general process of a look-around function by the surveying system. FIG. 1D is an image diagram illustrating an overview of the look-around function by the surveying system. FIG. 1F is a flowchart illustrating a detailed process of calculating an approximate rotation angle in the look-around function process. FIG. 1F is a diagram illustrating a method of calculating the center position of an approximate target for calculating the approximate rotation angle. FIG. 1F is a diagram illustrating a method of calculating a horizontal approximate rotation angle for calculating the approximate rotation angle. FIG. 1F is a flowchart illustrating a detailed process of setting an approximate target to look back at in the look-around function process. FIG. 1C is a diagram illustrating an example of a reference image for creating reference data to be registered in the surveying system. FIG. 1D is another example of a reference image for creating reference data to be registered in the surveying system. FIG. 1D is a diagram illustrating a modified example of a rough target detection unit constituting the surveying system. 10A and 10B are diagrams illustrating another modified example of the general target detection unit constituting the surveying system.

[0022] Preferred embodiments of the present invention will be described below with reference to the drawings. In the following description of the embodiments, the same components are given the same reference numerals, and the same functions are given the same names, and redundant description will be omitted as appropriate.

[0023] 1. First Embodiment (Hardware Configuration of Object Detection Device) Fig. 1(A) is a block diagram illustrating the hardware configuration of an object detection device 90 according to a first embodiment of the present invention. Fig. 1(B) is the same block diagram of an object detection device 90A according to another example of the first embodiment. The object detection device 90A will be described later.

[0024] 1A, the object detection device 90 includes a control and calculation unit 91, a storage unit 92, an input unit 94, and a display unit 93. The object detection device 90 is a device that functions as a so-called computer, and may be built into a personal computer or a server computer. It may also be built into a surveying device, a surveying system, or other device. The server computer referred to here may be a cloud server.

[0025] The control and calculation unit 91 includes at least one electronic circuit and at least one memory. The electronic circuit may include a CPU (Central Processing Unit) which is a processor, and may further include a GPU (Graphics Processing Unit). The control and calculation unit 91 may be configured with one processor or multiple processors. When configured with multiple processors, the processors only need to be connected to each other so that they can communicate with each other.

[0026] The control and calculation unit 91 also includes, as functional units, a reference data registration unit 911 and a detection target detection unit 912 .

[0027] The reference data registration unit 911 accepts a reference image 81 including an image of the detection target T, calculates features (reference features) 86 in the segment of the detection target T in the reference image 81 as reference data 80 based on the reference image 81, and stores them in the memory unit 92.

[0028] The detection target detection unit 912 receives the measurement image 70 and detects the detection target T from within the measurement image 70 .

[0029] In order to receive the reference image 81 and the measurement image 70, the control and calculation unit 91 may be connected to a camera or other external device, or may be configured to be connectable via communication. Furthermore, data input and output may be possible through an input / output interface such as a USB interface. Furthermore, the reference image 81 may be stored in the storage unit 92 in advance.

[0030] The storage unit 92 is a computer-readable recording medium that stores and transmits information in a format that can be processed by a computer. The storage unit 92 may be, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or an electronic medium such as a hard disk drive (HDD), a solid state drive (SSD), or an optical disk.

[0031] The storage unit 92 stores a program for causing the control and calculation unit 91 to execute the processing of each functional unit (reference data registration unit 911, detection target object detection unit 912) of the object detection device 90. The storage unit 92 also stores reference data 80 generated based on the reference image 81.

[0032] The input unit 94 is, for example, a keyboard, a mouse, etc. The input unit 94 allows a user to input various instructions, commands, etc. to the control and calculation unit 91.

[0033] The display unit 93 is a liquid crystal display, an organic EL display, or the like.

[0034] (Processing of Object Detection Device) (Registration of Reference Data) Prior to detecting a detection target T using the object detection device 90, reference data 80 to be referenced when detecting the detection target T must be calculated using a reference image 81 and registered. The process of registering the reference data 80 will be described with reference to FIGS. 2 to 4. FIG. 2 shows the processing flow of the control and calculation unit 91 in registering the reference data 80. FIG. 3 shows the processing flow of an example of the process of generating a mask image, which is one step in registering the reference data 80. FIG. 4 is a diagram illustrating the processing in each step using a specific image as an example.

[0035] When the process starts, in step S01, the reference data registration unit 911 receives the reference image 81. The reference image 81 is an RGB image including a detection object T. In the example of Fig. 4, the detection object T is a pipe.

[0036] Next, in step S02, the reference data registration unit 911 generates a mask image 82 from the reference image 81. The mask image 82 may be generated manually, or may be generated as shown in FIG.

[0037] Specifically, when the process starts, in step S11, the reference image 81 is input to the segmentation model M1 and segmented. The segmentation model performs machine learning using a large number of images and a large number of accompanying mask images as a set of training data, determining which object each pixel belongs to, and performing segmentation. For example, the Segment Anything Model described in Non-Patent Document 1 can be used. However, without being limited to this, any available segmentation model constructed using a known algorithm, such as a Region-Convolutional Neural Network (R-CNN) or a Recurrent Neural Network (RNN), may also be used. This results in a segmented image in which each pixel of the image is segmented for each object to which it belongs. However, at this stage, it is not known which segment corresponds to the detection target object T.

[0038] Next, in step S12, the pixel of the detection object T is manually designated in the segmented image, thereby selecting the detection object T.

[0039] Then, in step S13, the image is binarized so that the segments of the detection object T are white and the rest are black (masked area), thereby generating a mask image 82.

[0040] After generating the mask image, the process returns to FIG. 2 , and in step S03, the reference image 81 is input to the feature image generation model M2 to generate a reference feature image 84 by converting each pixel into a feature. The feature image generation model M2 is a trained model using machine learning that converts each pixel of the input image into a feature to generate a feature image shown in the image. For example, an image encoder that constitutes part of the above-mentioned Segment Anything Model is provided as a model that performs dimensionality reduction on input data and extracts features, and this can be used. However, without being limited to this, any available feature image generation model constructed using a Region-Convolutional Neural Network (R-CNN), a recurrent neural network (RNN), or other known algorithms may also be used.

[0041] Next, in step S04, the reference feature image 84 generated in step S03 is masked with the mask image 82, and in step S05, a reference feature image (detection object reference feature image) 85 of only the portion (extracted area) of the detection object T is generated.

[0042] Next, in step S06, the feature amounts of the extracted region are averaged to calculate a reference feature amount 86. Next, in step S07, the reference feature amount 86 is stored (registered) in the storage unit 92 as reference data 80, and the process ends. Note that it is sufficient to register at least one reference data item 80.

[0043] (Processing for detecting an object to be detected) Next, the processing for detecting an object to be detected will be described with reference to Figs. 5 and 6. Fig. 5 is a flowchart of the processing for detecting an object to be detected in the control and calculation unit 91. Fig. 6 is a diagram for explaining the processing at each step using a specific image as an example. In Fig. 6, the object to be detected T is also a pipe.

[0044] When the process starts, in step S21, the detection target detection unit 912 receives the measurement image 70. The measurement image 70 may be, for example, one acquired by a camera mounted on a surveying system incorporating the object detection device 90. Alternatively, the measurement image 70 may be one received by the object detection device 90 via a communication unit (not shown).

[0045] Next, in step S22, the detection target detection unit 912 inputs the measurement image 70 into the same feature image generation model M2 as in step S03 to generate a feature image 71. Next, in step S23, the feature amount of each pixel in the measurement image 70 is compared with the reference feature amount 86 read out from the storage unit 92 to generate a similarity map 72. In the similarity map 72, for example, similar parts are displayed in reddish colors and dissimilar parts are displayed in bluish colors, like a heat map. In similar parts, the higher the similarity, the darker the color, and in dissimilar parts, the higher the dissimilarity, the darker the color. In FIG. 6, the part indicated by reference numeral 73 is the most similar part.

[0046] Next, in step S24, the detection target object detection unit 912 identifies the most similar pixel on the similarity map 72 as a key point K.

[0047] Next, in step S25, the measurement image 70 is input into the same segmentation model M1 as in step S11 to obtain a segmentation image (not shown). Then, in step S26, a detection result image 74 is generated and the process ends. The detection result image 74 may be displayed on a display unit or may be output as is for subsequent processing. Note that the segmentation model M1 is the same as that used when generating the mask in step S02. Next, in step S26, the detection object detection unit 912 specifies a feature point K on the segmented image. As a result, in step S27, the detection object detection unit 912 detects a detection object T and the process ends. As shown in FIG. 6, it can be seen that the portion surrounded by a white frame in the detection result image 74 has been detected as a "pipe."

[0048] (Effects) As described above, the object detection device 90 according to this embodiment generates the feature image 85 of only the region of the detection target T in the reference image 81, and calculates the feature amount of the region of the detection target T as the reference feature amount. Then, during object detection, the detection target detection unit 912 is configured to compare the feature amount of each pixel of the feature image 71 generated from the measurement image 70 with the reference feature amount 86, and determine the pixel with the highest similarity as the feature point K. Then, by inputting the measurement image 70 into the segmentation model M1 and selecting the feature point K in the segment image (not shown) that is output, the object detection device 90 can automatically detect the detection target T.

[0049] Note that, when the object detection device 90 detects a detection target, if the reference data 80 is registered, the reference data registration unit 911 is not essential. For this reason, the object detection device 90 may have the configuration of the object detection device 90A shown in FIG. 1B in which the reference data 80 is registered in advance in the storage unit 92 and the control calculation unit 91 does not include the reference data registration unit 911.

[0050] The above-described configuration achieves the unique effect of the present invention, which reduces the workload during object detection and simplifies object detection, thereby solving the above-described problem. Specifically, it is possible to automatically detect a specific object by simply registering a single reference image and any pixel of the target object to be detected in advance, without preparing a large amount of training data. The object detection device 90 is suitable for automatic processing without human intervention. In particular, in this embodiment, at least one reference data item needs to be registered, thereby reducing the effort and cost required to implement such an object detection function in a surveying system or other system. In other words, the problem to be solved by the present invention is to adapt object detection technology to a surveying system by automating manual processes when using a model that can reduce the effort required for preparing training data in object detection using machine learning. The above-described configuration solves this problem.

[0051] 2. Second Embodiment (Outline of Surveying System 100) Fig. 7 is a schematic diagram of a surveying system 100 according to a second embodiment of the present invention in use. The surveying system 100 is configured as a surveying instrument (total station) equipped with the object detection device 90 or 90A described in the first embodiment and a camera 14. The following description will be given assuming that the surveying system is equipped with the object detection device 90A.

[0052] The surveying system 100 comprises, from below, a leveling unit 4, a base unit 6a provided on the leveling unit 4, a support unit 6b that rotates horizontally on the base unit 6a, and a telescope 6c that rotates vertically at the center of the support unit 6b. A camera 14 is attached to the top of the housing of the telescope 6c so that it rotates integrally with the telescope 6c. The optical axis B of the camera 14 coincides with the collimation optical axis A of the telescope 6c in the horizontal direction and extends parallel to the collimation optical axis A in the vertical plane. The surveying system 100 is installed at a known point P using a tripod 2.

[0053] The worker OP carries the pole 9 with the prism 8 attached to the tip to the measurement point Q. At the measurement point Q, the worker OP holds the pole 9 vertically facing the surveying system 100, and causes the surveying system 100 to automatically collimate the prism 8, which then measures the distance to the center O of the prism 8. At this time, it is preferable to hold the prism 8 so that the relative positional relationship between the prism 8 and the jacket 7 of the worker OP is roughly constant. As shown in the figure, the prism 8 is held near the center of the jacket 7 of the worker OP.

[0054] In order to capture the prism 8 within or near the narrow field of view of the telescope 6c, a general target RT for general turning is preset in the surveying system 100. The surveying system 1 acquires a wide-angle image (hereinafter simply referred to as image) 70 of the area in front of the telescope 6c using the camera 14 and detects the general target RT. Then, the surveying system 1 calculates a general rotation angle θt for directing the telescope 6c toward the general target RT, and rotates the telescope 6c by the general rotation angle θt to capture the prism 8 within the field of view. Then, precise automatic collimation is performed.

[0055] (Outline Target) The outline target RT is a target for outline turning rotation (hereinafter referred to as outline rotation) to roughly point the telescope 6c toward the prism 8, which is the final target, in order to precisely and automatically aim at the prism 8. In the surveying system 1, the prism 8 is set as the first outline target RT1, and the jacket (clothing) 7 of the operator OP is set as the second outline target RT2.

[0056] Generally, the prism 8 has a length and width of approximately 10 cm by approximately 10 cm (depending on the type), while the jacket of the worker OP has a length and width of approximately 50 cm by approximately 60 cm. In other words, the second outline target RT2 is larger in size than the first outline target RT1. The second outline target RT2 is not limited to the jacket 7 of the worker OP, but may also be a suit including a jacket and pants, or a safety reflective vest. Furthermore, it is preferable that the jacket 7 be a dedicated jacket with a unified size, design, and color scheme.

[0057] (Detailed configuration of surveying system 100) Figure 8 is a block diagram of the surveying system 100. The surveying system 100 comprises a distance measuring device 11, an angle detector 12, a rotation drive unit 13, a camera 14, an automatic collimation unit 15, a memory unit 16, an input unit 17, a display unit 18, and a control and calculation unit 130. The angle detector 12 comprises a horizontal angle detector 12a and a vertical angle detector 12b. The rotation drive unit 13 comprises a horizontal rotation drive unit 13a and a vertical rotation drive unit 13b.

[0058] The horizontal rotation drive unit 13a and horizontal angle detector 12a are housed in the base unit 6a. The vertical rotation drive unit 13b, vertical angle detector 12b, memory unit 16, and control and calculation unit 130 are housed in the base unit 6b. The input unit 17 and display unit 18 are provided outside the base unit 6b. The distance measuring device 11 and automatic collimation unit 15 are housed in the telescope 6c, and the camera 14 is attached to the top of the telescope 6c as described above.

[0059] The distance measuring device 11 is an optical distance meter including a light-transmitting unit having a light-emitting element such as a laser diode, a distance measuring optical system, a reference optical path, and a light-receiving unit having a light-receiving element such as an avalanche photodiode. The distance measuring device 11 emits distance measuring light such as infrared laser light via the distance measuring optical system to the prism 8 along the collimation optical axis A of the telescope 6c, and receives the reflected light at the light-receiving unit. The distance measuring device 11 then outputs light-receiving signals of the distance measuring light and the internal reference light that enters the light-receiving element via the reference optical path to the control and calculation unit 130. The control and calculation unit 130 calculates the distance to the center of the prism 8 from the phase difference or time difference between the distance measuring light and the internal reference light.

[0060] The horizontal angle detector 12a and the vertical angle detector 12b are rotary encoders that detect the rotation angles of the base unit 6b and the telescope 6c around their rotation axes, which are driven by the horizontal rotation drive unit 13a and the vertical rotation drive unit 13b, respectively. The control and calculation unit 130 calculates the horizontal angle and the vertical angle of the collimation optical axis A of the telescope 6c based on the detection signals.

[0061] The horizontal rotation drive unit 13a and the vertical rotation drive unit 13b are motors. The horizontal rotation drive unit 13a rotates the base unit 6b horizontally around the axis H-H under the control of the control and calculation unit 130. Similarly, the vertical rotation drive unit 13b rotates the telescope 6c vertically around the axis V-V under the control of the control and calculation unit 130.

[0062] The camera 14 is an RGB camera equipped with an objective lens and an image sensor such as a CCD (Charge-Coupled Device) or CMOS (Complementary Metal-Oxide-Semiconductor) sensor, and captures color images of the area in front of the telescope 6c. The image sensor has a Cartesian coordinate system with the camera center as its origin, and the local coordinates of each pixel are identified. The positional relationship between the camera center and the instrument center of the surveying instrument 10 is known, and the control and calculation unit 30 can convert and manage images captured by the camera 14 into the coordinates of the surveying instrument 10. The camera 14 has a wider field of view than the telescope 6c. For example, the telescope has a field of view of 1.5° × 1.5° (horizontal × vertical), while the camera 14 has a field of view of approximately 10° × approximately 15° (horizontal × vertical). The image sensor has a Cartesian coordinate system with the camera center as its origin, and the position of each pixel on the imaging surface can be identified. The positional relationship between the center of the camera, the instrument center of the surveying instrument 10, and the collimation optical axis A is known. The optical axis of the camera 14 coincides with the collimation optical axis A of the telescope 6c in the horizontal direction, and the horizontal angle of view with respect to the collimation optical axis A of the telescope 6c (the angle with respect to the optical axis obtained from the pixel position) can be obtained by determining the position of the pixel on the image sensor.

[0063] The automatic collimation unit 15 includes a collimation light transmitting unit having a light emitting element such as a laser diode, a collimation optical system having a lens and a dichroic prism, and a collimation light receiving unit having an image sensor such as a CCD or CMOS sensor (not shown). The automatic collimation unit 15 emits collimation light of a different wavelength from the distance measurement light along a common optical axis, receives the reflected light with the image sensor, and captures landscape images in the collimation direction when the collimation light is on and when the collimation light is off. Both images are output to the control and calculation unit 130. The image sensor is configured to identify the positions of pixels on the light receiving surface (image capture surface). Furthermore, the center of the image sensor is aligned with the optical axis of the distance measurement light. By determining the positions of the pixels on the image sensor, the angle of view relative to the optical axis of the distance measurement light (the angle relative to the optical axis obtained from the pixel position) can be calculated. The control and calculation unit 130 determines the center of the image of the prism 8 from the difference between the two images, and drives the rotation drive unit 13 based on the result of the position detection of the prism 8 to collimate the center O of the prism 8.

[0064] The storage unit 16 is a computer-readable storage medium such as a HDD, SSD, flash memory, etc. The storage unit 16 stores programs and related data for executing various functions of the surveying system 100, including the functions of each functional unit. The storage unit 16 stores reference data 80 for the approximate target RT, which is the object to be detected.

[0065] Specifically, the reference data 80 is a reference feature calculated by the method described in the first embodiment. At least one reference data 80 may be registered for each of the first outline target and the second outline target.

[0066] The input unit 17 is an operation button. The operator OP inputs commands for causing the surveying instrument 10 to perform operations and selects settings via the input unit 17.

[0067] The display unit 18 is, for example, a liquid crystal display or an organic EL display. It displays various information such as measurement results and calculation results under the control of the control and calculation unit 130. The input unit 17 and the display unit 18 may be integrated into one unit to form a touch panel display.

[0068] The control and calculation unit 130 includes at least one electronic circuit and at least one memory. The electronic circuit may be, for example, a processor such as a CPU. The memory may be, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). When a CPU is used, the control and calculation unit 30 executes various functions of the surveying instrument 10 by reading programs for the CPU to execute functions into the memory and executing the programs.

[0069] Note that a part of the control and calculation unit 130 may be configured in hardware using a CPLD (Complex Programmable Logic Device), an FPGA (Field Programmable Gate Array), or the like.

[0070] The control and calculation unit 130 is connected to the distance measuring device 11, angle detector 12, rotation drive unit 13, camera 14, automatic collimation unit 15, memory unit 16, input unit 17, and display unit 18, and performs the control necessary to execute the functions of the surveying system 100 and inputs and outputs information.

[0071] The control and calculation unit 130 includes, as functional units, a wide-angle image acquisition unit 131, a rough target detection unit 132, a rough rotation angle calculation unit 133, a rough target identification unit 134, a rough rotation execution unit 135, and a precision collimation execution unit 136.

[0072] The wide-angle image acquisition unit 131 controls the camera 14 to acquire an image 70 in front of the telescope 6c. The outline target detection unit 132 detects an outline target from the image 70. The outline target detection unit 132 corresponds to the detection object detection unit 912 in the first embodiment. The outline rotation angle calculation unit 133 calculates the angle between the center of the detected outline target and the direction of the collimation optical axis A as the outline rotation angle θt. The outline target identification unit 134 identifies an outline target RT to turn around from the detected outline target RT. The outline rotation execution unit 135 drives the rotation drive unit 13 to rotate the telescope 6c by the rotation angle calculated by the outline rotation angle calculation unit (described later). The precision collimation execution unit 136 drives the automatic collimation unit 15 to precisely collimate the prism 8 that has entered the field of view of the telescope 6c due to the outline rotation. Detailed operations of each unit will be described later.

[0073] In this way, the approximate target detection unit 132, memory unit 16, input unit 17, and display unit 18 of the control and calculation unit 130 of this embodiment function as the detection target detection unit 912, memory unit 92, display unit 93, and input unit 94 of the control and calculation unit 91 of the object detection device 90A of the first embodiment, respectively.

[0074] (Overall Processing of Turn-Around Function) Next, the overall processing of the turn-around function will be described with reference to Figures 9 and 10. When the turn-around function is started, in step S31, the wide-angle image acquisition unit 131 controls the camera 14 to acquire a measurement image (hereinafter simply referred to as image) 70 of the area in front of the telescope 6c. Figure 10(A) is an example of the image 70. The image 70 in Figure 10(A) includes the prism 8 (installed on a stand) as well as the operator OP holding the prism 8.

[0075] Next, in step S32, the outline target detection unit 132 detects whether an outline target RT is present in the image 70, and if so, detects its position and range. Once detected, a bounding box is set as shown by the dashed line in Figure 10(B).

[0076] Next, in step S33, the outline target detection unit 132 determines whether the detection of the outline target RT was successful. This is because the outline target RT may not be detected in the image 70 depending on the distance to the prism 8, the resolution of the camera 14, and other conditions. In Figure 10(B), no bounding box is set for the prism 8 in the upper right corner that was not detected.

[0077] If step S33 is successful (Yes), in step S34, the approximate rotation angle calculation unit 133 calculates, for all detected approximate targets RT, the approximate vertical and horizontal rotation angles θt (approximate vertical rotation angle θtv, approximate horizontal rotation angle θth) required to orient the telescope 6c from the current orientation of the collimation optical axis A toward the center of the approximate target RT. Details of the angle calculation will be described later.

[0078] Next, in step S35, the outline target identification unit 134 identifies a target outline target (hereinafter referred to as a target target) RTt (the target to be turned around in FIG. 10C is RTt) to be turned around from the detected outline targets RT. Details will be described later.

[0079] Next, in step S36, the rotation drive unit 13 is driven based on the approximate rotation angle θt calculated in step S34 for the set target RTt, and the telescope 6c is directed approximately in the direction of the target RT as shown in Figure 10 (C).

[0080] Thereafter, in step S37, the precision collimation execution unit 136 executes automatic collimation using the automatic collimation unit 15, drives the rotation drive unit 13 so as to collimate the center O of the prism 8, performs precise collimation, and ends the process.

[0081] 11 to 13, the calculation of the approximate rotation angle in step S34 will be described in detail. When the process of calculating the approximate rotation angle starts, in step S41, the approximate rotation angle calculation unit 133 calculates the approximate distance from the surveying instrument 10 to the approximate target RT from the dimensions of the approximate target RT in the image 70. For this purpose, the correlation between the dimensions of the approximate target RT in the image 70 and the distance from the surveying system 100 is stored in advance in the storage unit 51.

[0082] Next, in step S42, the approximate rotation angle calculation unit 133 calculates the approximate rotation angle C of the target RT in the image 70. ＲＴ The position of the surveying instrument 10 is １０ The angle θc (θcv, θch) is calculated based on the approximate center C of the target RT. ＲＴ For example, as shown in FIGS. 12(A) and 12(B), the center of gravity G of the approximate targets RT1 and RT2 detected as segments is ＲＴ１ , G ＲＴ２ is set as:

[0083] Next, in step S43, the approximate rotation angle calculation unit 133 calculates the approximate rotation angle θt (θtv, θth) for directing the collimation optical axis A of the telescope 6c toward the approximate target RT from the approximate target position in the image and the approximate distance to the approximate target.

[0084] The horizontal approximate rotation angle θth is calculated as follows: Because the optical axis of the camera 14 coincides with the collimation optical axis A of the telescope 6c in the horizontal direction, the approximate rotation angle calculation unit 133 can calculate the angle with respect to the optical axis of the camera 14 (i.e., the collimation optical axis A of the telescope 6c) by determining the position of the pixel on the image sensor. Therefore, the approximate rotation angle calculation unit 133 calculates the horizontal approximate rotation angle θth from the difference between the current horizontal angle of the collimation optical axis A of the telescope 6c and the horizontal angle θch of the approximate target RT.

[0085] The approximate vertical rotation angle θtv can be calculated as follows. For the sake of convenience, in FIG. 13, the telescope 6c and the camera 14 are depicted separated in the vertical direction. As shown in FIG. 13, the horizontal position is used as a reference for the collimation optical axis A of the telescope 6c. The center O of the camera 14 is １４is offset by Δp in the vertical direction from the center of the telescope 6c. Furthermore, using trigonometric functions, the approximate vertical rotation angle θtv for directing the telescope 6c from the vertical angle of the collimation optical axis A to approximately the θcv of the target RT is calculated from the position θc(θcv, θch) of the prism 8 calculated in step S12 from the optical axis B of the camera 14 and the distance ds to the prism 8 calculated in step S11, is calculated by the following equation: θtv=tan-1{(l+Δp) / dh} (Equation 1) Note that if the collimation optical axis A of the telescope 6c is not horizontal, the calculation can be performed in a similar manner by taking into account the deviation from the horizontal (vertical angle).

[0086] Then, in step S44, it is determined whether the approximate rotation angles have been calculated for all the detected approximate targets, and steps S41 to S44 are repeated until the rotation angles have been calculated for all the approximate targets RT (until the result is Yes).

[0087] In step S41, the distance d from the surveying instrument 10 to the approximate target RT is ｓ The approximate rotation angle calculation unit 133 may be configured to calculate the approximate rotation angle from a known depth estimation model such as ZoeDepth, instead of calculating it from the dimensions of the target RT in the image 70.

[0088] (Details of Identifying a General Target) Next, with reference to FIG. 14, the details of the process of identifying a general target RT in step S35 will be described. When the identification of a general target RT is started, at least one general target RT has been detected. For example, as shown in FIG. 10A, an image 70 may contain multiple general targets RT, and may contain both of two types of general targets RT1 and RT2. In step S35, one general target to be the target of turning and rotating is identified from such general targets.

[0089] When the process starts, first, in step S51, the outline target identification unit 134 determines whether or not the jacket 7 (second outline target RT2) of the two outline targets RT1 and RT2 has been detected in the image 70.

[0090] If the jacket 7 is detected (Yes), in step S52, the general target identification unit 134 determines whether the prism 8 (first general target RT1) is also detected. If the prism 8 is also detected (Yes), the process proceeds to step S53, where it is determined whether the approximate distance to the jacket 7 (second general target RT2) calculated in step S11 matches the approximate distance to the prism 8 (first general target RT1). Note that "match" does not require a strict match; it is sufficient to match approximately, and it is sufficient to match to the extent that it can be determined that they are in almost the same position.

[0091] If there is a match, it is estimated that the prism 8 (first outline target RT1) is being held by the worker OP wearing the jacket 7 (second outline target RT2), as shown in the right part of the image 70 in Figure 10 (A).

[0092] Furthermore, in step S53, it is simultaneously determined whether the approximate distance to the jacket 7 (second approximate target RT2) and the approximate distance to the prism 8 (first approximate target RT1) are equal to or less than a threshold value Th. The dimensions of the jacket 7 are greater than the dimensions of the prism 8. Therefore, due to the resolution of the image sensor of the camera 14, the detection limit distance of the prism 8 is shorter than that of the jacket 7. Furthermore, at the same distance, the detection accuracy of the approximate target RT and the center C of the approximate target RT are different. ＲＴ The calculation accuracy of the jacket 7 is higher than that of the prism 8. Therefore, the prism 8 is suitable for relatively short distances (for example, 5 m or less), while the jacket 7 is suitable for relatively long distances (for example, 10 to 100 m). On the other hand, the center O of the prism 8, which is the final collimation target, is located at the center C of the jacket 7. ＲＴ２ Therefore, when the prism 8 is rotated approximately, it may not be possible to capture the prism 8 within the range of the telescope 6c at a short distance. The threshold value Th is set taking this into consideration.

[0093] In order to reliably place the prism 8 within the field of view of the telescope 6c when the jacket 7 is the target of turning, it is preferable that the operator OP always hold the prism 8 near the center of the jacket 7. However, this is not limiting, and the positional relationship between the jacket 7 and the prism 8 may be kept constant, and the approximate rotation angle θt may be calculated from this positional relationship when turning.

[0094] If, in step S53, the approximate distance to the jacket 7 and the approximate distance to the prism 8 match and are equal to or less than the threshold value (Yes), the process proceeds to step S54, where the outline target identification unit 134 designates the prism 8 closest to the jacket 7 as the target target RTt, and ends the outline target setting process. On the other hand, if, in step S53, no prism is detected, or the distance to the jacket 7 and the prism 8 (first outline target RT1) is greater than the threshold value Th (No), the detected jacket 7 (first outline target RT1) is designated as the target target RTt, and ends the outline target setting process.

[0095] Furthermore, if the jacket 7 is not detected in step S51 (No), the rough target identification unit 134 determines in step S56 whether or not multiple prisms 8 (first rough targets RT1) have been detected. This is because, at the surveying site, the operator OP does not necessarily hold the prism 8, and multiple prisms 8 (first rough targets RT1) may be installed in preset positions, and automatic collimation and measurement may be performed sequentially.

[0096] If multiple prisms 8 are detected in step S56 (Yes), the process proceeds to step S57, where the general target identification unit 134 designates the prism 8 (first general target RT1) closest to the center of the telescope 6c as the target target RTt, and ends the general target setting process. On the other hand, if multiple prisms are not detected in step S56, the process proceeds to step S58, where the detected one prism 8 (first general target RT1) is designated as the target target RTt, and ends the general target identification process.

[0097] Furthermore, if multiple prisms are detected in step S56 (Yes), instead of step S57, the operator may be notified that multiple prisms have been detected, and the information may be displayed on the display unit 18, allowing the operator OP to select which prism to turn around at via the input unit 17.

[0098] (Technical Effect) As described above, the surveying system 100 is configured to incorporate the object detection device 90A according to the first embodiment, and therefore, each time a rough target RT is detected, the image 70 is displayed on the display unit 18, and the object to be detected can be automatically identified without the operator OP having to manually identify pixels, thereby reducing the workload, thereby achieving the same effect as the object detection device 90A according to the first embodiment. It goes without saying that the same effect can be achieved even if the object detection device 90 is incorporated instead of the object detection device 90A. The object detection devices 90 and 90A do not require the operator to manually specify segments from a segmentation image, and are therefore particularly suitable for surveying devices that are operated remotely.

[0099] Furthermore, by applying the object detection device 90A, it is only necessary to prepare at least one reference image for the outline target RT, which has the effect of reducing manufacturing effort and costs compared to the case where other object detection technologies, in particular object detection technologies using machine learning that require a large number of sets of annotated training data, are introduced.

[0100] Furthermore, the surveying system 100 according to the second embodiment has the following problems with its automatic turning function, which detects the direction of the prism and automatically rotates the telescope in the direction of the prism so that the surveying instrument can collimate the prism when starting measurement or when automatic tracking fails. For example, the technology of Patent Document 1 has the problem that it takes time to search for the prism. Furthermore, the technology of Patent Document 2 has the problem that in addition to software, hardware such as a device for transmitting and receiving the fan beam is required, resulting in high implementation costs.

[0101] However, the surveying system 100 according to this embodiment is configured to provide a camera 14 with a wider angle of view than the telescope 6c, detect the approximate target RT from the image 70 acquired by the camera 14, calculate the approximate rotation angle for pointing the telescope 6c toward the approximate target RT from that position, and then perform approximate turning-around rotation. As a result, faster automatic collimation is possible compared to the conventional technology of searching for the target by rotating the telescope 6c. With the above configuration, the automatic turning-around function can be added simply by adding the camera 14 to a conventional surveying instrument, thereby achieving the unique effect of providing a surveying system with an automatic turning-around function at a relatively low cost.

[0102] Furthermore, in the surveying system 100, two types of approximate targets RT are set, each having different dimensions: a prism 8 and the worker's jacket (clothing) 7. In relation to the resolution of the image 70, the prism 8 is set as the turning target within the distance range within the threshold Th at which the prism 8 can be detected, and the jacket 7 is set as the turning target within the distance range outside the threshold Th. This makes it possible to achieve an approximate turning function over a wider distance range than when only one of the targets is set as the approximate target RT.

[0103] However, this does not mean that the present invention is limited to using two types of outline targets RT, and one type may be used. For example, in a distance range where the prism 8 can be detected with appropriate accuracy, only the prism 8 may be set as the outline target RT.

[0104] In the above explanation, the prism 8 and the jacket 7 of the worker are set as the general target RT, but this is not limiting, and a highly visible object (for example, a flat plate colored in a predetermined color) may be attached near the prism 8 and used as the general target RT. However, by setting the prism 8 and the jacket 7 of the worker OP, which are existing elements that have been used conventionally to operate a surveying system, as the general target RT, there is no need to use a new element as the general target RT.

[0105] Furthermore, in the above explanation, an example was described in which the turning-around function of this embodiment is used as a preliminary step to automatic aiming, but if the surveying system 100 is equipped with an automatic tracking function, it can be configured to use the turning-around function to capture the approximate target RT, and then execute the automatic tracking function.

[0106] 3. Modifications The surveying system according to the second embodiment can be modified as follows. (Modification 1) As described above, at least one reference image 81 is required for calculating the reference feature 86, which is the reference data 80. However, as shown in FIG. 15 , it is also possible to register not only a front image (left image) of one type of rough target RT2, but also multiple images such as a side image (right image) or a back image (not shown), or images of various postures. This is because doing so improves the detection accuracy of the rough target RT. Such modifications can also be made when the object detection device according to the first embodiment is applied to other surveying systems.

[0107] Furthermore, although not shown in the figure, reference data 80 may be created using reference images 81 at multiple distances, such as 10 m, 20 m, 50 m, etc., from the camera 14 to the approximate target RT. This also improves the detection accuracy of the approximate target RT.

[0108] (Variation 2) There may be, for example, multiple types of prisms 8 (e.g., prisms 8a, 8b, and 8c) used in the surveying system 100 according to the second embodiment. In such a case, the reference image 81 for calculating the reference feature 86, which is the reference data 80, may be configured to register three types of reference data for the first rough targets RT1a, RT1c, and RT1c, as shown in Fig. 16, so that each prism can be detected individually. In this way, even when multiple types of inspection targets are mixed, prisms can be accurately detected, reducing the likelihood of detection errors and improving work efficiency.

[0109] 17, in the surveying system 100 according to the second embodiment, the general target detection unit 132 may be configured as a general target 132A including a detection object detection unit 132a and a color determination unit 132b having the same functions as the general target detection unit 132. In this case, the color of the jacket 7 is stored in the memory unit 16. Then, in detecting the general target RT in the image in step S32, the detection object detection unit 132a first detects whether or not the general target RT is present in the image 70, and if present, detects its position and range.

[0110] Next, if the detected outline target RT is the outline target RT2 corresponding to the jacket 7, the color determination unit 132b determines what color the outline target RT2 in the image 70 is.

[0111] In this case, in step S51, the outline target identification unit 134 determines whether or not a jacket 7 (second outline target RT2) of a pre-stored color is detected among the two outline targets RT1 and RT2 in the image 70. In this way, by determining the color in addition to whether or not it is a jacket 7, it is possible to improve the detection accuracy of the jacket 7.

[0112] 18 , in the surveying system 100 according to the second embodiment, the general target detection unit 132 may be configured as a general target 132B including a detection object detection unit 132a having the same function as the general target detection unit 132, and a pose estimation unit 132c. The pose estimation unit 132c estimates the pose of the worker OP wearing the jacket 7 in the image using a known pose recognition model configured to estimate a person's pose from an image by machine learning.

[0113] In detecting the outline target RT in the image in step S32, the detection object detection unit 132a first detects whether or not there is a outline target RT in the image 70, and if there is, detects its position and range. Next, if the detected outline target RT is the outline target RT2 corresponding to the jacket 7, the pose estimation unit 132c estimates the pose of the worker OP wearing the outline target RT2.

[0114] In this case, in step S51, the general target identification unit 134 determines whether the pose estimated from the jacket 7 in the image 70 is a specific pose (for example, a pose facing the camera 14 directly as in Figure 14 (A)), and if it is a specific pose, proceeds to step S52.

[0115] In this way, by determining the pose of the worker OP wearing the jacket 7, the jacket 7 to be detected can be accurately identified, thereby improving the detection accuracy.

[0116] 4. Other Modifications The surveying system 100 according to the second embodiment may be modified as follows.

[0117] (1) In the surveying system 100 according to the second embodiment, the camera 14 is provided on top of the telescope 6c, but the camera 14 may be provided inside the telescope 6c with its optical axis B coaxial with the collimation optical axis A of the telescope 6c. In this case, the same effect as that of the surveying system 100 can be achieved by configuring the objective lens common to the camera 14 and the telescope 6c to be zoomable in the range from the wide angle of the camera 14 to the narrow angle of the telescope 6c.

[0118] (2) In the surveying system 100 according to the second embodiment, the camera 14 is provided on top of the telescope, but this is not limiting, and the surveying system 100 may be configured so that the positional relationship between the camera 14 and the center of the surveying system 100 and the relationship between the optical axis B of the camera and the direction of the collimation optical axis A of the telescope 6c are known (or are assumed to be known). This is because a person skilled in the art would be able to calculate the approximate rotation angle in accordance with the method described above.

[0119] (3) Furthermore, the surveying system 100 according to the second embodiment is configured as a single total station. However, as described in the second embodiment, the present invention is not limited to this, and some of the functions of the control and calculation unit 130 may be configured to be executed by another processor, that is, another computer terminal, a server computer, or a cloud computer.

[0120] REFERENCE SIGNS LIST 1: Surveying system 6c: Telescope 7: Jacket 8: Prism 8a: Prism 8b: Prism 8c: Prism 11: Distance measuring device 12: Angle detector 13: Rotation drive unit 14: Camera 16: Memory unit 30: Control and calculation unit 51: Memory unit 70: Image 70: Measurement image 71: Feature image 72: Similarity map 80: Reference data 81: Reference image 82: Mask image 85: Feature image 86: Reference feature amount 90: Object detection device 90A: Object detection device 91: Control and calculation unit 92: Memory unit 100: Surveying system 130: Control and calculation unit 132A: Outline target 132B: Outline target 132a: Detection object detection unit 132b: Color judgment unit 912: Detection object detection unit 991: Control and calculation unit A: Collimation optical axis B: Optical axis CRT: Center CRT2: Center K: Feature point M1: Segmentation model OP: Operator RT: Outline target RT1: First outline target RT1a: First outline target RT2: Second outline target T: Detection object ds: Distance θc: Angle θt: Outline rotation angle

Claims

1. An object detection device comprising: a control and calculation unit having at least one processor and at least one memory; and a storage unit having a computer-readable storage medium, wherein the storage unit stores, as reference data, reference feature amounts that are feature amounts of the detection object in a reference image including the detection object; the control and calculation unit generates a feature image by converting each pixel from a measurement image into a feature amount; compares the feature amount of each pixel with the reference feature amount to generate a similarity map; identifies the most similar pixel in the similarity map as a feature point; segmentes the measurement image using a segmentation model based on machine learning using a large number of images and a mask image; automatically designates the pixel of the feature point in the segmentation image; and detects a segment including the designated pixel as the detection object.

2. The object detection device described in claim 1, characterized in that the reference feature is obtained by segmenting the reference image using the segmentation model and averaging the feature of segments selected from the segmentation image by specifying pixels included in the object to be detected.

3. A surveying system comprising: an object detection device according to claim 1 or 2; a distance measuring device housed in a telescope, which sends distance measuring light to a prism and receives reflected light from the prism to measure the distance to the center of the prism; an angle detector which detects the angle of the collimation optical axis of the telescope; a rotation drive unit which drives and rotates the telescope; and a camera which has an angle of view wider than the field of view of the telescope and which captures an image in front of the telescope as the measurement image; wherein a general target for roughly turning the prism is set, the general target is the object to be detected, and the reference image is an image of the general target, and the control and calculation unit detects the general target from the measurement image, calculates a general rotation angle for roughly turning the collimation optical axis of the telescope in the direction of the general target based on the position of the general target, and sets a general target to be turned from the general target in the measurement image, A surveying system characterized in that the surveying system is configured to drive the rotation drive unit so as to rotate the telescope by the approximate rotation angle.

4. The surveying system according to claim 3, characterized in that the outline targets include a first outline target and a second outline target larger in size than the first outline target, and the target outline target is set according to the approximate distance to the first outline target and the approximate distance to the second outline target.

5. A surveying system according to claim 4, wherein the first outline target is the prism, and the second outline target is clothing of a worker holding the prism.

6. The surveying system according to claim 3, wherein the general target is the prism.

7. The surveying system according to claim 3, wherein the reference images are images of the object to be detected viewed from a plurality of directions.

8. The surveying system according to claim 3, wherein the reference images are images of the approximate target taken at a plurality of distances from the camera.

9. The surveying system described in claim 3, characterized in that the detection object detection unit includes a color determination unit that determines the color of the outline target in the image, the memory unit stores a predetermined color as the color of the outline target, and the control and calculation unit detects the outline target of the predetermined color as the outline target.

10. The surveying system according to claim 3, characterized in that the outline target is clothing worn by a worker, the detection object detection unit includes a pose estimation unit that estimates the pose of the worker, and the control and calculation unit detects the outline target as the outline target when it exhibits a predetermined pose.

Images