Positioning system and positioning method

Abstract

To accurately pinpoint the location of the subject. [Solution] The positioning system comprises a camera that captures images of a subject, a sensor attached to the subject that detects the subject's acceleration, and a processing device that determines the subject's position, wherein if it is determined that the subject is located within a range less than a predetermined threshold distance from the camera, the processing device uses the camera image to determine the subject's position, and if it is determined that the subject is located within a range greater than or equal to the threshold distance from the camera, or if it is determined that the subject is located in a blind spot of the camera, the processing device uses the sensor's detection result to determine the subject's position.

Description

Technical Field

[0006] , ,

[0005] , ,

[0001] The present disclosure relates to a positioning system and a positioning method.

Background Art

[0002] A technique for positioning an object using a plurality of types of sensors such as a camera, a GNSS (Global Navigation Satellite System) receiver, and an IMU (Inertial Measurement Unit) is known (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Advantages and disadvantages vary depending on the type of sensor. Therefore, in the technique of positioning an object using a plurality of types of sensors, there is room for improvement in order to improve the positioning accuracy.

Means for Solving the Problems

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to a first embodiment of the present disclosure, a positioning system is provided. This positioning system includes a camera for capturing images of a subject, a sensor attached to the subject for detecting the subject's acceleration, and a processing device for determining the subject's position, which determines that if the subject is located within a range less than a predetermined threshold distance from the camera, it uses the image from the camera to determine the subject's position, and if the subject is located within a range greater than or equal to the threshold distance from the camera, or if the subject is located in a blind spot of the camera, it uses the detection result of the sensor to determine the subject's position. This type of positioning system allows for the advantage of both camera-based and sensor-based positioning to compensate for the disadvantages of each. Therefore, positioning accuracy can be improved. (2) In the positioning system of the above form, the processing device may detect the left and right leg ends of the subject from the image of the camera, and if it determines that the subject is located within a range where the distance from the camera is less than the threshold, it may identify the center position of the left and right leg ends as the position of the subject. This type of positioning system can improve the accuracy of camera-based positioning. (3) In the positioning system of the above form, the sensor may be attached to the torso of the subject. This type of positioning system can improve the accuracy of positioning using sensors. (4) A second embodiment of the present disclosure provides a positioning method. This positioning method comprises the steps of: capturing an image of a subject with a camera; detecting the acceleration of the subject with a sensor attached to the subject; and identifying the position of the subject, which, if it is determined that the subject is located within a range less than a predetermined threshold distance from the camera, uses the image from the camera to identify the position of the subject; and if it is determined that the subject is located within a range greater than or equal to the threshold distance from the camera, and if it is determined that the subject is located in a blind spot of the camera, uses the detection result of the sensor to identify the position of the subject. This type of positioning method allows the advantages of both camera-based and sensor-based positioning to compensate for the disadvantages of each. Therefore, positioning accuracy can be improved. This disclosure can also be implemented in various forms other than positioning systems and positioning methods. It can be implemented in the form of devices, computer programs, and recording media on which the computer programs are recorded. [Brief explanation of the drawing]

[0007] [Figure 1] The first explanatory diagram showing the configuration of the positioning system. [Figure 2] A second explanatory diagram showing the configuration of the positioning system. [Figure 3] An explanatory diagram showing how to position a worker using a camera. [Figure 4] An explanatory diagram showing how to position a worker using sensors. [Figure 5] A flowchart illustrating the positioning process. [Modes for carrying out the invention]

[0008] A. Embodiments: Figure 1 is a first explanatory diagram showing the configuration of a positioning system 10 in one embodiment of the present disclosure. Figure 2 is a second explanatory diagram showing the configuration of the positioning system 10. As shown in Figure 1, the positioning system 10 includes a camera 20, a sensor 30, and a processing unit 40. The positioning system 10 measures the position of a target person. In this embodiment, the target person is a worker WK performing work in the factory work area WR. The worker WK moves on foot within the work area WR. In Figure 1, the movement path RT of the worker WK is represented by a dashed line.

[0009] Camera 20 is positioned such that at least a portion of the work area WR is within its field of view. The position and orientation of camera 20 are fixed. In this embodiment, camera 20 is a monocular camera. Camera 20 images the work area WR where worker WK is present and generates an image of the work area WR including worker WK. However, camera 20 has blind spots. Therefore, even if worker WK is present in the work area WR, worker WK may not be included in the image. For example, if there is an obstacle OB in the work area WR, the blind spot of camera 20 will include the shadow of the obstacle OB.

[0010] Sensor 30 is attached to worker WK and detects physical quantities related to the worker WK's motion state. In this embodiment, sensor 30 is an inertial measurement unit (IMU) and includes an acceleration sensor that detects acceleration in three axes and an angular velocity sensor that detects angular velocity around three axes. That is, in this embodiment, sensor 30 detects acceleration and angular velocity as physical quantities related to the worker WK's motion state. Sensor 30 is fixed to the torso of the work clothes worn by worker WK, more specifically to the waist of the work clothes. A communication device (not shown) that communicates with the processing device 40 via wireless communication is also fixed to the work clothes, and the detection results of sensor 30 are transmitted to the processing device 40 via the communication device.

[0011] The processing unit 40 performs positioning of the worker WK using the camera 20 and sensor 30. In this embodiment, the processing unit 40 is located outside the work area WR. However, the processing unit 40 may be located inside the work area WR.

[0012] As shown in Figure 2, the processing unit 40 is composed of a computer comprising a processor 41, memory 42, input / output interface 43, and internal bus 44. The processor 41, memory 42, and input / output interface 43 are connected via the internal bus 44 to enable bidirectional communication. A camera 20 and a communication device 45 that communicates wirelessly with the sensor 30 are connected to the input / output interface 43 via a communication cable. Note that the camera 20 may be connected to the processing unit 40 by wireless communication instead of wired communication.

[0013] The processor 41 functions as a first positioning unit 11 that positions worker WK using camera 20, a second positioning unit 12 that positions worker WK using sensor 30, and an integration unit 13 that determines the location of worker WK using the positioning results of the first positioning unit 11 and the second positioning unit 12, by executing a positioning program PG pre-stored in memory 42. Memory 42 pre-stores a skeleton detection model MD, which is a machine learning model that detects the skeleton of worker WK from images taken by camera 20.

[0014] Figure 3 is an explanatory diagram showing a method for positioning a worker WK using a camera 20. First, the first positioning unit 11 acquires an image from the camera 20. The image from the camera 20 shows a view of the work area WR from diagonally above. The first positioning unit 11 uses a skeleton detection model MD to detect the skeleton BN of the worker WK from the image from the camera 20. The skeleton BN includes multiple key points KP that indicate the positions of facial features and joint points. In this embodiment, the multiple key points KP include key points KP that indicate the end positions of the left and right legs, more specifically, the positions of the left and right heels. The first positioning unit 11 identifies the heel midpoint PM, which is the center position of the left and right heels. However, if the key point KP indicating the end position of the leg is a key point KP indicating the ankle instead of the heel, the first positioning unit 11 may identify the center positions of the left and right ankles instead of the center positions of the left and right heels. The first positioning unit 11 converts the position coordinates of the heel midpoint PM in the image coordinate system to the position coordinates of the heel midpoint PM in the work area coordinate system. The image coordinate system is a two-dimensional orthogonal coordinate system having an H-axis parallel to the horizontal direction of the image and a V-axis parallel to the vertical direction of the image, and the work area coordinate system is a two-dimensional orthogonal coordinate system having an X-axis and a Y-axis parallel to the floor surface of the work area WR. The first positioning unit 11 can convert the position coordinates of the heel midpoint PM in the image to the position coordinates of the heel midpoint PM in the work area coordinate system by performing an affine transformation based on the correspondence between each position on the actual floor surface of the work area WR and each of the above positions in the image, thereby converting an image of looking down on the work area WR from diagonally above to an image of looking down on the work area WR from directly above. The first positioning unit 11 transmits the position coordinates of the heel midpoint PM in the work area coordinate system to the integration unit 13 as the positioning result of the worker WK measured by the camera 20.

[0015] FIG. 4 is an explanatory diagram showing a method for positioning operator WK using sensor 30. First, the second positioning unit 12 acquires the detection result of sensor 30. The detection result of sensor 30 includes the accelerations of three axes and the angular velocities around the three axes in the sensor coordinate system. The second positioning unit 12 calculates the acceleration of operator WK in the work area coordinate system using the detection result of sensor 30, and calculates the position coordinates of operator WK in the work area coordinate system based on the acceleration of operator WK in the work area coordinate system. As is well known, velocity can be obtained by integrating acceleration over time, and the amount of movement can be obtained by integrating velocity over time. If the position coordinates before movement are known, the position coordinates after movement can be obtained from the position coordinates before movement and the amount of movement. The second positioning unit 12 transmits the position coordinates of operator WK in the work area coordinate system to the integration unit 13 as the positioning result of operator WK by sensor 30.

[0016] FIG. 5 is a flowchart showing the procedure of the positioning process executed by the integration unit 13 of the processing device 40. In the present embodiment, the positioning process is started in a state where operator WK is stationary at a predetermined initial position with a predetermined initial posture. First, in step S110, the integration unit 13 acquires the initial position of operator WK at the start of the positioning process.

[0017] In step S120, the integration unit 13 causes the first positioning unit 11 to perform the positioning of operator WK by camera 20, and causes the second positioning unit 12 to perform the positioning of operator WK by sensor 30, and acquires the positioning results by camera 20 and the positioning results by sensor 30. In the first step S120 after starting the positioning process, the integration unit 13 acquires the positioning result indicating the position coordinates of operator WK at a time interval Δt after starting the positioning process, and in the second and subsequent steps S120 after starting the positioning process, the integration unit 13 acquires the positioning result indicating the position coordinates of operator WK at Δt after the previous positioning. In the present embodiment, the length of Δt is 0.1 second. The length of Δt is not limited to 0.1 second, and may be shorter than 0.1 second or longer than 0.1 second. When positioning a walking operator WK, the length of Δt is preferably within 0.3 second.

[0018] In step S130, the integration unit 13 determines whether an operator WK exists outside the blind spot of the camera 20. In this embodiment, when the operator WK exists in the blind spot of the camera 20, the operator WK is not detected by the camera 20, and the position coordinates of the operator WK do not exist in the positioning result by the camera 20. Therefore, in this embodiment, when the position coordinates of the operator WK exist in the positioning result by the camera 20, the integration unit 13 determines that the operator WK exists outside the blind spot of the camera 20, and when the position coordinates of the operator WK do not exist in the positioning result by the camera 20, the integration unit 13 determines that the operator WK exists in the blind spot of the camera 20.

[0019] If it is determined in step S130 that an operator WK exists outside the blind spot of the camera 20, the integration unit 13 determines in step S140 whether an operator WK exists in a range (see FIG. 1) where the distance from the camera 20 is less than a predetermined threshold value DL. In this embodiment, the integration unit 13 determines whether an operator WK exists in the above range by using the positioning result by the camera 20. The threshold value DL can be determined based on the results of a test performed in advance, for example. In the test, the positioning by the camera 20 is performed multiple times while changing the position of the operator WK within the work area WR, and the positioning result by the camera 20 is compared with the actual position of the operator WK. The position where the distance from the camera 20 is the maximum among the positions where the deviation between the positioning result by the camera 20 and the actual position of the operator WK is within the allowable range is specified, and the distance between the specified position and the camera 20 is determined as the threshold value DL.

[0020] If it is determined in step S140 that an operator WK exists in a range where the distance from the camera 20 is less than the threshold value DL, the integration unit 13 specifies the position of the operator WK by using the positioning result by the camera 20 in step S150, and records the specified position of the operator WK in the memory 42. The position of the operator WK specified in step S150 can be expressed by the following equations (1A) and (2A). X(T)=Xc(T) ···(1A) Y(T)=Yc(T) ···(2A) Here, X(T) and Y(T) are the X and Y coordinates of worker WK at the Tth positioning measurement, and Xc(T) and Yc(T) are the X and Y coordinates of worker WK obtained from the Tth positioning measurement by camera 20.

[0021] If it is determined in step S130 that worker WK is in the blind spot of camera 20, and if it is determined in step S140 that worker WK is within a range where the distance from camera 20 is greater than or equal to the threshold DL, the integration unit 13 identifies the position of worker WK using the positioning results from sensor 30 in step S155 and records the identified position of worker WK in memory 42. The position of worker WK identified in step S155 can be expressed by the following equations (1B) and (2B). X(T)=X(T-1)+(Xb(T)-Xb(T-1)) ···(1B) Y(T)=Y(T-1)+(Yb(T)-Yb(T-1)) ···(2B) Here, X(T) and Y(T) are the X and Y coordinates of worker WK at the Tth positioning measurement, X(T-1) and Y(T-1) are the X and Y coordinates of worker WK at the T-1th positioning measurement, Xb(T) and Yb(T) are the X and Y coordinates of worker WK obtained from the Tth positioning measurement by sensor 30, and Xb(T-1) and Yb(T-1) are the X and Y coordinates of worker WK obtained from the Tth positioning measurement by sensor 30. Note that (Xb(T)-Xb(T-1)) and (Yb(T)-Yb(T-1)) correspond to the X and Y components of the amount of movement of worker WK from the T-1th positioning measurement to the Tth positioning measurement.

[0022] After step S150 or step S155, in step S160, the integration unit 13 determines whether or not to terminate the positioning process. If it is determined in step S160 that the positioning process should not be terminated, the integration unit 13 repeats the processes from step S120 to step S160 until it is determined in step S160 that the positioning process should be terminated. If it is determined in step S160 that the positioning process should be terminated, the integration unit 13 terminates the positioning process.

[0023] However, if worker WK is in the blind spot of camera 20, there is a problem in that positioning by camera 20 is impossible. To address this problem, in this embodiment, if the processing unit 40 determines that worker WK is in the blind spot of camera 20, it uses the positioning result from sensor 30 to determine the position of worker WK. Therefore, even if worker WK is in the blind spot of camera 20, the position of worker WK can be determined using sensor 30.

[0024] Furthermore, as the distance between the camera 20 and the worker WK increases, the worker WK is represented by fewer pixels in the image from the camera 20, which leads to a problem of reduced positioning accuracy by the camera 20. To address this problem, in this embodiment, if the processing unit 40 determines that the worker WK is located within a range greater than or equal to a threshold DL from the camera 20, it uses the positioning results from the sensor 30 to determine the position of the worker WK. Therefore, even when the distance between the camera 20 and the worker WK is large, the position of the worker WK can be accurately determined using the sensor 30.

[0025] On the other hand, the more consecutive positioning measurements are taken by the sensor 30, the more positional errors accumulate, leading to a decrease in positioning accuracy. To address this problem, in this embodiment, the processing unit 40 determines that worker WK is located within a range where the distance from camera 20 is less than the threshold DL, and uses the positioning results from camera 20 to determine the position of worker WK. Therefore, when the distance between camera 20 and worker WK becomes shorter, the errors accumulated by consecutive positioning measurements by sensor 30 can be reset.

[0026] According to the positioning system 10 of this embodiment described above, if the processing unit 40 determines that a worker WK is located within a range where the distance from the camera 20 is less than a threshold DL, it uses the positioning result from the camera 20 to determine the position of the worker WK. If the processing unit 40 determines that a worker WK is located within a range where the distance from the camera 20 is greater than or equal to the threshold DL, or if it determines that a worker WK is located in the blind spot of the camera 20, it uses the positioning result from the sensor 30 to determine the position of the worker WK. In this way, the advantages of the other compensate for the disadvantages of the positioning by the camera 20 and the positioning by the sensor 30. Therefore, the position of the worker WK can be determined with high accuracy.

[0027] Furthermore, in this embodiment, the processing device 40 detects the center positions of the left and right leg ends of worker WK from the image of the camera 20, more specifically, the center positions of the left and right heels, and identifies these center positions as the position of worker WK. Since the leg ends of worker WK are close to the floor surface of the work area WR, errors are less likely to occur when converting the position coordinates in the image coordinate system to the position coordinates in the work area coordinate system. For this reason, the position of worker WK can be accurately identified using the camera 20.

[0028] Furthermore, in this embodiment, the sensor 30 is attached to the torso of the worker WK, more specifically, to the waist of the worker WK. If the sensor 30 were attached to the head or arms of the worker WK, the worker WK might move their head or arms as they work, potentially leading to a false detection that the worker WK is moving even though they are not. To address this problem, in this embodiment, the processing unit 40 uses the sensor 30, which is attached to the waist, a part of the body whose position is less likely to change during work compared to the head or arms, to determine the worker WK's position. Therefore, the position of the worker WK can be accurately determined using the sensor 30.

[0029] Furthermore, in this embodiment, the walking trajectory of worker WK is obtained from the time-series data of the worker WK's position identified by the processing device 40. Therefore, the walking trajectory can be used to examine the quality of the layout of the work area WR and the quality of the work procedure.

[0030] B. Other embodiments: (B1) In the positioning system 10 of the above embodiment, a monocular camera is used for the camera 20. In contrast, the camera 20 may be a stereo camera, for example, instead of a monocular camera. The camera 20 may be an infrared camera, instead of a visible light camera.

[0031] (B2) In the positioning system 10 of the above embodiment, the processing unit 40 detects the left and right leg ends of worker WK from the image during positioning using the camera 20, and identifies the center position of the left and right leg ends as the position of the subject. Alternatively, the processing unit 40 may identify a position other than the center position of the left and right leg ends as the position of the subject during positioning using the camera 20.

[0032] (B3) In the positioning system 10 of the above embodiment, the sensor 30 is attached to the torso of the worker WK. In contrast, the sensor 30 may be attached to a part of the worker WK other than the torso, such as the head.

[0033] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features in the embodiments corresponding to the technical features in each form described in the summary of the invention can be replaced or combined as appropriate in order to solve some or all of the above-described problems, or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. [Explanation of symbols]

[0034] 10...Positioning system, 11...First positioning unit, 12...Second positioning unit, 13...Integration unit, 20...Camera, 30...Sensor, 40...Processing device, 41...Processor, 42...Memory, 43...Input / output interface, 44...Internal bus, 45...Communicator, BN...Skeleton, DL...Threshold, KP...Keypoint, MD...Skeleton detection model, OB...Obstacle, PG...Positioning program, PM...Heel midpoint, RT...Movement path, WK...Worker, WR...Work area

Claims

1. A positioning system, A camera that images the subject, A sensor attached to the subject to detect the subject's acceleration, A processing device for determining the location of the subject, wherein if it is determined that the subject is located within a range less than a predetermined threshold distance from the camera, the processing device determines the location of the subject using the image from the camera, and if it is determined that the subject is located within a range greater than or equal to the threshold distance from the camera, or if it is determined that the subject is located within a blind spot of the camera, the processing device determines the location of the subject using the detection result of the sensor, A positioning system equipped with the following features.

2. A positioning system according to claim 1, The aforementioned processing apparatus is The left and right leg ends of the subject are detected from the image of the camera. A positioning system that, when it is determined that the subject is located within a range of less than the threshold distance from the camera, identifies the center position of the left and right leg ends as the subject's position.

3. A positioning system according to claim 1, The aforementioned sensor is a positioning system attached to the torso of the subject.

4. A positioning method, The process of capturing images of the subject using a camera, A step of detecting the acceleration of the subject using a sensor attached to the subject, A step to identify the location of the subject, wherein if it is determined that the subject is located within a range where the distance from the camera is less than a predetermined threshold, the location of the subject is identified using the image from the camera; if it is determined that the subject is located within a range where the distance from the camera is greater than or equal to the threshold, or if it is determined that the subject is located within a range that is a blind spot of the camera, the location of the subject is identified using the detection result of the sensor. A positioning method comprising the following features.

Images