Positioning device, positioning method, and program

Abstract

To derive a position of a moving object more properly.SOLUTION: A positioning device includes a processing unit that derives a position in a real space of a moving object on the basis of a light-emitting position in an image during a first period in which the light-emitting position of a light-emitting device in the image obtained by imaging the moving object attached with the light-emitting device can be specified, and drives the position in the real space of the moving object by estimating the light-emitting position in the image on the basis of features related to the appearance of the moving object in the image during a second period which is different from the first period and in which the light-emitting position in the image cannot be specified.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to a positioning device, a positioning method, and a program. [Background technology]

[0002] 2. Description of the Related Art There is a known technique in which a moving object moving in a space derives its own position based on a light emission signal from a light emitting device provided in the space (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-76401 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned conventional technology has a problem in that when the light-emitting device is made smaller or when the distance between the light-emitting device and the moving object is increased, it may become impossible to properly detect the light-emitting signal and derive the position.

[0005] An object of the present invention is to more appropriately derive the position of a moving object. [Means for solving the problem]

[0006] In order to solve the above problems, a positioning device according to the present invention comprises: deriving a position of the moving object in real space based on the light-emitting position in an image captured during a first period during which a light-emitting position of the light-emitting device can be identified in the image captured of the moving object; In a second period different from the first period during which the light-emitting position in the image cannot be identified, the light-emitting position in the image is estimated based on features related to an appearance of the moving object in the image, thereby deriving a position of the moving object in real space. A processing unit is provided.

[0007] In order to solve the above problems, a positioning device according to the present invention comprises: deriving a position of the moving body in real space based on the light-emitting position of the light-emitting device in an image captured by an imaging device attached to the moving body during a first period during which a light-emitting position of the light-emitting device in the image can be identified; deriving a position of the moving object in real space by estimating a light-emitting position of the light-emitting device in the image based on an external appearance feature of a place in the image where the light-emitting device is provided, during a second period different from the first period during which the light-emitting position in the image cannot be identified; A processing unit is provided. Effect of the Invention

[0008] According to the present invention, the position of a moving object can be derived more appropriately. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a positioning system according to a first embodiment. [Diagram 2] FIG. 2 is a diagram for explaining the principle of positioning in the positioning system of the first embodiment. [Diagram 3] FIG. 2 is a block diagram showing a functional configuration of the positioning device. [Figure 4] FIG. 11 is a diagram showing the contents of light emitting device management data. [Diagram 5] FIG. 4 is a diagram showing the contents of positioning history data. [Figure 6] FIG. 1 is a diagram showing an image captured by a certain camera at a certain timing. [Figure 7] FIG. 7 is a diagram showing an image captured by a certain camera at a timing different from that in FIG. 6. [Figure 8] 13 is a flowchart showing a control procedure of the positioning process. [Figure 9] 10 is a flowchart showing a control procedure of a light-emitting position specifying process. [Figure 10] FIG. 11 is a diagram illustrating a configuration of a positioning system according to a second embodiment. [Figure 11] FIG. 11 is a diagram for explaining the principle of positioning in a positioning system according to a second embodiment. [Figure 12] FIG. 2 is a block diagram showing the functional configuration of a vehicle. [Figure 13] FIG. 11 is a diagram showing the contents of light emitting device management data in the second embodiment. [Figure 14] FIG. 11 is a diagram showing the contents of positioning history data in the second embodiment. [Figure 15] FIG. 2 is a diagram showing an image captured by a camera at a certain timing. [Figure 16] 16 is a diagram showing an image captured by a camera at a timing different from that in FIG. 15. FIG. [Figure 17] 10 is a flowchart showing a control procedure of a positioning process in the second embodiment. [Figure 18] 10 is a flowchart showing a control procedure of a light emission position specifying process in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] (First embodiment) <Positioning system configuration> FIG. 1 is a diagram showing the configuration of a positioning system 1 according to the first embodiment. The positioning system 1 of this embodiment derives the positions of subjects M1, M2, ... (hereinafter, referred to as "subject M" when referring to any one subject) in a real space 2, and records and manages the history of the derived positions of each subject M. In this embodiment, the real space 2 is a factory floor, and the subject M is a worker who performs work on that floor. The subject M corresponds to a moving object.

[0012] The positioning system 1 includes a positioning device 10, a plurality of light-emitting devices 30a, 30b, ... (hereinafter, any one of the light-emitting devices will be referred to as "light-emitting device 30"), and a plurality of cameras C1, C2, ... (imaging devices) (hereinafter, any one of the cameras will be referred to as "camera C"). In the following, the positions of the subject M and the camera C in the real space 2 are represented by an XYZ Cartesian coordinate system. The X and Y directions are parallel to the horizontal plane, and the Z direction is vertically upward. In addition, in FIG. 1, the positioning device 10 is installed on a factory floor, but this is not limiting, and the positioning device 10 may be installed outside the floor or in a remote location.

[0013] The light-emitting device 30 functions as a position indicator that indicates the position of the light-emitting device 30 in the real space 2, and is also called a marker. The light-emitting device 30 is attached to the top of the head of the helmet worn by the subject M, and moves together with the subject M. One light-emitting device 30 is attached to each subject M. The light-emitting device 30 may be attached to a position other than the top of the head, as long as the light emitted from the light-emitting device 30 can be seen from the outside. The light-emitting device 30 has, for example, three LEDs (Light Emitting Diodes) that emit red, green, and blue light, and emits light in a unique light-emitting pattern. That is, each light-emitting device 30 changes the light-emitting color in a time series in a different pattern that is predetermined for each light-emitting device 30. The light-emitting device 30 repeatedly executes an operation of changing the light-emitting color according to this light-emitting pattern. Hereinafter, the length of the period from the start to the end of the light-emitting pattern is also referred to as a "light-emitting pattern period." By decoding the light-emitting pattern of each light-emitting device 30, a light-emitting device ID (see FIG. 4) previously assigned to each light-emitting device 30 can be identified. In this manner, the light emitting device 30 is used for visible light communication that transmits information (here, the light emitting device ID) by visible light.

[0014] Each camera C is provided at a predetermined position in the real space 2. In this embodiment, the cameras are attached to the ceiling of the floor. The cameras C are connected to the positioning device 10 via a wireless or wired communication path so as to be able to transmit and receive data. The cameras C capture images 40 (images) (see FIGS. 6 and 7) at a predetermined frame rate and transmit the captured image data 132 (see FIG. 3) to the positioning device 10. Each camera C in this embodiment is provided at a position where it can capture an image of the entire factory floor.

[0015] The positioning device 10 analyzes the moving images transmitted from each camera C to derive the position of each subject M in the real space 2.

[0016] FIG. 2 is a diagram for explaining the principle of positioning in the positioning system 1 of the first embodiment. 2 shows one light emitting device 30, two cameras C1 and C2, a captured image 40a captured by the camera C1, and a captured image 40b captured by the camera C2 at the same time as the captured image 40a. Hereinafter, when the captured images 40a and 40b are not distinguished from each other, they will be referred to as "captured image 40". The position in the captured image 40 is represented by an xy Cartesian coordinate system set in the captured image 40. The x-axis is parallel to the horizontal direction of the matrix arrangement direction of pixels in the captured image 40, and the y-axis is parallel to the vertical direction. The positioning device 10 derives a position P3 (X1, Y1, Z1) of the light-emitting device 30 (subject M) in the real space 2 by the principle of triangulation based on the light-emitting position Q1 (x1, y1) of the light-emitting device 30 in the captured image 40a, the light-emitting position Q2 (x2, y2) of the light-emitting device 30 in the captured image 40b, the position P1 (Xc1, Yc1, Zc1) of the camera C in the real space 2, and the position P2 (Xc2, Yc2, Zc2) of the camera C2 in the real space. Here, the position P1 of the camera C1 and the position P2 of the camera C2 in the real space 2 are measured in advance and stored in the storage unit 13. When multiple light-emitting devices 30a, 30b, ... are captured in the captured images 40a, 40b, each light-emitting device 30 is identified based on its light-emitting pattern, and the light-emitting positions Q1, Q2 of each light-emitting device 30 in the captured images 40a, 40b are specified. Thereafter, each light-emitting device 30 can be positioned in the same manner as above. In the following, any one light-emitting position will be referred to as "light-emitting position Q." It should be noted that by using the images 40 captured by three or more cameras C to derive the position P3 of the subject M, more accurate positioning is possible.

[0017] FIG. 3 is a block diagram showing the functional configuration of the positioning device 10. As shown in FIG. The positioning device 10 includes a CPU 11 (Central Processing Unit) (processing unit, computer), a RAM 12 (Random Access Memory), a storage unit 13, a communication unit 14, and a bus 15. The components of the positioning device 10 are connected to each other via the bus 15. The positioning device 10 may further include an operation unit, a display unit, and the like that are used by an administrator of the positioning system 1.

[0018] The CPU 11 is a processor that reads and executes a program 131 stored in the storage unit 13 and performs various arithmetic processing to control the operation of each unit of the positioning device 10. The positioning device 10 may have multiple processors (e.g., multiple CPUs), and the multiple processes performed by the CPU 11 of this embodiment may be executed by the multiple processors. In this case, a "processing unit" is configured by the multiple processors. In this case, the multiple processors may be involved in a common process, or the multiple processors may independently execute different processes in parallel.

[0019] The RAM 12 provides a working memory space for the CPU 11 and stores temporary data.

[0020] The storage unit 13 is a non-transitory recording medium readable by the CPU 11 as a computer, and stores a program 131 and various data. The storage unit 13 includes a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD). The program 131 is stored in the storage unit 13 in the form of a computer-readable program code. Data stored in the storage unit 13 includes captured image data 132 received from each camera C, light-emitting device management data 133, positioning history data 134, and a machine learning model 135.

[0021] FIG. 4 is a diagram showing the contents of the light emitting device management data 133. As shown in FIG. The light-emitting device management data 133 stores information related to a plurality of light-emitting devices 30 used in the positioning system 1. One data row of the light-emitting device management data 133 corresponds to one light-emitting device 30. Each data row includes data items of "light-emitting device ID," "light-emitting pattern," and "tag information." The “light emitting device ID” is a unique code assigned to each light emitting device 30 . "Light Emitting Pattern" indicates the light emitting pattern of the light emitting device 30 of the data row. In the example shown in Fig. 4, the switching order of the light emitting colors of red (R), green (R), and blue (B) is indicated by the character strings "R", "G", and "B". The "tag information" is metadata added to the training data in the creation (annotation) of training data used in the machine learning of the machine learning model 135 described later. In this embodiment, the subject M to which the light-emitting device 30 of the data row is attached is used as the tag information.

[0022] FIG. 5 is a diagram showing the contents of the positioning history data 134. As shown in FIG. The positioning history data 134 stores information related to the positioning results of the subject M, which are repeatedly performed at each positioning timing. One data row of the positioning history data 134 corresponds to the positioning result at one positioning timing. Each data row includes the data items of "time", "light emission position in image", and "positioning result". "Time" indicates the date and time when the positioning for that data line was performed. "Light-emitting position Q in image" represents the light-emitting position Q (xy coordinates) of the light-emitting device 30 in the image 40 captured by each camera C used in the positioning of that data row. The "positioning result" represents the position P3 (XYZ coordinates) of the subject M in the real space 2, derived based on the "light emitting position in the image".

[0023] The machine learning model 135 is machine-trained by teacher data including at least a part of the captured image 40 in which a predetermined subject such as a subject M appears and tag information related to the subject. When the teacher data is input, the machine learning model 135 identifies the subject included in the image of the input teacher data and outputs the identification result to the CPU 11. More specifically, the machine learning model 135 is machine-trained in advance to extract a person included in the input image without identifying the individual, and further, by additionally learning from the captured image 40 in which a specific subject M appears, the specific subject M can be identified and extracted in distinction from other people (i.e., the individual can be identified). The configuration of the machine learning model 135 is not particularly limited, and may be, for example, a neural network or a support vector machine. The machine learning model 135 may be provided in an external device separate from the positioning device 10. In this case, the CPU 11 of the positioning device 10 transmits and receives data to and from the external device to cause the machine learning model 135 to perform machine learning and obtain the identification result by the machine learning model 135.

[0024] 3 performs a communication operation according to a predetermined communication standard. For example, the communication unit 14 receives captured image data 132 from the camera C through this communication operation. The communication unit 14 may also transmit positioning results and the like included in the positioning history data 134 to a predetermined higher-level device (not shown).

[0025] <Positioning system operation> Next, the operation relating to positioning of the positioning system 1 in the first embodiment will be described. When the positioning operation is started in the positioning system 1, the camera C repeatedly captures the captured images 40 at a predetermined frame rate, and the captured image data 132 is transmitted to the positioning device 10. The frame rate is set so that the interval between captures of the captured images 40 is shorter than the cycle at which the emitted color is switched in the light emission pattern of the light emitting device 30. For example, if the cycle at which the emitted color is switched in the light emission pattern is about 100 milliseconds, the frame rate of the capture by the camera C is set to about 20 fps. Thus, a moving image is captured by the camera C. Each captured image 40 is a frame image of the moving image.

[0026] FIG. 6 is a diagram showing a captured image 40 captured by a certain camera C at a certain timing. Two subjects M1 and M2 are shown in a captured image 40 shown in Fig. 6. The positioning operation will be described with a focus on subject M1. 6, if it is possible to identify the light-emitting position Q of the light-emitting device 30a attached to the subject M1, the light-emitting position Q is identified. Here, "it is possible to identify the light-emitting position Q of the light-emitting device 30a" means that the light emission of the light-emitting device 30a identified by the light-emitting pattern in the moving image can be detected in the captured image 40, and its position in the image can be identified.

[0027] Specifically, to identify the light-emitting device 30a, the light-emitting pattern of the light-emitting device 30 is detected from a video (a plurality of captured images 40) captured during a predetermined period up to (just before) the capture of the captured image 40 in FIG. 6. The predetermined period is set to be longer than the light-emitting pattern period of the light-emitting device 30. When a light-emitting pattern is detected in the video, this light-emitting pattern is compared with each light-emitting pattern in the light-emitting device management data 133 shown in FIG. 4. When the pattern detected from the video matches any of the light-emitting patterns in the light-emitting device management data 133, the light-emitting device is identified as the light-emitting device 30 (here, light-emitting device 30a) with the light-emitting device ID corresponding to the matched light-emitting pattern. Once the light-emitting device 30a is identified, the light-emitting position Q in the captured image 40 is specified. In Fig. 6, radial lines drawn above the light-emitting devices 30a, 30b indicate that the light emitted by the light-emitting devices 30a, 30b can be detected in the captured image 40, and that the position of the light-emitting device in the image can be specified.

[0028] Similarly, the light-emitting position Q of the light-emitting device 30a is identified in the image 40 captured by the other camera C. When the light-emitting position Q of the light-emitting device 30a is identified in each of the images 40 captured by the multiple cameras C (in this embodiment, the cameras C1 and C2), the position P3 of the light-emitting device 30a in the real space 2, i.e., the position P3 of the subject M1, is derived by the method shown in FIG. For the subject M2, a position P3 in the real space 2 is derived by a similar operation.

[0029] In the following, for a certain light-emitting device 30, a period during which the light-emitting position Q of the light-emitting device 30 can be identified from the captured image 40 is referred to as a "first period." Also, for a certain light-emitting device 30, a period during which the light-emitting position Q of the light-emitting device 30 cannot be identified from the captured image 40 is referred to as a "second period." The second period is different from the first period. The first period and the second period may be periods corresponding to the time axis of a moving image having a plurality of captured images 40 as frame images, and do not necessarily correspond to real time. That is, in a case where the transition of the position P3 of the subject M is analyzed retrospectively based on a moving image captured in the past, a part of the playback period of the moving image may correspond to the first period, and another part may correspond to the second period.

[0030] FIG. 7 is a diagram showing a captured image 40 captured by a certain camera C at a timing different from that in FIG. In the captured image 40 shown in Fig. 7, the subject M1 has moved away from the camera C, so that the light-emitting device 30a of the subject M1 is not captured with a sufficient number of pixels in the captured image 40, and the light-emitting position Q of the light-emitting device 30a cannot be identified. Therefore, the capture timing of the captured image 40 shown in Fig. 7 corresponds to the second period for the light-emitting device 30a. On the other hand, the light emitting position Q of the light emitting device 30b is identifiable in the captured image 40 shown in Fig. 7. Therefore, the capture timing of the captured image 40 shown in Fig. 7 corresponds to the first period for the light emitting device 30b.

[0031] When attaching the light-emitting device 30 to the subject M as in this embodiment, there is a tendency to use a small light-emitting device 30 so as not to impede the movement of the subject M, and therefore, when the subject M moves away from the camera C as shown in Fig. 7, a situation where the light-emitting position Q cannot be identified easily occurs. In addition, when the light-emitting device 30 is directly exposed to sunlight, or when the surroundings of the light-emitting device 30 are too bright, a situation where the light-emitting position Q cannot be identified may also occur. In addition, when there is an obstruction between the light-emitting device 30 and the camera C, or when a foreign object is attached to the light-emitting device 30, and the light of the light-emitting device 30 is temporarily blocked, a situation where the light-emitting position Q cannot be identified may also occur.

[0032] As shown in FIG. 7, when the light-emitting position Q of the light-emitting device 30a in the captured image 40 cannot be identified, the subject M1 included in the captured image 40 (appearing in the captured image 40) is identified based on the external characteristics of the subject M1 to which the light-emitting device 30a is attached. 7 is input to a machine learning model 135 that has been trained in advance to identify the subject M1, and the subject M1 included in the photographed image 40 is identified by the machine learning model 135. The entire photographed image 40 may be input to the machine learning model 135, or a portion of the photographed image 40 located near the light-emitting position Q of the light-emitting device 30a in the last identified photographed image 40 may be extracted and input to the machine learning model 135.

[0033] In order to enable the machine learning model 135 to identify the subject M1, the machine learning model 135 is machine-trained (additional learning) in advance during a first period using teacher data (annotation data) including the image and tag information of the subject M1. As described above, the machine learning model 135 is machine-trained in advance to identify people included in an input image, but this additional learning during the first period further enables the specific subject M1 to be distinguished from other people and identified.

[0034] Specifically, for example, as shown in Fig. 6, in the captured image 40 corresponding to the first period, training data including a portion within a predetermined annotation frame R1 based on the light-emitting position Q of the light-emitting device 30a and tag information corresponding to the light-emitting device 30a is generated, and the training data is input to the machine learning model 135 for machine learning. In the following, any one annotation frame will be referred to as "annotation frame R."

[0035] The tag information is previously associated with the light-emitting device ID of the light-emitting device 30a in the light-emitting device management data 133 shown in Fig. 4. In this example, the light-emitting device ID of the light-emitting device 30a is "M001", and the tag information includes information on the subject M1. The tag information may include information related to the external characteristics of the subject M1 (such as height).

[0036] When the tag information includes information on the subject M, the size of the annotation frame R is set so that the subject M fits within the frame. That is, as shown in FIG. 6, the top side of the rectangular annotation frame R is set at a position passing through the light-emitting position Q of the light-emitting device 30, the distance between the left side and the right side of the annotation frame R is set to be larger than the width of the subject M in the left-right direction, and the distance between the top side and the bottom side of the annotation frame R is set to be larger than the height of the subject M. For example, the annotation frame R is set so that the closer the distance between the position P1 or P2 where the camera C is provided and the latest position P3 of the subject M in the real space 2 derived in the first period is, the larger the annotation frame R is. When the tag information includes information on the height of the subject M, the size of the annotation frame R may be adjusted based on the information. In addition, the captured image 40 may be input to the machine learning model 135 to identify a person, and the annotation frame R may be set to a size that includes the identified person. The size of the annotation frame R described above can be set to the length of the diagonal line of the rectangle formed by the annotation frame R, for example, in an adjustment method in which the annotation frame R has a similar shape before and after size adjustment.

[0037] Alternatively, an annotation frame R may be set on the captured image 40 reduced at a predetermined reduction rate, and training data including a partial image within the annotation frame R may be input to the machine learning model 135. As described below, when identifying the subject M using the machine learning model 135, the light emitting device 30 is far from the camera C, and the subject M shown in the captured image 40 is also small. For this reason, by performing machine learning using a captured image 40 that has been reduced in advance, machine learning can be performed after appropriately ignoring minute features that cannot be seen from a distance among the features related to the appearance of the subject M. This makes it easier to identify the subject M in the captured image 40 that shows the subject M at a distance.

[0038] 6, machine learning for subject M2 is performed in parallel with machine learning for subject M1. The size of each annotation frame R corresponding to each subject M is set separately and independently according to the size of each subject M in the image.

[0039] When the subject M1 is identified by the machine learning model 135 in the captured image 40 shown in Fig. 7, a light-emitting position Q1 of the light-emitting device 30a in the captured image 40 is estimated from the position of the identified subject M1 in the captured image 40 (for example, the center point of the entire subject M1 or the center point of a predetermined part such as the head). In this embodiment, the top of the head of the identified subject M1 is estimated as the light-emitting position Q1 of the light-emitting device 30a. When the light-emitting position Q1 is estimated, a position P3 of the light-emitting device 30a in the real space 2, i.e., the position P3 of the subject M1, is derived by a method similar to that in Fig. 6.

[0040] Thus, in this embodiment, among each of the captured images 40 captured by the cameras C1 and C2, in the captured image 40 corresponding to the first period for the light-emitting device 30 of a certain subject M, the light-emitting position Q of the light-emitting device 30 in the captured image 40 is derived, and in the captured image 40 corresponding to the second period for the light-emitting device 30 of the subject M, the light-emitting position Q of the light-emitting device 30 in the captured image 40 is estimated based on the identification result by the machine learning model 135. Then, based on the light-emitting position Q derived or estimated in each captured image 40, the position P3 of the light-emitting device 30b in the real space 2, i.e., the position P3 of the subject M2, is derived.

[0041] In the above, the subject M included in the photographed image 40 is identified using the machine learning model 135 in the second period, but this is not limited thereto, and the subject M may be identified without using the machine learning model 135. That is, in the first period, features related to the appearance of the subject M may be extracted from a portion of the photographed image 40 within the annotation frame R based on the light emitting position Q of the light emitting device 30, and in the second period, the position of the subject M in the photographed image 40 may be identified based on the extracted features. The feature related to the appearance of the subject M used here may be any feature that can distinguish the subject M from other subjects M and is larger than the light-emitting device 30 (that can be identified even when the light-emitting device 30 becomes unidentifiable). For example, if each subject M is wearing a helmet of a different color, the color of the helmet may be extracted as a feature related to the appearance. Also, if each subject M is wearing a mark (a band attached to the helmet, an armband, etc.) that is different from each other, the mark may be extracted as a feature related to the appearance. Also, the appearance of the subject M itself, for example, the face, may be extracted as a feature. As a method for identifying the subject M in the photographed image 40 based on the features during the second period, various known methods can be used. For example, the subject M may be identified in the photographed image 40 by performing a pattern matching process using the extracted features.

[0042] Next, a positioning process executed by the CPU 11 of the positioning device 10 to realize the above-mentioned operation will be described.

[0043] FIG. 8 is a flowchart showing a control procedure of the positioning process. The positioning process starts when the positioning system 1 is started. When the positioning process is started, the CPU 11 assigns the number of cameras C used in the positioning system 1 to a variable N (step S101). The CPU 11 repeatedly determines whether it is a predetermined positioning timing (step S102), and if it is determined that it is the positioning timing ("YES" in step S102), assigns 1 to a variable n representing the ordinal number of the camera C (step S103). The CPU 11 acquires an image 40 captured by the n-th camera C (step S104), and executes a light emission position identification process (step S105).

[0044] FIG. 9 is a flowchart showing a control procedure of the light emission position specifying process. 9, a process will be described for one target person M. When there are a plurality of targets M to be position-measured, the light emission position identification process for each target person M is executed in parallel or in sequence.

[0045] When the light emission position identification process is called, the CPU 11 determines whether or not the light emission pattern of the light emitting device 30 attached to the subject M has been detected from the moving images captured during a predetermined period up to (just before) the time the captured image 40 was captured (step S201).

[0046] If it is determined that a light emission pattern has been detected ("YES" in step S201), the CPU 11 determines that it is the first period. Then, the CPU 11 identifies the light emission position Q (x, y coordinates) of the light emitting device 30 in the captured image 40, and records it in the positioning history data 134 (step S202).

[0047] Next, the CPU 11 executes steps S203 to S205 for machine learning (additional learning) for the subject M. First, the CPU 11 refers to the light-emitting device management data 133 and acquires tag information corresponding to the light-emitting device ID of the light-emitting pattern identified in step S201 (step S203). Next, the CPU 11 sets the annotation frame R by the above-mentioned method based on the light-emitting position Q of the light-emitting device 30 and the like (step S204). Then, the CPU 11 generates teacher data including a partial image and tag information within the annotation frame R, inputs the teacher data to the machine learning model 135, and performs machine learning to identify the subject M (step S205).

[0048] On the other hand, in step S201, if it is determined that the light emission pattern of the light emitting device 30 attached to the subject M has not been detected ("NO" in step S201), the CPU 11 determines that it is the second period and determines whether machine learning (additional learning) has been performed on the subject M corresponding to the light emitting device 30 (step S206). If it is determined that machine learning is being performed ("YES" in step S206), the CPU 11 inputs the captured image 40 (or a part thereof) and tag information to the machine learning model 135, and causes the subject M corresponding to the light emitting device 30 to be identified and searched for (step S207). If the corresponding subject M is identified ("YES" in step S208), the CPU 11 estimates the light emission position Q (xy coordinates) in the captured image 40 of the light emitting device 30 corresponding to the subject M, and records it in the positioning history data 134 (step S209). When step S205 or S209 is completed, or when step S206 or S208 branches to "NO", the CPU 11 ends the light emission position identification process, and returns the process to the positioning process of FIG.

[0049] 8 (step S105) ends, CPU 11 determines whether variable n matches variable N (step S106), and if it is determined that variable n does not match variable N ("NO" in step S106), increments variable n (step S107) and returns the process to step S104. Through the loop process of steps S104 to S107, light emission position Q of light emitting device 30 in captured image 40 of each camera is identified or estimated.

[0050] If it is determined that the variable n matches the variable N ("YES" in step S106), the CPU 11 derives the position P3 of the light-emitting device 30 in the real space 2, i.e., the position P3 (XYZ coordinates) of the subject M, based on the positions P1, P2 of each camera C in the real space 2 and the light-emitting position Q identified or estimated in each captured image 40 captured by each camera C at the same time (step S108). If there are multiple subjects M to be positioned, the position P3 of each subject M in the real space 2 is derived. The CPU 11 records the derived position P3 in the positioning history data 134.

[0051] When step S108 ends, CPU 11 determines whether or not to continue positioning (step S109). Here, CPU 11 determines to continue positioning if the administrator of positioning system 1 has not instructed to end positioning. If it is determined to continue positioning ("YES" in step S109), CPU 11 returns the process to step S102, and if it is determined not to continue positioning ("NO" in step S109), CPU 11 ends the positioning process.

[0052] <Effects> As described above, the positioning device 10 of the first embodiment is equipped with a CPU 11, which derives a position P3 in the real space 2 of the subject M based on the light-emitting position Q in the captured image 40 during a first period in which the light-emitting position Q of the light-emitting device 30 can be identified in a captured image 40 of the subject M wearing the light-emitting device 30, and derives a position P3 in the real space 2 of the subject M based on the light-emitting position Q in the captured image 40 during a second period different from the first period in which the light-emitting position Q in the captured image 40 cannot be identified, by estimating the light-emitting position Q in the captured image 40 based on characteristics related to the appearance of the subject M in the captured image 40. As a result, even if the light-emitting position Q of the light-emitting device 30 cannot be identified in the captured image 40 due to an insufficient number of pixels occupied by the light-emitting device 30, the light-emitting position Q can be estimated based on the characteristics related to the appearance of the subject M and positioning can be continued. Therefore, the position P3 of the subject M can be derived more appropriately. For example, when a small light-emitting device 30 that can be attached to the subject M is used, the light-emitting position Q is likely to become unidentifiable due to the distance between the camera C and the light-emitting device 30, but with the configuration of this embodiment, even in such a case, the light-emitting position Q can be estimated and positioning can be continued. By making the light-emitting device 30 smaller, the power consumption can be reduced and the continuous operation time of the light-emitting device 30 can be increased. In addition, it is possible to make it easier to attach the light-emitting device 30 to the subject M and to make it less likely that the light-emitting device 30 will cause a malfunction that interferes with the movements of the subject M. Furthermore, the light emitted by the light emitting device 30 is less likely to be blocked and easier to detect than radio waves used in wireless communication, so positioning can be performed more reliably and appropriately than with conventional positioning methods that use radio waves. Moreover, it is possible to achieve both high positioning accuracy and more reliable positioning by directly identifying the light-emitting position Q of the light-emitting device 30 in the first period and estimating the light-emitting position Q based on the characteristics related to the appearance of the subject M only in the second period. For example, in a method of performing positioning using only the characteristics related to the appearance of the subject M, erroneous detection of the subject M is likely to occur due to differences in how the subject M appears from each camera, whereas the method of this embodiment using the light-emitting position Q of the light-emitting device 30 makes such erroneous detection less likely to occur and enables highly accurate positioning.

[0053] In addition, during the first period, the CPU 11 inputs training data including a portion of the captured image 40 within the annotation frame R based on the light-emitting position Q into the machine learning model 135, thereby causing the machine learning model 135 to perform machine learning to identify the subject M included in the captured image 40, and during the second period, causes the machine learning model 135 that has performed the above machine learning to identify the subject M included in the captured image 40, identifies the position of the subject M in the captured image 40 based on the result of the identification, and estimates the light-emitting position Q based on the result of the identification. This allows machine learning to be performed based on the features related to the appearance of the subject M actually photographed in the first period, thereby improving the accuracy of identifying the subject M by the machine learning model 135. As a result, in the second period, the machine learning model 135 can distinguish the learned subject M from other subjects M, that is, identify and identify the individual. For example, in a factory, a plurality of subjects M wearing similar work clothes and helmets can be distinguished from each other and identified. In addition, during learning in the first period, the annotation frame R is determined based on the light-emitting position Q of the light-emitting device 30, so that the light-emitting device 30 can be identified to accurately identify the subject M to be learned and generate teacher data.

[0054] Furthermore, the CPU 11 may extract features relating to the appearance of the subject M within the annotation frame R based on the light-emitting position Q in the photographed image 40 in the first period, identify the position of the subject M in the photographed image 40 based on the extracted features in the second period, and estimate the light-emitting position Q based on the identification result. This makes it possible to identify the position of the subject M in the photographed image 40 in the second period and estimate the light-emitting position Q of the light-emitting device 30 with simpler processing.

[0055] Furthermore, the CPU 11 sets the annotation frame R so that the closer the distance between a predetermined position P1 or P2 where a camera C that captures the captured image 40 is provided and a position P3 of the subject M in the real space 2 derived in the first period is, the larger the annotation frame R becomes. This makes it possible to set the annotation frame R with an appropriate size according to the number of pixels occupied by the subject M in the captured image 40.

[0056] Furthermore, for each of the multiple captured images 40, the CPU 11 specifies the light-emitting position Q when it corresponds to the first period, estimates the light-emitting position Q when it corresponds to the second period, and derives the position P3 of the subject M in the real space 2 based on the multiple light-emitting positions Q derived or estimated in the multiple captured images 40. This makes it possible to perform highly accurate positioning using the multiple captured images 40.

[0057] In addition, in the positioning method according to the present embodiment, the CPU 11 derives the position P3 of the subject M in the real space 2 based on the light-emitting position Q in the photographed image 40 during a first period in which the light-emitting position Q of the light-emitting device 30 can be identified in the photographed image 40 of the subject M to which the light-emitting device 30 is attached, and estimates the light-emitting position Q in the photographed image 40 based on the characteristics related to the appearance of the subject M in the photographed image 40 during a second period different from the first period in which the light-emitting position Q in the photographed image 40 cannot be identified, thereby deriving the position P3 of the subject M in the real space 2. As a result, even if the light-emitting position Q of the light-emitting device 30 cannot be identified in the photographed image 40 due to an insufficient number of pixels occupied by the light-emitting device 30, the light-emitting position Q can be estimated based on the characteristics related to the appearance of the subject M and positioning can be continued. Therefore, the position P3 of the subject M can be more appropriately derived.

[0058] In addition, the program 131 according to the present embodiment causes the CPU 11 to execute a process of deriving the position P3 of the subject M in the real space 2 based on the light-emitting position Q in the photographed image 40 during a first period during which the light-emitting position Q of the light-emitting device 30 can be specified in the photographed image 40 of the subject M to which the light-emitting device 30 is attached, and a process of deriving the position P3 of the subject M in the real space 2 by estimating the light-emitting position Q in the photographed image 40 based on the characteristics related to the appearance of the subject M in the photographed image 40 during a second period different from the first period during which the light-emitting position Q in the photographed image 40 cannot be specified. As a result, even if the light-emitting position Q of the light-emitting device 30 cannot be specified in the photographed image 40 due to a shortage of the number of pixels occupied by the light-emitting device 30, the light-emitting position Q can be estimated based on the characteristics related to the appearance of the subject M and positioning can be continued. Therefore, the position P3 of the subject M can be more appropriately derived.

[0059] Second embodiment Next, a second embodiment will be described. Below, differences from the first embodiment will be described, and descriptions of features common to the first embodiment will be omitted.

[0060] <Positioning system configuration> FIG. 10 is a diagram showing the configuration of a positioning system 1 according to the second embodiment. The positioning system 1 of this embodiment includes a vehicle 20 (moving body) having the above-mentioned positioning device 10 and a camera C, and a plurality of light emitting devices 30. In this embodiment, the vehicle 20 moves in an outdoor space as a real space 2. The vehicle 20 shown in Fig. 10 includes two cameras C that capture images of the front and rear, but is not limited thereto, and the number of cameras C may be one as long as the desired capture range can be captured. A plurality of light emitting devices 30 are installed at predetermined positions in the real space 2. The plurality of light emitting devices 30 are installed in a range in which the vehicle 20 may move in the real space 2, in such an arrangement that two or more light emitting devices 30 are captured by the camera C no matter where the vehicle 20 is located. The installation positions of the light emitting devices 30 are surveyed in advance and recorded in the light emitting device management data 133 (see FIG. 13). The vehicle 20 captures the light emitting devices 30 by the camera C while moving in the real space 2. The positioning device 10 mounted on the vehicle 20 derives the position of the vehicle 20 based on the obtained captured image data 132. More specifically, the positioning device 10 derives the position of the vehicle 20 based on the captured images 40 captured by the camera C so that two or more captured images 40 are captured, and the positions in the real space 2 of the captured images 40. The positioning device 10 may be provided outside the vehicle 20. In this case, data of the image 40 captured by the camera C is transmitted from the vehicle 20 to the positioning device 10. In the present embodiment, an outdoor space is exemplified as the real space 2, but the present invention is not limited to this and may be an indoor space. The real space 2 is not limited to a space on the earth and may be, for example, a space on the moon.

[0061] FIG. 11 is a diagram for explaining the principle of positioning in the positioning system 1 of the second embodiment. 11 illustrates two light-emitting devices 30a and 30b, one camera C, and a captured image 40 obtained by capturing images of the light-emitting devices 30a and 30b by the camera C. The positioning device 10 derives a position P4 (X1, Y1, Z1) of the vehicle 20 (camera C) in the real space 2 by the principle of triangulation based on a light-emitting position Q3 (x1, y1) of the light-emitting device 30a and a light-emitting position Q4 (x2, y2) of the light-emitting device 30b in the captured image 40, and a position P5 (Xa, Ya, Za) of the light-emitting device 30a and a position P6 (Xb, Yb, Zb) of the light-emitting device 30b in the real space 2. However, it is assumed here that the height (Z1) of the camera C is constant. The positioning device 10 can also derive the orientation (horizontal direction) of the vehicle 20 (camera C) from the position P4 of the vehicle 20 in the real space 2 and the light emission positions Q3 and Q4 of the light emitting devices 30a and 30b in the captured image 40. With this method, the position P4 and orientation of the vehicle 20 can be derived on the premise that the height and two degrees of freedom (pitch and roll) of the attitude angle of the vehicle 20 (camera C) are fixed. Note that, in cases where the attitude angle of the vehicle 20 may vary, such as when the road surface is uneven, the position P4 and orientation may be derived that are corrected based on the detection result of the attitude angle by the attitude angle detection unit 22 (see FIG. 12). For example, the position P4 may be derived after correcting the orientation of the captured image 40 (the light emission positions of the light emitting devices 30a and 30b in the captured image 40) based on the detection result of the attitude angle by the attitude angle detection unit 22.

[0062] Moreover, by capturing images of three or more light-emitting devices 30 and using them to derive the position, it is possible to perform positioning corresponding to more degrees of freedom. In other words, when two light-emitting devices 30 are used, positioning with three degrees of freedom, namely, horizontal position (XY coordinates) and horizontal orientation, is possible, whereas using three or more position indicators makes it possible to perform positioning with four or more degrees of freedom, including at least a part of the two degrees of freedom, namely, vertical coordinate (Z coordinate) and attitude angle (pitch, roll). Ideally, using six light-emitting devices 30 makes it possible to perform positioning with six degrees of freedom, namely, three-dimensional coordinates (horizontal position and height) and three-axis attitude angle (yaw (horizontal orientation), pitch, roll). In addition, FIG. 11 illustrates an example in which two light-emitting devices 30 are captured by one camera C, but if a total of two or more light-emitting devices 30 are captured in multiple captured images 40 captured by multiple cameras C, positioning can be performed in a similar manner to that described above by using the orientation information of each camera C.

[0063] FIG. 12 is a block diagram showing the functional configuration of the vehicle 20. The vehicle 20 includes a positioning device 10, a camera C, a driving unit 21, and an attitude angle detection unit 22. The various units of the vehicle 20 are connected to each other via a communication path 23 such as a bus. The functional configurations of the positioning device 10 and the camera C are similar to those of the first embodiment. However, the contents of the light-emitting device management data 133 and the positioning history data 134 stored in the storage unit 13 are partially different from those of the first embodiment.

[0064] FIG. 13 is a diagram showing the contents of the light emitting device management data 133 in the second embodiment. Each data row of the light-emitting device management data 133 includes the data items of "light-emitting device ID", "light-emitting pattern", "tag information" and "real space position of light-emitting device". The contents of the "light emitting device ID" and "light emitting pattern" are the same as those in the first embodiment. In this embodiment, information that represents the light emitting device 30 in that data row is recorded as the "tag information." The "real space position of the light emitting device" is the position (P5, P6, etc.) in the real space 2 of the light emitting device 30 in that data row, which has been measured in advance.

[0065] FIG. 14 is a diagram showing the contents of the positioning history data 134 in the second embodiment. In the present embodiment, the "in-image light emitting position Q" in the positioning history data 134 represents the light emitting position Q (xy coordinates) in the captured image 40 of each light emitting device 30 appearing in the captured image 40. The "positioning result" indicates the position P4 (XYZ coordinates) of the vehicle 20 in the real space 2, which is derived based on the "light emitting position in the image".

[0066] The travel drive unit 21 shown in FIG. 12 drives (rotates) the wheels for traveling the vehicle 20, changes the direction of travel, and so on, in accordance with a control signal transmitted from the CPU 11 or a control processor (not shown).

[0067] The attitude angle detection unit 22 is, for example, a six-axis sensor including a three-axis acceleration sensor that detects acceleration in three-axis directions and a three-axis angular velocity sensor (gyro sensor) that detects angular velocity around three axes. The attitude angle detection unit 22 derives the attitude angles (yaw, pitch, roll) of the vehicle 20 based on the output values ​​of the three-axis acceleration sensor and the three-axis angular velocity sensor, and outputs the deriving angles to the positioning device 10.

[0068] <Positioning system operation> Next, the operation relating to positioning of the positioning system 1 in the second embodiment will be described. In this embodiment, when the vehicle 20 starts to operate, the camera C starts to capture the captured image 40 at the same frame rate as in the first embodiment.

[0069] FIG. 15 is a diagram showing a captured image 40 captured by a camera C at a certain timing. Two light emitting devices 30a and 30b are captured in a captured image 40 shown in Fig. 15. In the positioning operation, a light emitting position Q in the captured image 40 is derived or estimated for each of the light emitting devices 30a and 30b by the same method as in the first embodiment.

[0070] In detail, when it is possible to identify the light-emitting device 30 by its light-emitting pattern and to specify the light-emitting position Q in the photographed image 40, the light-emitting position Q is specified. In Fig. 15, a sawtooth figure drawn radially around the circle of the light-emitting device 30 indicates that the light-emitting position Q of the light-emitting device 30 can be specified in the photographed image 40. In the example shown in Fig. 15, the light-emitting positions Q1 and Q2 of both the light-emitting devices 30a and 30b can be specified in the photographed image 40, and therefore the light-emitting positions Q1 and Q2 are derived from the photographed image 40.

[0071] FIG. 16 is a diagram showing a captured image 40 captured by a camera C at a timing different from that in FIG. In the captured image 40 shown in Fig. 16, the vehicle 20 has moved away from the light-emitting device 30b, so that the light-emitting device 30b is not captured with a sufficient number of pixels in the captured image 40, and the light-emitting position Q2 of the light-emitting device 30b cannot be identified. Therefore, the capture timing of the captured image 40 shown in Fig. 16 corresponds to the second period for the light-emitting device 30b. On the other hand, the light emitting position Q1 of the light emitting device 30a is identifiable in the captured image 40 shown in Fig. 16. Therefore, the capture timing of the captured image 40 shown in Fig. 16 corresponds to the first period for the light emitting device 30a.

[0072] As shown in Fig. 16, when the light-emitting position Q2 of the light-emitting device 30b in the photographed image 40 cannot be identified, the place where the light-emitting device 30b is provided (the installation place or mounting place; hereinafter, referred to as the "installation place") is identified based on the characteristics related to the appearance of the place. The appearance of the installation place includes information such as the pattern and color of the object on which the light-emitting device 30 is installed or the object to which it is attached (the ground in this embodiment). To make it easier to distinguish the appearance of the installation place for each light-emitting device 30, the installation place of each light-emitting device 30 may be given a different pattern or color in advance.

[0073] 16 is input to a machine learning model 135 that has been trained in advance to identify an installation location, and the machine learning model 135 identifies the installation location included in the photographed image 40. The entire photographed image 40 may be input to the machine learning model 135, or a portion of the photographed image 40 located near the light-emitting position Q2 of the light-emitting device 30b in the last identified photographed image 40 may be extracted and input to the machine learning model 135.

[0074] In order to enable the machine learning model 135 to identify the installation location, the machine learning model 135 is trained (additional learning) in advance during a first period using teacher data including an image of the installation location of the light emitting device 30 and tag information. Specifically, teacher data including a portion within a predetermined annotation frame R2 based on the light emitting position Q2 of the light emitting device 30b and tag information corresponding to the light emitting device 30b in the captured image 40 corresponding to the first period as shown in FIG. 15 is generated, and the teacher data is input to the machine learning model 135 to perform machine learning.

[0075] When the tag information includes information on the light-emitting device 30, the size of the annotation frame R is set so that the installation location around the light-emitting device 30 is included within the frame. For example, as shown in Fig. 15, the annotation frame R is set so that the light-emitting unit of the light-emitting device 30 is positioned at the center of the rectangular annotation frame R. For example, the annotation frame R is set so that the closer the distance between the position where the light-emitting device 30 is provided and the latest position P4 of the vehicle 20 in the real space 2 derived in the first period is, the larger the annotation frame R is.

[0076] 15, in addition to machine learning of the installation location of the light-emitting device 30b, machine learning of the installation location of the light-emitting device 30a is also performed in parallel. The size of each annotation frame R corresponding to each light-emitting device 30 is set separately and independently depending on the size of each light-emitting device 30 in the image, etc.

[0077] When the installation location of the light-emitting device 30b is identified by the machine learning model 135 in the captured image 40 shown in FIG. 16, the light-emitting position Q2 of the light-emitting device 30b in the captured image 40 is estimated from the position of the identified installation location in the captured image 40 (e.g., the overall center point of the installation location).

[0078] As described above, in this embodiment, for each of the multiple light-emitting devices 30 captured in the captured image 40, the light-emitting position Q of the light-emitting device 30 in the captured image 40 is derived for the light-emitting device 30 corresponding to the first period, and for the light-emitting device 30 corresponding to the second period, the light-emitting position Q of the light-emitting device 30 in the captured image 40 is estimated based on the identification result by the machine learning model 135. Then, based on the derived or estimated light-emitting positions Q of the multiple light-emitting devices 30, the position P4 of the vehicle 20 in the real space 2 is derived.

[0079] In the above, the installation location of the light-emitting device 30 is identified using the machine learning model 135 in the second period, but the present invention is not limited to this, and the installation location of the light-emitting device 30 may be identified without using the machine learning model 135. That is, in the first period, features related to the appearance of the installation location of the light-emitting device 30 may be extracted from a portion within the annotation frame R based on the light-emitting position Q of the light-emitting device 30 in the photographed image 40, and in the second period, the position of the installation location of the light-emitting device 30 in the photographed image 40 may be identified based on the extracted features. The features related to the appearance of the installation location used here may be any features that can be distinguished from other installation locations.

[0080] Next, a positioning process executed by the CPU 11 of the positioning device 10 to realize the above-mentioned operation will be described.

[0081] FIG. 17 is a flowchart showing a control procedure of the positioning process in the second embodiment. When the positioning process is started, the CPU 11 repeatedly determines whether or not it is a predetermined positioning timing (step S301). If it is determined that it is the positioning timing ("YES" in step S301), the CPU 11 acquires the image 40 captured by the camera C (step S302) and executes the light emission position identification process (step S303).

[0082] FIG. 18 is a flowchart showing a control procedure of the light emission position specifying process in the second embodiment. When the light emission position specifying process is called, the CPU 11 determines whether or not the light emission pattern of the light emitting device 30 has been detected from the moving images captured during a predetermined period up to (just before) the capture of the captured image 40 (step S401).

[0083] If it is determined that at least one light emission pattern has been detected ("YES" in step S401), the CPU 11 determines that it is the first period for each light emission device 30 corresponding to the detected light emission pattern. The CPU 11 also selects one light emission device 30 corresponding to one of the detected light emission patterns (step S402). The CPU 11 then identifies the light emission position Q (x, y coordinates) of the light emission device 30 in the captured image 40, and records it in the positioning history data 134 (step S403).

[0084] Next, the CPU 11 executes steps S404 to S406 for machine learning (additional learning) about the installation location of the light-emitting device 30. The processing of steps S404 to S406 is similar to the processing of steps S203 to S205 in FIG. 9 except that the learning target is the installation location instead of the subject M, and therefore the description will be omitted. When step S406 ends, the CPU 11 determines whether or not there is an unselected light-emitting device 30 among the light-emitting devices 30 whose light-emitting patterns have been detected (step S407). If it is determined that there is an unselected light-emitting device 30 ("YES" in step S407), the CPU 11 returns the processing to step S402 and performs identification of the light-emitting position Q and machine learning for the next light-emitting device 30.

[0085] If it is determined that there is no unselected light-emitting device 30 ("NO" in step S407), or if it is determined in step S401 that the light emission pattern of the light-emitting device 30 has not been detected ("NO" in step S401), the CPU 11 determines whether or not there is a lost light-emitting device 30 (step S408). Here, the lost light-emitting device 30 is a light-emitting device 30 whose light emission pattern has been detected at any past positioning timing, but whose light-emitting pattern has not been detected in the light-emitting position identification process being executed. Note that it may be determined whether or not there is a lost light-emitting device 30 among only the light-emitting devices 30 whose light-emitting patterns have been detected within the most recent predetermined period.

[0086] If it is determined that there is a lost light-emitting device 30 ("YES" in step S408), the CPU 11 determines that the lost light-emitting device 30 is in the second period. The CPU 11 also selects one lost light-emitting device 30 (step S409). The CPU 11 inputs the captured image 40 (or a part thereof) and tag information to the machine learning model 135, and causes the model 135 to identify and search for the installation location of the lost light-emitting device 30 (step S410). If the corresponding installation location is identified ("YES" in step S411), the CPU 11 estimates the light-emitting position Q (xy coordinates) in the captured image 40 of the light-emitting device 30 corresponding to the identified installation location, and records it in the positioning history data 134 (step S412). When step S412 is completed or when the process branches to "NO" in step S411, the CPU 11 determines whether or not there is an unselected lost light-emitting device 30 (step S413). When it is determined that there is an unselected lost light-emitting device 30 ("YES" in step S413), the CPU 11 returns the process to step S409 and estimates the light-emitting position Q of the next lost light-emitting device 30. If it is determined that there is no unselected lost light-emitting device 30 ("NO" in step S413), or if it is determined that there is no lost light-emitting device 30 in step S408 ("NO" in step S408), the CPU 11 terminates the light-emitting position identification process and returns the process to the positioning process of Figure 8.

[0087] 17 is completed (step S303), the CPU 11 refers to the light-emitting device management data 133 and acquires the positions in the real space 2 of each light-emitting device 30 for which the light-emitting position Q has been specified or estimated (for example, positions P5 and P6 in FIG. 11) (step S304). Then, the CPU 11 derives the position P4 of the camera C in the real space 2, i.e., the position P4 (XYZ coordinates) of the vehicle 20, based on the light-emitting position Q of each light-emitting device 30 specified or estimated in the captured image 40 and the positions P5 and P6 of each light-emitting device 30 in the real space 2 (step S305). The CPU 11 also records the derived position P4 in the positioning history data 134.

[0088] When step S305 ends, the CPU 11 determines whether or not to continue positioning (step S306). If it is determined that positioning is to be continued ("YES" in step S306), the CPU 11 returns the process to step S301, and if it is determined that positioning is not to be continued ("NO" in step S306), the CPU 11 ends the positioning process.

[0089] <Effects> As described above, the positioning device 10 of the second embodiment is equipped with a CPU 11, which derives a position P4 in the real space 2 of the vehicle 20 based on the light-emitting position Q of the light-emitting device 30 in the captured image 40 during a first period in which the light-emitting position Q of the light-emitting device 30 can be identified in the captured image 40 in which the light-emitting device 30 is photographed by the camera C attached to the vehicle 20, and derives the position P4 in the real space 2 of the vehicle 20 by estimating the light-emitting position Q of the light-emitting device 30 in the captured image 40 based on characteristics related to the appearance of the installation location where the light-emitting device 30 is installed in the captured image 40 during a second period different from the first period in which the light-emitting position Q in the captured image 40 cannot be identified. As a result, even if the light-emitting position Q of the light-emitting device 30 cannot be identified in the captured image 40 due to an insufficient number of pixels occupied by the light-emitting device 30, the light-emitting position Q can be estimated based on the characteristics related to the appearance of the subject M and positioning can be continued. Therefore, the position P4 of the vehicle 20 can be derived more appropriately.

[0090] Moreover, in the first period, the CPU 11 inputs teacher data including a portion of the annotation frame R based on the light-emitting position Q in the captured image 40 to the machine learning model 135, thereby causing the machine learning model 135 to perform machine learning to identify the installation location included in the captured image 40, and in the second period, the machine learning model 135 that has performed the above machine learning causes the machine learning model 135 to identify the installation location included in the captured image 40, identifies the position of the installation location in the captured image 40 based on the result of the identification, and estimates the light-emitting position Q based on the result of the identification. This allows machine learning to be performed based on features related to the appearance of the installation location that is actually photographed in the first period, thereby improving the identification accuracy of the installation location by the machine learning model 135. Furthermore, during learning in the first period, the annotation frame R is determined based on the light-emitting position Q of the light-emitting device 30, so that the installation location of the learning target can be accurately identified by identifying the light-emitting device 30 to generate teacher data.

[0091] Furthermore, the CPU 11 may extract features relating to the appearance of the installation location within the annotation frame R based on the light-emitting position Q in the captured image 40 during the first period, identify the position of the installation location within the captured image 40 based on the extracted features during the second period, and estimate the light-emitting position Q based on the identification result. This allows the position of the installation location within the captured image 40 during the second period to be identified and the light-emitting position Q of the light-emitting device 30 to be estimated by simpler processing.

[0092] Furthermore, the CPU 11 sets the annotation frame R so that the closer the distance between the predetermined position where the light-emitting device 30 is provided and the position P4 of the vehicle 20 in the real space 2 derived in the first period is, the larger the annotation frame R becomes. This makes it possible to set the annotation frame R with an appropriate size according to the number of pixels that the installation location occupies in the captured image 40.

[0093] Furthermore, when a plurality of light-emitting devices 30 are included in the photographed image 40, the CPU 11 distinguishes the plurality of light-emitting devices 30 based on the light-emitting patterns of the plurality of light-emitting devices 30, and when a plurality of light-emitting devices 30 are included in the photographed image 40, the CPU 11 distinguishes the plurality of light-emitting devices 30 based on the light-emitting patterns of the plurality of light-emitting devices 30, and distinguishes whether each of the plurality of light-emitting devices 30 corresponds to a first period or a second period, and for the light-emitting device 30 corresponding to the first period, the CPU 11 specifies the light-emitting position Q of the light-emitting device 30, and for the light-emitting device 30 corresponding to the second period, the CPU 11 estimates the light-emitting position Q of the light-emitting device 30 in the photographed image 40 from the position in the photographed image 40 of the installation location identified based on the characteristics related to the appearance of the installation location. This makes it possible to perform highly accurate positioning based on the light-emitting positions Q of the plurality of light-emitting devices 30.

[0094] In addition, in the positioning method according to the present embodiment, the CPU 11 derives the position P4 of the vehicle 20 in the real space 2 based on the light-emitting position Q of the light-emitting device 30 in the captured image 40 during a first period in which the light-emitting position Q of the light-emitting device 30 can be identified in the captured image 40 captured by the camera C attached to the vehicle 20, and estimates the light-emitting position Q of the light-emitting device 30 in the captured image 40 based on the characteristics related to the appearance of the installation location where the light-emitting device 30 is installed in the captured image 40 during a second period different from the first period in which the light-emitting position Q cannot be identified in the captured image 40, thereby deriving the position P4 of the vehicle 20 in the real space 2. As a result, even if the light-emitting position Q of the light-emitting device 30 cannot be identified in the captured image 40 due to a lack of pixels occupied by the light-emitting device 30, the light-emitting position Q can be estimated based on the characteristics related to the appearance of the subject M and positioning can be continued. Therefore, the position P4 of the vehicle 20 can be more appropriately derived.

[0095] In addition, the program 131 according to the present embodiment causes the CPU 11 to execute a process of deriving the position P4 in the real space 2 of the vehicle 20 based on the light-emitting position Q of the light-emitting device 30 in the captured image 40 during a first period during which the light-emitting position Q of the light-emitting device 30 can be specified in the captured image 40 captured by the camera C attached to the vehicle 20, and a process of deriving the position P4 in the real space 2 of the vehicle 20 by estimating the light-emitting position Q of the light-emitting device 30 in the captured image 40 based on the characteristics related to the appearance of the installation location where the light-emitting device 30 is installed in the captured image 40 during a second period different from the first period during which the light-emitting position Q cannot be specified in the captured image 40. As a result, even if the light-emitting position Q of the light-emitting device 30 cannot be specified in the captured image 40 due to a shortage of the number of pixels occupied by the light-emitting device 30, the light-emitting position Q can be estimated based on the characteristics related to the appearance of the subject M and positioning can be continued. Therefore, the position P4 of the vehicle 20 can be more appropriately derived.

[0096] (others) The present invention is not limited to the above-described embodiment, but may be modified in various ways. For example, in the first embodiment, a target person M moving in a factory is exemplified as a moving object to be positioned, and in the second embodiment, a vehicle 20 moving outdoors is exemplified as a moving object to be positioned, but the present invention is not limited to these. The positioning system 1 may be one that targets any moving object moving in the real space 2 as a moving object to be positioned. For example, the moving object may be a forklift moving in a warehouse.

[0097] In addition, the light emission pattern of the light emitting device 30 is exemplified as one in which the light emission colors of three colors, red, green, and blue, are changed, but the present invention is not limited to this. For example, the color used may be changed, or the number of colors used may be two or less, or four or more. The light emission pattern may be configured by a combination of a light emission period and a non-light emission period of a single light emission color. The light emission colors of the multiple light emitting devices 30 may be made different from each other, and the light emission in the light emission color specific to each light emitting device 30 may be used as the light emission pattern. Li-Fi (Light Fidelity) may be used, which transmits information by light modulated at high speed. Optical communication using infrared or ultraviolet light instead of visible light may also be used.

[0098] In the above description, an example has been disclosed in which the HDD or SSD of the storage unit 13 is used as a computer-readable medium for the program according to the present invention, but the present invention is not limited to this example. As other computer-readable media, information recording media such as flash memory and CD-ROM can be applied. In addition, a carrier wave is also applied to the present invention as a medium for providing data of the program according to the present invention via a communication line.

[0099] Furthermore, it goes without saying that the detailed configuration and detailed operation of each component of the positioning system 1 in the above embodiment can be modified as appropriate without departing from the spirit of the present invention.

[0100] Although the embodiments of the present invention have been described, the scope of the present invention is not limited to the above-described embodiments, but includes the scope of the invention described in the claims and its equivalents. [Explanation of symbols]

[0101] 1 Positioning System 2 Real Space 10 Positioning device 11 CPU (processing unit) 13 Storage section 131 Programs 135 Machine Learning Models 14 Communications Department 20 Vehicles (moving objects) 30, 30a, 30b Light-emitting device 40, 40a, 40b Photographed images (images) C, C1, C2 cameras (photography equipment) M, M1, M2 Target person (mobile) P1, P2 camera positions P3 Position of the subject (moving object) P4 Vehicle (moving object) position P5, P6 Light emitting device position Q, Q1~Q4 Light emitting position R, R1, R2 annotation frame (predetermined frame)

Claims

1. During a first period during which a light-emitting position of the light-emitting device can be identified in an image captured of a moving object having a light-emitting device attached thereto, a position of the moving object in a real space is derived based on the light-emitting position in the image; deriving a position of the moving object in real space by estimating the light-emitting position in the image based on features related to an appearance of the moving object in the image during a second period different from the first period during which the light-emitting position in the image cannot be identified; A positioning device comprising a processing unit.

2. The processing unit includes: During the first period, training data including a portion of the image within a predetermined frame based on the light emission position is input to a machine learning model, thereby causing the machine learning model to perform machine learning so as to identify the moving object included in the image; During the second period, the machine learning model is used to identify the moving object included in the image, the position of the moving object within the image is identified based on a result of the identification, and the light emission position is estimated based on the result of the identification. The positioning device according to claim 1 .

3. The processing unit includes: extracting the feature related to the appearance of the moving object within a predetermined frame based on the light emission position in the image during the first period; In the second period, a position of the moving object in the image is identified based on the extracted features, and the light emission position is estimated based on the identification result. The positioning device according to claim 1 .

4. The processing unit includes: setting the predetermined frame such that the closer the distance between a predetermined position where an image capturing device that captures the image is provided and the position of the moving object in real space derived in the first period is, the larger the predetermined frame becomes; The positioning device according to claim 2 or 3.

5. The processing unit identifies the light-emitting position for each of the plurality of images when the image corresponds to the first period, estimates the light-emitting position when the image corresponds to the second period, and derives a position of the moving object in real space based on the light-emitting positions derived or estimated in the plurality of images. The positioning device according to any one of claims 1 to 3.

6. deriving a position of the moving body in real space based on the light-emitting position of the light-emitting device in an image captured by an imaging device attached to the moving body during a first period during which a light-emitting position of the light-emitting device in the image can be identified; deriving a position of the moving object in real space by estimating a light-emitting position of the light-emitting device in the image based on features related to an appearance of a place in the image where the light-emitting device is provided, during a second period different from the first period during which the light-emitting position in the image cannot be identified; A positioning device comprising a processing unit.

7. The processing unit includes: During the first period, training data including a portion of the image within a predetermined frame based on the light-emitting position is input to a machine learning model, thereby causing the machine learning model to perform machine learning so as to identify the location included in the image; During the second period, the place included in the image is identified by the machine learning model that has been subjected to the machine learning, a position of the place in the image is specified based on a result of the identification, and the light emission position is estimated based on the result of the specification. The positioning device according to claim 6.

8. The processing unit includes: extracting the feature related to the appearance of the location within a predetermined frame based on the light-emitting position in the image during the first period; During the second period, a position of the place in the image is identified based on the extracted features, and the light emission position is estimated based on the identification result. The positioning device according to claim 6.

9. The processing unit includes: setting the predetermined frame such that the closer the distance between the predetermined position where the light emitting device is provided and the position of the moving object in real space derived in the first period is, the larger the predetermined frame becomes; 9. The positioning device according to claim 7 or 8.

10. The processing unit includes: When a plurality of the light emitting devices are included in the image, distinguishing the plurality of light emitting devices based on a light emission pattern of each of the plurality of light emitting devices; determining whether each of the plurality of light emitting devices corresponds to the first period or the second period; For a light-emitting device corresponding to the first period, a light-emitting position of the light-emitting device is identified; For a light-emitting device corresponding to the second period, an emission position of the light-emitting device in the image is estimated from a position in the image of the place identified based on the feature related to the appearance of the place. The positioning device according to any one of claims 6 to 8.

11. 1. A computer implemented positioning method, comprising: During a first period during which a light-emitting position of the light-emitting device can be identified in an image captured of a moving object having a light-emitting device attached thereto, a position of the moving object in a real space is derived based on the light-emitting position in the image; deriving a position of the moving object in real space by estimating the light-emitting position in the image based on features related to an appearance of the moving object in the image during a second period different from the first period during which the light-emitting position in the image cannot be identified; Positioning method.

12. 1. A computer implemented positioning method, comprising: deriving a position of the moving body in real space based on the light-emitting position of the light-emitting device in an image captured by an imaging device attached to the moving body during a first period during which a light-emitting position of the light-emitting device in the image can be identified; deriving a position of the moving object in real space by estimating a light-emitting position of the light-emitting device in the image based on features related to an appearance of a place in the image where the light-emitting device is provided, during a second period different from the first period during which the light-emitting position in the image cannot be identified; Positioning method.

13. On the computer, a process of deriving a position of a moving object in a real space based on a light-emitting position of the light-emitting device in an image captured during a first period during which a light-emitting position of the light-emitting device can be identified in the image; a process of deriving a position of the moving object in a real space by estimating the light-emitting position in the image based on a feature related to an appearance of the moving object in the image during a second period different from the first period during which the light-emitting position in the image cannot be identified; A program that executes the following.

14. On the computer, a process of deriving a position of the moving body in a real space based on a light-emitting position of the light-emitting device in an image captured by an imaging device attached to the moving body during a first period during which a light-emitting position of the light-emitting device in the image can be identified; a process of deriving a position of the moving object in a real space by estimating a light-emitting position of the light-emitting device in the image based on features related to an appearance of a place in the image where the light-emitting device is provided, during a second period different from the first period during which the light-emitting position in the image cannot be identified; A program that executes the following.

Images