Lighting control device, program, and lighting control system

Abstract

To provide a lighting control device capable of tracking an illumination target that moves arbitrarily and illuminating a desired irradiation target region.SOLUTION: A lighting control device (3) includes: image input means (31) for inputting illuminometer pattern images for each reference point; illuminometer position and posture calculation means (32) for calculating the position and the posture of an illuminometer on the basis of the positional relationship between a pattern and the illuminometer in the illuminometer pattern image input by the image input means (31); and lighting position and posture calculation means (33) for calculating, by triangulation, the position and the posture of a robot spotlight (2) on the basis of the position and posture of the illuminometer when the illuminometer pattern is placed at each reference point and an irradiation direction of the robot spotlight (2).SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a lighting control device, a program therefor, and a lighting control system. [Background technology]

[0002] When lighting a stage, studio, or other space, if the lighting target needs to move, it is necessary to mark the predetermined destination on the floor, move the lighting target to the marked position, and then aim the light at the lighting target in advance.

[0003] A method has been proposed for remotely controlling the illumination direction (pan angle, tilt angle) of lighting (for example, Patent Document 1). In the method described in Patent Document 1, in order for the lighting to follow a moving illumination target, it is necessary to program the illumination direction in advance so that the illumination target moves according to a script and illuminates the illumination target. Also, a method has been proposed in which the direction of illumination is controlled so as to follow a beacon attached to an illumination target (for example, Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 3-29296 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-208222 Summary of the Invention [Problem to be solved by the invention]

[0005] The method described in Patent Document 1 has a problem in that the direction of illumination is determined in advance, making it impossible to freely move the illumination target. Also, the method described in Patent Document 2 concentrates illumination at the position where the beacon is attached, so illumination may stray from the desired illumination target part (for example, a person's face).

[0006] Therefore, an object of the present invention is to provide a lighting control device, a program therefor, and a lighting control system that can follow an arbitrarily moving lighting target and illuminate a desired illumination target area. [Means for solving the problem]

[0007] In order to solve the above problems, the lighting control device of the present invention is a lighting control device that calibrates lighting to control the direction of lighting irradiation using illuminance meter pattern images captured by multiple cameras, the illuminance meter pattern consisting of an illuminance meter that measures illuminance and multiple patterns arranged around the illuminance meter, and is configured to include an image input means, an illuminance meter position and orientation calculation means, and an lighting position and orientation calculation means.

[0008] The image input means inputs an illuminance meter pattern image for each reference point when the illumination is fixed while illuminating the moving range of the illumination object and the illuminance meter pattern is placed at the reference point where the illuminance is maximum. The illuminance meter position and orientation calculation means calculates the position and orientation of the illuminance meter for each reference point from the positional relationship between the illuminance meter and the pattern in the illuminance meter pattern image input by the image input means. The illumination position and orientation calculation means calculates the position and orientation of the illumination by triangulation from the position and orientation of the illumination meter when the illumination meter pattern is placed at each reference point and the illumination direction.

[0009] With this configuration, the lighting control device can control the lighting direction using the calculated lighting position, eliminating the need to program the lighting direction in advance and allowing the lighting to follow an arbitrarily moving lighting target. Furthermore, with the lighting control device, since the lighting target does not need to be equipped with a beacon, lighting will not be concentrated on the beacon and the desired lighting target can be illuminated.

[0010] The present invention can also be realized in a lighting control system comprising the above-mentioned lighting control device, an illuminance meter that measures illuminance, and an illuminance meter pattern consisting of a plurality of patterns arranged around the illuminance meter.

[0011] The present invention can also be realized by a program for causing a computer to function as the lighting control device described above. [Effects of the Invention]

[0012] According to the present invention, it is possible to follow an illumination target that moves arbitrarily, and illuminate a desired illumination target portion. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a schematic diagram illustrating a lighting control system according to an embodiment when calibrating a robot spotlight. [Figure 2] FIG. 1 is a schematic diagram illustrating a lighting control system according to an embodiment controlling a robot spotlight. [Figure 3] FIG. 2A is an explanatory diagram illustrating an illuminance meter pattern according to an embodiment, and FIG. 2B is an explanatory diagram illustrating an illuminance meter. [Figure 4] 1 is a block diagram showing a configuration of a lighting control device according to an embodiment; [Figure 5] 10 is a flowchart showing an operation during calibration of the robot spotlight in the lighting control system according to the embodiment. [Figure 6] FIG. 2 is an explanatory diagram illustrating the positional relationship between an illuminance meter and a pattern in the embodiment. [Figure 7] FIG. 4 is an explanatory diagram for explaining the photographing of an illuminance meter pattern in the embodiment. [Figure 8] In the embodiment, (a) is an explanatory diagram illustrating the positional relationship between a robot spotlight and an illuminance meter pattern, and (b) is an explanatory diagram illustrating a method for calculating the position of the illuminance meter. [Figure 9] FIG. 4 is an explanatory diagram illustrating a method for calculating the position of an illuminance meter in the embodiment. [Figure 10] 10 is a flowchart showing an operation when controlling a robot spotlight in the lighting control system according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, each embodiment described below is intended to embody the technical concept of the present invention, and unless otherwise specified, the present invention is not limited to the following. Furthermore, the same means will be given the same reference numerals, and their description may be omitted.

[0015] (Embodiment) [Lighting control system overview] An overview of a lighting control system 100 according to an embodiment will be described with reference to FIGS. 1 and 2. FIG. The lighting control system 100 controls a robot spotlight (lighting) 2. First, the lighting control system 100 calibrates the robot spotlight 2 using an illuminance meter pattern 4 (FIG. 1). By calibrating the robot spotlight 2, the position and posture of the robot spotlight 2 become known. Next, the lighting control system 100 controls the irradiation direction of the robot spotlight 2 so that it follows the lighting target 9 (FIG. 2).

[0016] The lighting target 9 is the target illuminated by the robot spotlight 2, and may be a person or an object, for example. The lighting target 9 can be moved freely within the illumination range of the robot spotlight 2 (for example, the floor of a film studio). Furthermore, a desired portion of the lighting target 9 to be illuminated can be set in advance. This desired portion of the lighting target 9 is a portion that is desired to be illuminated by the robot spotlight 2 (for example, an actor's face or hands).

[0017] 1, the lighting control system 100 includes multiple cameras 1, a robot spotlight 2, a lighting control device 3, and an illuminance meter pattern 4. In this embodiment, the lighting control system 10 is described as including two cameras 1 and one robot spotlight 2, but the number of cameras 1 and robot spotlights 2 is not limited to this.

[0018] The cameras 1 are fixed cameras that capture images of the illuminance meter pattern 4 and the lighting target 9. Specifically, each camera 1 is fixed to a different location in the photography studio (for example, the top of the left or right side wall) and captures the range of movement of the lighting target 9. Each camera 1 also outputs an image of the illuminance meter pattern 4 or the lighting target 9 and the camera parameters at that time to the lighting control device 3. Hereinafter, an image of the illuminance meter pattern 4 may be referred to as an "illuminance meter pattern image," and an image of the lighting target 9 may be referred to as a "captured image."

[0019] The robot spotlight 2 is a light that illuminates an illumination target 9, and is, for example, a spotlight fixed to the ceiling of a photography studio. The robot spotlight 2 arbitrarily controls the illumination direction (pan angle, tilt angle) according to commands from the lighting control device 3, and also feeds back the irradiation direction to the lighting control device 3. In this embodiment, the placement position of the robot spotlight 2 is fixed, and only the irradiation direction changes.

[0020] The lighting control device 3 calibrates the robot spotlight 2 to control the irradiation direction of the robot spotlight 2, using illuminance meter pattern images obtained by photographing an illuminance meter pattern 4 with multiple cameras 1. The lighting control device 3 controls the irradiation direction of the robot spotlight 2 by referring to the measured position of the illumination target 9 and the position and posture of the robot spotlight 2 determined by calibration. The configuration of the lighting control device 3 will be described later.

[0021] The illuminance meter pattern 4 is used when calibrating the robot spotlight 2. As shown in FIG. 3(a), the illuminance meter pattern 4 consists of an illuminance meter 40 that measures illuminance and multiple patterns 41 arranged around the illuminance meter 40. Also, as shown in FIG. 3(b), the illuminance meter 40 includes a sensor unit 40A that measures illuminance and a main unit 40B that performs the calculations necessary for measuring illuminance. The sensor unit 40A and all patterns 41 are on the same plane, and their relative positions and orientations are known. In the example of FIG. 3, the illuminance meter pattern 4 is arranged such that the sensor unit 40A of the illuminance meter 40 is positioned at the center and the patterns 41 are positioned at the four corners of a plate-like member so that each pattern 41 is equidistant from the sensor unit 40A. In the illuminance meter pattern 4, the main unit 40B is fixed to the back side of the plate-like member. The patterns 41 are image features, and examples of such features include two-dimensional codes such as AR codes and markers such as figures. It should be noted that the illuminance meter pattern 4 does not need to be placed when controlling the robot spotlight 2 (FIG. 2). Furthermore, each pattern 41 may be placed at a different distance from the sensor unit 40A. Although each pattern 41 is illustrated as being identical, in practice, each pattern 41 needs to be a different two-dimensional code or marker, or placed in a predetermined position, so that it can be uniquely identified.

[0022] [Lighting control device configuration] The configuration of the lighting control device 3 will be described with reference to FIG. 4, the lighting control device 3 includes lighting calibration means 30, subject position measurement means (illumination target position measurement means) 34, and lighting control means 37. In the lighting control device 3, when calibrating the robot spotlight 2, the lighting calibration means 30 mainly performs processing, and when controlling the robot spotlight 2, the subject position measurement means 34 and lighting control means 37 mainly perform processing.

[0023] The illumination calibration means 30 calibrates the robot spotlight 2. As shown in FIG. 4, the illumination calibration means 30 includes an image input means 31, an illuminance meter position and orientation calculation means 32, and an illumination position and orientation calculation means 33.

[0024] The image input means 31 receives, for each reference point, an illuminometer pattern image when the robot spotlight 2 is fixed while illuminating the movement range of the illumination target and the illuminometer pattern 4 is placed at the reference point where the illuminance is maximum. In this embodiment, the image input means 31 receives two illuminometer pattern images for each reference point from the two cameras 1. The image input means 31 then outputs the illuminometer pattern image for each input reference point to the illuminometer position and orientation calculation means 32.

[0025] The illuminometer position and orientation calculation means 32 calculates the position and orientation of the illuminometer 40 from the positional relationship between the pattern 41 in the illuminometer pattern image input by the image input means 31 and the illuminometer 40 (more precisely, the sensor unit 40A) for each reference point. At this time, the illuminometer position and orientation calculation means 32 refers to the camera parameters determined by the camera calibration means 350, which will be described later. Then, the illuminometer position and orientation calculation means 32 outputs the calculated position and orientation of the illuminometer 40 to the illumination position and orientation calculation means 33.

[0026] The illumination position and orientation calculation means 33 calculates the position and orientation of the robot spotlight 2 by triangulation from the position and orientation of the illuminometer 40 when the illuminometer pattern 4 is placed at each reference point and the irradiation direction of the robot spotlight 2. Here, the irradiation direction (pan angle, tilt angle) of the robot spotlight 2 is fed back from the robot spotlight 2. Then, the illumination position and orientation calculation means 33 outputs the calculated position and orientation of the robot spotlight 2 to the illumination control means 37.

[0027] The subject position measuring means 34 measures the position of the illumination target 9. The subject position measuring means 34 measures the position of the illumination target 9 using an arbitrary method, and outputs the measured position of the illumination target 9 to the illumination control means 37.

[0028] In this embodiment, the subject position measuring means 34 measures the position of the illumination target 9 as the position of the subject using the method described in JP 2023-162561 A. Here, the subject position measuring means 34 (subject area estimating means 351) can measure each desired irradiation portion set in advance on the illumination target 9 by deep learning. As shown in FIG. 4 , the subject position measuring means 34 includes a two-dimensional subject area estimating means 35 and a three-dimensional subject area reconstructing means 36.

[0029] The two-dimensional subject area estimation means 35 estimates, for each camera 1, a two-dimensional subject area of ​​the captured image input (two-dimensional subject area). The two-dimensional object region estimation means 35 includes a camera calibration means 350 and an object region estimation means 351 .

[0030] The camera calibration means 350 performs camera calibration of the camera 1. Here, camera calibration is a process of estimating the camera parameters of the camera 1. The camera parameters include the camera position, camera attitude, focal length, principal point position, etc. of camera 1. The camera position is the world coordinate of the optical center of camera 1. The camera attitude is the three-axis attitude angles (tilt, pan, and roll) and rotation matrix that define the attitude of the imaging surface relative to the world coordinate.

[0031] Note that camera calibration can be performed using general methods described in, for example, Reference 1 below, Japanese Patent Application Laid-Open No. 2018-189580, etc., and therefore detailed explanations will be omitted here.

[0032] Reference 1: Zhengyou Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.22, no. 11, pp. 1330-1334, 2000.

[0033] The camera calibration means 350 performs camera calibration as a preprocessing step before the subject position measurement means 34 estimates the three-dimensional subject area. The camera calibration means 350 outputs the estimated camera parameters to the three-dimensional object region reconstruction means 36 and the illuminance meter position and orientation calculation means 32. The camera calibration means 350 may be configured as a separate camera calibration device outside the object position measurement means 34.

[0034] The subject area estimation means 351 estimates a two-dimensional subject area for each image captured by the plurality of cameras 1. The subject region estimation means 351 can estimate the subject region from the captured image by general object detection. For example, the subject region estimation means 351 estimates a two-dimensional subject region from the captured image captured by the camera 1 using a machine learning model such as a neural network that has been trained in advance to detect the subject region from the captured image. Specifically, the subject region estimation means 351 can simultaneously measure the positions of the actor's head and hands by using "YOLOX-Body-Head-Face-HandLR-Dist" as deep learning (Reference 2).

[0035] Reference 2: Katsuya Hyodo, KazuhitoTakahashi, iwatake, Nobuo Tsukamoto, Ibai Gorordo, fateshelled, Ar-Ray, Fabio M. Sim, w-okada, genarks, Aditya Ardiya, Kenji Asaba, N■ L■ Pe, karaage, Yuki Masaoka, & wok!. (2023). PINTO0309 / PINTO_model_zoo: DOI (1.0.0). Zenodo.

[0036] Although an example of a subject will be described as a person, it does not necessarily have to be a person. That is, the subject area estimation means 351 can set any object as a subject by learning in advance an object other than a person as a machine learning model.

[0037] The subject area estimation means 351 outputs the estimated two-dimensional subject area for each camera 1, specifically the positions of the vertices of the two-dimensional subject area on the image plane of the captured image, to the three-dimensional subject area reconstruction means .

[0038] The three-dimensional object region reconstructing means 36 reconstructs the three-dimensional object region of the object from the two-dimensional object region for each camera estimated by the two-dimensional object region estimating means 35 based on the camera parameters. The three-dimensional object region reconstruction means 36 includes a projection line calculation means 360 , a nearest point vertical foot calculation means 361 , an object region vertex calculation means 362 , and a three-dimensional shape approximation means 363 .

[0039] The projection line calculation means 360 converts the center of gravity (center of the bounding box) of the two-dimensional subject area for each camera 1 into world coordinates, and calculates a straight line passing through the center of gravity in the world coordinates and the optical center of the camera as a projection line. The projection line calculation means 360 outputs information specifying the projection line for each camera 1 (here, the camera origin (optical center) and the inclination of the projection line) to the nearest point vertical foot calculation means 361.

[0040] The nearest point vertical foot calculation means 361 calculates the position of the vertical foot from the nearest point of the projection line to each projection line for each camera 1. Here, the vertical foot is the point where the foot of a perpendicular line dropped to a straight line (here, the projection line) intersects with the straight line. The nearest neighbor point vertical foot calculation means 361 outputs the calculated vertical foot position of the projection line for each camera 1 to the object area vertex calculation means 362 .

[0041] The object area vertex calculation means 362 projects the vertices of the two-dimensional object area onto a projection plane that projects the center of gravity of the two-dimensional object area onto the position of the foot drop for each camera 1, and calculates the world coordinates of the vertices after projection. The subject area vertex calculation means 362 outputs the world coordinates (three-dimensional coordinates) corresponding to the calculated vertices to the three-dimensional shape approximation means 363.

[0042] The three-dimensional shape approximation means 363 approximates the vertex positions of the projected world coordinates calculated by the subject area vertex calculation means 362 with a three-dimensional shape, and calculates shape information of the three-dimensional shape as the three-dimensional subject area. In this way, the three-dimensional shape approximation means 363 can express the area in three-dimensional space where the subject exists as a single three-dimensional shape from the point group of multiple vertices. Here, the three-dimensional shape approximation means 363 approximates all the vertices calculated by the subject area vertex calculation means 362 with a spherical shape, and calculates shape information of the approximated sphere as the three-dimensional subject area.

[0043] Each means of the subject position measuring means 34 is the same as the method described in Japanese Patent Application Laid-Open No. 2023-162561, and therefore further explanation will be omitted.

[0044] The lighting control means 37 calculates the irradiation direction of the robot spotlight 2 based on the position of the lighting target 9 and the position and posture of the robot spotlight 2 so that the lighting target 9 is positioned on the optical axis of the robot spotlight 2, and controls the robot spotlight 2 according to the calculated irradiation direction.

[0045] In this embodiment, the lighting control means 37 calculates the lighting direction of the robot spotlight 2 so that one of the desired lighting portions of the lighting target 9 is positioned on the optical axis of the robot spotlight 2. As described above, the subject position measurement means 34 measures the position of the lighting target 9 for each desired lighting portion, such as the actor's head or hands. Therefore, the lighting control means 37 can control the robot spotlight 2 to illuminate the desired lighting portion specified by the user of the lighting control system 100 (hereinafter referred to as "user").

[0046] The lighting control means 37 generates a control signal for driving the robot spotlight 2 in the calculated irradiation direction, and outputs the generated control signal to the robot spotlight 2.

[0047] [Robot Spotlight calibration behavior] The operation of calibrating the robot spotlight 2 will be described with reference to FIG. 5, in step S1, the user illuminates the robot spotlight 2 within the movement range of the illumination target 9. As a result, the robot spotlight 2 is fixed in a state where it illuminates the movement range of the illumination target 9, and the illumination direction (pan angle, tilt angle) of the robot spotlight 2 is determined.

[0048] In step S2, the user places the illuminance meter pattern 4 at the position (reference point) where the illuminance is maximum. When the illuminance meter 40 indicates the maximum illuminance due to the robot spotlight 2, it can be approximated that the sensor unit 40A is located on the optical axis of the robot spotlight 2, and the robot spotlight 2 and the illuminance meter pattern 4 are directly facing each other.

[0049] At this point, the user can visually check the illuminance meter 40 to determine whether the illuminance has reached its maximum. The position at which the illuminance is maximum within the range of movement of the illumination target 9 is called the "reference point," and one or more reference points exist within the range of movement of the illumination target 9. In this state, the two cameras 1 capture the illuminance meter pattern 4 and output the illuminance meter pattern image to the image input means 31.

[0050] In step S3, the illuminance meter position and orientation calculation means 32 calculates the position and orientation of the illuminance meter 40 from the illuminance meter pattern images of the two cameras 1.

[0051] In step S4, the lighting control device 3 determines whether or not the processing of steps S1 to S3 has been performed two or more times. In other words, the lighting control device 3 determines whether or not illuminometer pattern images have been input from two or more reference points. If the processing of steps S1 to S3 has been executed two or more times (Yes in step S4), the lighting control device 3 proceeds to step S5. If the processing of steps S1 to S3 has not been executed two or more times (No in step S4), the lighting control device 3 returns to the processing of step S1.

[0052] <Calculating the position and orientation of the illuminance meter> Calculation of the position and orientation of the illuminance meter 40 will be described in detail with reference to FIG. In FIG. 6, for ease of explanation, it is assumed that there are three patterns 41 (411 to 413), and the patterns 41 are represented by black circle markers.

[0053] As shown in FIG. 6, the world coordinates of patterns 411 to 413 are calculated using the stereo method from the illuminance meter pattern images taken by the two cameras 1 and the camera parameters calculated by the camera calibration means 350. W p1~ W p3 is found.

[0054] Since the relative positions of the illuminance meter 40 (sensor unit 40A) and the patterns 411 to 413 are known, the world coordinates of the illuminance meter 40 are W p m and posture W R m For example, in FIG. 6, when the distances a, b, and c between the illuminance meter 40 and the patterns 411 to 413 are known, the world coordinates of the illuminance meter 40 can be calculated by finding s and t that satisfy the following equation (1): W p m can be calculated.

[0055]

number

[0056] In addition, the attitude of the illuminance meter 40 (sensor unit 40A) in the world coordinate system W R m can be obtained by approximating the orientation of the illuminance meter pattern 4. For example, if the normal direction of the illuminance meter pattern 4 is the Z axis, the orientation of the illuminance meter 40 is W R m can be calculated using the following formula (2): (A, B) indicates the inner product of A and B.

[0057]

number

[0058] In summary, the illuminance meter position and orientation calculation means 32 calculates the world coordinates of the illuminance meter 40 using equation (1). W p m Furthermore, the illuminance meter position and orientation calculation means 32 calculates the orientation of the illuminance meter 40 using equation (2). W R m Calculate.

[0059] It is assumed that the position (world coordinate system) of the illuminance meter 40 at the reference points α and β is on a straight line in the optical axis direction of the robot spotlight 2, with the position (world coordinate system) of the rotation center of the robot spotlight 2 as the starting point. In this case, the position of the rotation center of the robot spotlight 2 is a fixed value that is unrelated to panning and tilting. Furthermore, the attitude (world coordinate system) of the robot spotlight 2 in the optical axis direction at the reference points α and β is calculated by adding the amount of change in attitude of the robot spotlight 2 at the reference points α and β (a rotation matrix calculated from panning and tilting, and the rotation center coordinate system of the robot spotlight 2) to the attitude (world coordinate system) of the rotation center of the robot spotlight 2.

[0060] In the example of Figure 6, there are three patterns 41, but there may also be four patterns 41 (Figure 4). In this way, when the number of patterns 41 is four or more, the position and orientation of the illuminance meter 40 is found by optimization so that the reprojection error is minimized. Specifically, four positions and orientations of the patterns 41 are found for each camera 1, and after optimizing the position and orientation of each pattern 41 with the values ​​between cameras 1, the position and orientation of the illuminance meter 40 is found. For example, the position and orientation of each pattern 41 is optimized by solving the following equation (3) using a nonlinear optimization method such as the Rubenberg-Marquardt method.

[0061]

number

[0062] Returning to FIG. 5, we continue with the operation of calibrating the robot spotlight 2. In step S5, the lighting position and orientation calculation means 33 calculates the position and orientation of the center of rotation of the robot spotlight 2 using two or more sets of pan and tilt angles of the robot spotlight 2 and the position and orientation of the illuminance meter 40.

[0063] <Calculation of the position and orientation of Robot Spotlight 2> Calculation of the position and orientation of the robot spotlight 2 will be described in detail with reference to FIGS.

[0064] As described above, steps S1 to S3 are repeated at least twice while changing the position of the reference point. As shown in FIG. 7, it is assumed that the camera 1 captures an image of the illuminance meter pattern 4 with the illuminance meter pattern 4 positioned at the reference point α, and the position and orientation of the illuminance meter 40 at the reference point α are calculated. Next, it is assumed that the camera 1 captures an image of the illuminance meter pattern 4 with the illuminance meter pattern 4 positioned at the reference point β, and the position and orientation of the illuminance meter 40 at the reference point α are calculated. In FIG. 7, the illuminance meter pattern 4 positioned at the reference point α is denoted by the symbol 4 α and the illuminance meter pattern 4 located at the reference point β is designated by the symbol 4 β is attached.

[0065] Here, at the reference point α, the pan angle θ of the robot spotlight 2 p,α and tilt angle θ t,α is known. Pan angle θ p,α and tilt angle θ t,α represents the change in posture due to the control of the robot spotlight 2 in the robot spotlight coordinate system. Also, the position of the illuminance meter 40 in the world coordinate system when the illuminance meter 40 indicates the maximum illuminance W p α and posture W R α is also known.

[0066] At the reference point β, the pan angle θ of the robot spotlight 2 p,β and tilt angle θ t,β is known. Pan angle θ p,β and tilt angle θ t,βrepresents the change in posture due to the control of the robot spotlight 2 in the robot spotlight coordinate system, similar to the reference point α. Also, the position of the illuminance meter 40 in the world coordinate system when the illuminance meter 40 indicates the maximum illuminance W p β and posture W R β is also known.

[0067] On the other hand, the position of the rotation center of the robot Spotlight 2 in the world coordinate system W p l and posture W R l is unknown. W p l and posture W R l is when there is no change in posture due to the control of the robot spotlight 2 (pan angle θ p,α and tilt angle θ t,α is zero) and is expressed in the world coordinate system.

[0068] Position of illuminance meter 40 at reference points α and β in the world coordinate system W p α , W p β and the position of the rotation center of the robot Spotlight 2 in the world coordinate system. W p l and posture W R l The relationship is expressed by the following equation (4).

[0069]

number

[0070] As shown in FIG. 8(a), the distance L from the illuminance meter 40 to the robot spotlight 2 at the reference points α and β α ,L β is expressed by the following equation (5): α ,L βIn FIG. 8(a), the robot spotlight 2 and the illuminance meter 40 are abbreviated as black circles to make the drawing easier to see. The illuminance meter 40 located at the reference point α is denoted by the symbol 40. α and the illuminance meter 40 located at the reference point β is designated by the symbol 40 β is attached.

[0071]

number

[0072] Unit vector representing the direction of the optical axis of the robot spotlight 2 at the reference points α and β α e LightAxis , β e LightAxis is expressed by the following equation (6): α e LightAxis , β e LightAxis is defined in the robot spotlight coordinate system and is known. For example, if the positive direction of the Z axis is the positive direction of the optical axis, 0 e LightAxis is expressed by the following equation (7).

[0073]

number

number

[0074] Rotation matrix representing the change in posture due to the control of robot spotlight 2 at reference points α and β l R α, l R β is the pan angle θ of the robot spotlight 2 p,α ,θ p,β and tilt angle θ t,α ,θ t,β Using the rotation matrix, it can be simply expressed by the following equation (8). l R α, l R βis an orthogonal matrix defined in the robot spotlight coordinate system and is known.

[0075]

number

[0076] The relationship between the orientation of the illuminance meter 40 in the world coordinate system at the positions α and β and the orientation of the center of rotation of the robot spotlight 2 in the world coordinate system is expressed by the following equation (9). As shown in FIG. 8(b), the orientation of the illuminance meter 40 at the reference points α and β W R α 0 e LightAxis · W R β 0 e LightAxis is the orientation of the robot spotlight 2 in the optical axis direction at the reference points α and β. W R l l R α 0 e LightAxis · W R l l R β 0 e LightAxis Since the definition is that the object faces the object, the formula (9) is established. Then, the postures at the reference points α and β are W R l is calculated, so the posture W R l The constraint that is an orthogonal matrix and the determinant is 1 ( W R l ∈SO(3)), we optimize Equation (10). This results in the orientation of the center of rotation of the robot spotlight 2 in the world coordinate system. W R l is obtained.

[0077]

number

number

[0078] As shown in Figure 9, by triangulation, the distance L α ,L β Calculate the distance L α ,L β The relationship is expressed by the following equation (11). Point Ω is located between the robot spotlight 2 and the illuminance meter 40. α ,40 β The intersection point of the perpendicular line to the line segment αβ connecting the two is the line segment αβ. Also, the distance L Ω represents the distance between the robot spotlight 2 and the point Ω, which is unknown.

[0079]

number

[0080] Φ α The robot spotlight 2 is a lux meter 40 α represents the angle between the line segment αβ and the line segment Φ β The robot spotlight 2 is a lux meter 40 β represents the angle between the line segment αβ and the line segment Φ α ,Φ β is expressed by the following equation (12) and is known.

[0081]

number

[0082] L αβ is the distance between the reference points α and β, and is expressed by the following equations (13) and (14). αβ is known. This allows us to calculate the distance L α ,L β is obtained.

[0083]

number

number

[0084] In equation (4), the distance L α ,L β and the attitude in the world coordinate system W R l Substituting the above, we get equation (15). This gives the coordinates of robot spotlight 2. W p l is obtained.

[0085]

number

[0086] To summarize the above, the lighting position and orientation calculation means 33 calculates the orientation of the robot spotlight 2 using equations (9) and (10). W R l Furthermore, the lighting position and orientation calculation means 33 calculates the coordinates of the robot spotlight 2 using equation (15). W p l Calculate.

[0087] Returning to FIG. 5, we continue with the operation of calibrating the robot spotlight 2. In step S6, the user verifies whether there is a problem with the control of the robot spotlight 2 using the position and posture of the robot spotlight 2 calculated in step S5. For example, the user visually verifies whether there is a problem with the control of the robot spotlight 2. If it is verified that there is no problem with the control of the robot spotlight 2 (Yes in step S6), the lighting control device 3 ends the calibration of the robot spotlight 2. If it cannot be verified that there is no problem with the control of the robot spotlight 2 (No in step S6), the lighting control device 3 returns to the processing in step S1.

[0088] [Operation when controlling the robot spotlight] The operation when controlling the robot spotlight 2 will be described with reference to FIG. It is assumed here that the camera calibration means 350 has previously performed camera calibration of the camera 1 and acquired the camera parameters.

[0089] 10, in step S10, the lighting control device 3 determines whether to continue controlling the robot spotlight 2. For example, when a user issues a command to end control, the lighting control device 3 determines not to continue controlling the robot spotlight 2. If the control of the robot spotlight 2 is to be continued (Yes in step S10), the lighting control device 3 proceeds to the processing of step S11. If the control of the robot spotlight 2 is not to be continued (No in step S10), the lighting control device 3 ends the control of the robot spotlight 2.

[0090] In step S11, the subject area estimation means 351 estimates a two-dimensional subject area for each camera 1 from the captured image captured by the camera 1 using a trained machine learning model. Here, the two-dimensional subject area is a bounding box.

[0091] In step S12, projection line calculation means 360 converts the center of gravity of the two-dimensional object region (center of the bounding box) into world coordinates for each camera 1, and calculates a straight line passing through the center of gravity from the camera origin (optical center) as a projection line. The center of gravity of the two-dimensional object region is projected onto this projection line for each camera 1.

[0092] In step S13, the nearest point vertical foot calculation means 361 calculates the position of the vertical foot perpendicularly extending from the nearest point of the projection lines of the multiple cameras 1 calculated in step S12 to each projection line. Based on this vertical foot position, the nearest point vertical foot calculation means 361 can identify the position where the center of gravity of the two-dimensional subject area is projected.

[0093] In step S14, the object region vertex calculation means 362 projects the vertices of the two-dimensional object region onto a projection plane that projects the center of gravity of the two-dimensional object region onto the position of the foot drop calculated in step S13, and calculates the world coordinates of the vertices after projection. The object will be located inside the vertices of the bounding box calculated for this camera 1.

[0094] In step S15, the three-dimensional shape approximation means 363 approximates all the vertices (three-dimensional coordinates) calculated in step S4 to the surface of a sphere, and calculates the shape of the approximated sphere (center position and radius) as the three-dimensional subject area.

[0095] In step S16, the illumination control means 37 calculates the irradiation direction of the robot spotlight 2 so that the center of the three-dimensional subject area is on the optical axis of the robot spotlight 2. In step S17, the lighting control means 37 outputs a control signal to the robot spotlight 2 to control the irradiation direction of the robot spotlight 2. Thereafter, the lighting control device 3 returns to the processing of step S10.

[0096] [Actions and Effects] As described above, the lighting control system 100 can control the illumination direction of the robot spotlight 2 using the calculated position and posture of the robot spotlight 2. This eliminates the need for the lighting control system 100 to pre-program the illumination direction of the robot spotlight 2, and allows the robot spotlight 2 to follow the arbitrarily moving illumination target 9. Furthermore, with the lighting control system 100, the illumination target 9 does not need to be equipped with a beacon, so illumination does not concentrate on the beacon and the desired illumination location can be illuminated.

[0097] (Variation) Although the embodiments have been described in detail above, the present invention is not limited to the above-described embodiments, and includes design modifications and the like within the scope of the present invention.

[0098] In the above-described embodiment, the robot spotlight is described as tracking an actor moving on the floor of a film studio, but the present invention is not limited to this. For example, the lighting control device can emit measurement light to measure the three-dimensional position of an illumination target (subject, background structure) appearing in a captured image. Furthermore, the lighting control device can be used as smart home lighting to make the robot spotlight track a resident in a home. Furthermore, the lighting control device can be used as a security device to detect a suspicious person from a captured image and make the robot spotlight track the detected suspicious person.

[0099] In the above embodiment, the subject position measuring means is described as using the technique described in Reference 2, but the subject position measuring technique is not particularly limited.

[0100] In the above-described embodiment, the position of the robot spotlight is fixed, but this is not limiting. That is, the position of the robot spotlight may also be variable. In this case, the translation direction is calibrated as many times as the number of degrees of freedom of translation of the robot spotlight, and the position of the robot spotlight can be determined from the correspondence between the degrees of freedom of translation of the robot spotlight and the control value for translation.

[0101] In the above-described embodiment, the lighting control device is described as independent hardware, but the present invention is not limited to this. For example, the present invention can be realized by a program that causes hardware resources such as a CPU, memory, and hard disk of a computer to function as the lighting control device. This program may be distributed via a communication line or written to a recording medium such as a CD-ROM or flash memory. [Explanation of symbols]

[0102] 100 Lighting Control System 1 camera 2 Robot Spotlight (Lighting) 3 Lighting control device 4 Illuminance meter pattern 9 Lighting Target 30 Lighting calibration means 31 Image input means 32 Illuminance meter position / orientation calculation means 33 Lighting position and orientation calculation means 34 Subject position measurement means (illumination target position measurement means) 35 2D object area estimation means 36 3D object area reconstruction means 37 Lighting control means 40 illuminance meter 40A sensor part 40B Main body 41 patterns 350 Camera calibration method 351 Subject area estimation means 360 Projection line calculation means 361 Nearest point drop foot calculation method 362 Subject area vertex calculation means 363 Three-dimensional shape approximation means

Claims

1. A lighting control device that calibrates lighting in order to control the direction of illumination of lighting, using illuminometer pattern images that are captured by a plurality of cameras and that include an illuminometer that measures illuminance and a plurality of patterns arranged around the illuminometer, an image input means for inputting an image of the illuminance meter pattern for each reference point when the illumination is fixed while illuminating the moving range of an illumination target and the illuminance meter pattern is placed at the reference point where the illuminance is maximum; an illuminance meter position and orientation calculation means for calculating the position and orientation of the illuminance meter for each of the reference points from the positional relationship between the pattern in the illuminance meter pattern image input from the image input means and the illuminance meter; an illumination position and orientation calculation means for calculating the position and orientation of the illumination by triangulation from the position and orientation of the illumination meter and the irradiation direction of the illumination when the illumination meter pattern is placed at each reference point; A lighting control device comprising:

2. an illumination target position measuring means for measuring the position of the illumination target; an illumination control means for calculating an illumination direction of the illumination based on the position of the illumination target and the position and orientation of the illumination so that the illumination target is positioned on an optical axis of the illumination, and for controlling the illumination in accordance with the calculated illumination direction; The lighting control device according to claim 1 , further comprising:

3. The illumination target position measuring means measures each desired irradiation portion set in advance on the illumination target by deep learning, 3. The lighting control device according to claim 2, wherein the lighting control means calculates the lighting direction so that any desired portion of the lighting target is positioned on the optical axis of the lighting.

4. A program for causing a computer to function as the lighting control device according to any one of claims 1 to 3.

5. The lighting control device according to any one of claims 1 to 3; an illuminance meter pattern including an illuminance meter for measuring illuminance and a plurality of patterns arranged around the illuminance meter; A lighting control system comprising:

Images