Camera system and signal processing device

Abstract

The present disclosure relates to a camera system and a signal processing device which make it possible to easily determine attachment play between an imaging camera and a sensor unit. A camera system according to the present invention comprises: an imaging camera that includes an inertial sensor; a sensor unit that includes an inertial sensor; and a signal processing device that detects play in the sensor unit attached to the imaging camera. The signal processing device includes: a first direction determination unit that determines, from a sensor signal of the inertial sensor of the imaging camera, a prescribed axial direction of the imaging camera; a second direction determination unit that determines, from a sensor signal of the inertial sensor of the sensor unit, an axial direction of the sensor unit which is identical to the prescribed axial direction of the imaging camera; and an orientation determination unit that compares the axial directions of the imaging camera and the sensor unit to detect play. Technology according to the present disclosure can be applied to, for example, camera systems and the like in which a sensor unit is attached to an imaging camera.

Description

Camera system and signal processing device

[0001] The present disclosure relates to a camera system and a signal processing device, and more particularly to a camera system and a signal processing device that allow easy confirmation of looseness in the attachment of an imaging camera and a sensor unit.

[0002] There is a camera tracking system in which a sensor unit is attached to an imaging camera and the sensor unit tracks the movement of the imaging camera. In a camera tracking system, when attaching the sensor unit to the imaging camera, for example, if the screws are loose, rattles can occur between the imaging camera and the sensor unit. For example, as disclosed in Patent Document 1, there are attitude stabilization devices that have a mount to which a camera can be attached and maintain the attitude of the mount constant, but there are no devices that detect rattles between the imaging camera and the sensor unit.

[0003] International Publication No. 2018 / 016514

[0004] Generally, when checking for looseness, users check the looseness by hand after fastening the housings together. However, when checking by hand, there is a risk that users may forget to check for looseness in a specific direction. There is also a risk that cases where the looseness gradually increases over time may go unnoticed.

[0005] The present disclosure has been made in view of the above circumstances, and makes it possible to easily check for looseness in the attachment between an imaging camera and a sensor unit.

[0006] A camera system according to a first aspect of the present disclosure comprises: an imaging camera having an inertial sensor; a sensor unit having the inertial sensor; and a signal processing device that detects wobble of the sensor unit attached to the imaging camera, wherein the signal processing device has: a first direction determination unit that determines a predetermined axial direction of the imaging camera from a sensor signal of the inertial sensor of the imaging camera; a second direction determination unit that determines an axial direction of the sensor unit that is the same as the predetermined axial direction from a sensor signal of the inertial sensor of the sensor unit; and an attitude determination unit that compares the axial directions of the imaging camera and the sensor unit to detect wobble.

[0007] In a first aspect of the present disclosure, there is provided an imaging camera having an inertial sensor, a sensor unit having an inertial sensor, and a signal processing device that detects wobble of the sensor unit attached to the imaging camera, wherein the signal processing device determines a predetermined axial direction of the imaging camera from a sensor signal of the inertial sensor of the imaging camera, determines an axial direction of the sensor unit that is the same as the predetermined axial direction from a sensor signal of the inertial sensor of the sensor unit, and compares the axial directions of the imaging camera and the sensor unit to detect wobble.

[0008] A signal processing device according to a second aspect of the present disclosure includes a display control unit that controls the display of a screen that detects any looseness in the attachment of the sensor unit based on the sensor signal of the inertial sensor of the imaging camera and the sensor signal of the inertial sensor of the sensor unit attached to the imaging camera, and the display control unit controls the display of a first screen that indicates at least one of a swing motion or a stationary state of the imaging camera, and a second screen that displays a comparison result between the sensor signal of the inertial sensor of the imaging camera in the stationary state and the sensor signal of the inertial sensor of the sensor unit.

[0009] In a second aspect of the present disclosure, a control is performed to display a screen for detecting wobble in the attachment of the sensor unit based on a sensor signal of the inertial sensor of the imaging camera and a sensor signal of the inertial sensor of a sensor unit attached to the imaging camera, and a control is performed to display a first screen for instructing at least one of a swing operation or a stationary state of the imaging camera, and a second screen for displaying a comparison result between the sensor signal of the inertial sensor of the imaging camera in the stationary state and the sensor signal of the inertial sensor of the sensor unit.

[0010] The signal processing device according to the first and second aspects of the present disclosure can be realized by causing a computer to execute a program. The program executed by the computer to realize the signal processing device can be provided by transmitting it via a transmission medium or by recording it on a recording medium.

[0011] The signal processing device may be an independent device or an internal block constituting a single device.

[0012] 1 is a diagram illustrating an example in which a sensor unit is attached to an imaging camera. FIG. 2 is a diagram illustrating an example in which a sensor unit is attached to an imaging camera. FIG. 3 is a diagram illustrating an example configuration of a camera system according to an embodiment of the present disclosure. FIG. 4 is a block diagram illustrating an example configuration of the camera system of FIG. 3. FIG. 5 is a diagram illustrating details of rattle detection. FIG. 6 is a diagram illustrating details of rattle detection. FIG. 7 is a diagram illustrating details of rattle detection. FIG. 8 is a diagram illustrating an example screen displayed in rattle detection processing. FIG. 9 is a diagram illustrating an example screen displayed in rattle detection processing. FIG. 10 is a diagram illustrating an example screen displayed in rattle detection processing. FIG. 11 is a diagram illustrating an example screen displayed in rattle detection processing. FIG. 12 is a flowchart illustrating rattle detection processing in manual mode. FIG. 13 is a flowchart illustrating rattle detection processing in auto mode. FIG. 14 is a block diagram illustrating an example configuration of a computer as a signal processing device.

[0013] Hereinafter, a mode for carrying out the technology of the present disclosure (hereinafter referred to as an embodiment) will be described with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration will be assigned the same reference numerals to avoid redundant description. The description will be given in the following order: 1. Causes of rattle occurrence 2. Example configuration of a camera system 3. Detailed explanation of rattle detection 4. Rattle detection UI 5. Flowchart of rattle detection processing in manual mode 6. Flowchart of rattle detection processing in auto mode 7. Example computer configuration

[0014] 1. Causes of Wobble A tracking system attaches a sensor unit to a target object to be tracked, and tracks the movement of the target object. When the target object is an imaging camera, a sensor unit is attached to the imaging camera.

[0015] 1 and 2 show an example in which a sensor unit is attached to an imaging camera.

[0016] FIG. 1 shows an example in which a sensor unit is directly attached to the handle of an imaging camera.

[0017] FIG. 2 shows an example in which a sensor unit is attached to an imaging camera using a flexible arm.

[0018] When the sensor unit is attached directly to the handle of the imaging camera, as shown in Figure 1, rattle is unlikely to occur. On the other hand, when the sensor unit is attached using a flexible arm, as shown in Figure 2, rattle is likely to occur. Other methods, such as attaching the sensor unit to the imaging camera using a rig, vary depending on the user. During attachment, rattle can occur between the imaging camera and the sensor unit, for example, due to loose screws. Here, rattle refers to a state in which there is a gap or space between the imaging camera and the sensor unit, allowing for movement while one is fixed (stationary), allowing the other to move slightly (vibrate). When the target object to be tracked is an imaging camera, the imaging camera serves as the reference, and rattle occurs in the sensor unit. When rattle occurs, the sensor unit vibrates even when the imaging camera being tracked is stationary, resulting in vibration noise from the sensor unit being superimposed on the tracking data.

[0019] Therefore, in the following, we propose a system that allows easy confirmation of looseness in the attachment of a sensor unit in a tracking system in which an imaging camera is used as the object to be tracked and a sensor unit is attached to the imaging camera.

[0020] 2. Configuration Example of Camera System> FIG. 3 shows a configuration example of a camera system according to an embodiment of the present disclosure.

[0021] 3 includes at least an imaging camera 11, a sensor unit 12, a signal processing device 13, and a display device 14. The imaging camera 11, the sensor unit 12, the signal processing device 13, and the display device 14 are connected to each other so as to be able to communicate with each other via any communication interface, whether wired or wireless, such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface), Bluetooth (registered trademark), Wi-Fi, or LAN (Local Area Network).

[0022] The camera system 1 is part of a system that tracks the movement of an imaging camera 11 using a sensor unit 12 attached to the imaging camera 11. The camera system 1 is configured so that any wobble of the sensor unit 12 attached to the imaging camera 11 can be easily confirmed.

[0023] FIG. 4 is a block diagram showing an example of the configuration of the camera system 1 of FIG.

[0024] As described above, the camera system 1 includes at least the imaging camera 11, the sensor unit 12, the signal processing device 13, and the display device .

[0025] The imaging camera 11 includes, for example, an imaging lens, an image sensor, a signal processor, etc. (none of which are shown), and in addition, at least an IMU 11m. The sensor unit 12 includes, for example, an image sensor, a depth sensor, etc. (none of which are shown), and in addition, at least an IMU 12m. The IMU 11m is, for example, an inertial sensor including a triaxial acceleration sensor and a triaxial angular velocity sensor, and outputs sensor signals of acceleration and angular velocity in three axial directions, i.e., the X-axis, the Y-axis, and the Z-axis, to the signal processing device 13. The IMU 11m is, for example, attached inside the imaging camera 11 so that the positive direction of the Z-axis is facing upward (toward the ceiling). The IMU 12m is, for example, an inertial sensor including a triaxial acceleration sensor and a triaxial angular velocity sensor, and outputs sensor signals of acceleration and angular velocity in three axial directions, i.e., the X-axis, the Y-axis, and the Z-axis, to the signal processing device 13. The IMU 12m is, for example, attached inside the sensor unit 12 so that the positive direction of the Z-axis is facing upward (toward the ceiling). For convenience of explanation, the IMUs 11m and 12m are mounted so that the upward direction is the positive direction of the Z axis, but this is not limiting and the IMUs can be mounted in any axial direction. The sensor signals output by the IMU 11m are signals of acceleration and angular velocity in the three axial directions of the X-axis, Y-axis, and Z-axis in a coordinate system owned by the IMU 11m itself in the imaging camera 11 (hereinafter referred to as the camera IMU coordinate system). The sensor signals output by the IMU 12m are signals of acceleration and angular velocity in the three axial directions of the X-axis, Y-axis, and Z-axis in a coordinate system owned by the IMU 12m itself in the sensor unit 12 (hereinafter referred to as the sensor unit IMU coordinate system).

[0026] The signal processing device 13 is a device that detects wobble of the sensor unit 12 attached to the imaging camera 11. After the sensor unit 12 has been attached without wobble, the signal processing device 13 performs signal processing to track the movement (position and orientation) of the imaging camera 11 by, for example, SLAM (Simultaneous Localization and Mapping) using a sensor signal from the IMU 12m of the sensor unit 12 and an image captured by an image sensor (not shown). The signal processing device 13 is configured with, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an SoC (System on a Chip), a GPU (Graphics Processing Unit), a GPGPU (General-purpose computing on graphics processing units), an AI (Artificial Intelligence) processor, etc.

[0027] The signal processing device 13, for rattle detection processing, has a rest state determination unit 31, Z direction determination units 32 and 33, an attachment attitude determination unit 34, and a display control unit 35. The sensor signal output from the IMU 11m of the imaging camera 11 is input to the rest state determination unit 31 and the Z direction determination unit 32, which is a first direction determination unit. The sensor signal output from the IMU 12m of the sensor unit 12 is input to the Z direction determination unit 33, which is a second direction determination unit.

[0028] The rest state determination unit 31 determines whether the imaging camera 11 is stationary or not using the sensor signal of the imaging camera 11. Specifically, the rest state determination unit 31 determines that the imaging camera 11 is stationary if it can determine that the velocity calculated from the sensor signal output from the acceleration sensor of the IMU 11m is almost 0 (can approximate to 0 within a predetermined threshold). If the rest state determination unit 31 determines that the imaging camera 11 is stationary, it supplies a determination signal indicating the stationary state to the mounting attitude determination unit 34.

[0029] The Z direction determination unit 32 determines the Z direction of the imaging camera 11 as the device Z direction from the sensor signal of the IMU 11m of the imaging camera 11.ｗ Z ｃ This Z direction ｗ Z ｃ As will be described in detail later, represents the Z direction after the unit vector of the camera IMU coordinate system is converted into the world coordinate system. ｗ Z ｃ = (x １ , y １ , z １ ) is supplied to the mounting attitude determination unit 34.

[0030] The Z direction determination unit 33 determines the Z direction of the sensor unit 12 as the device Z direction from the sensor signal of the IMU 12m of the sensor unit 12. ｗ Z ｓ This Z direction ｗ Z ｓ As will be described in detail later, represents the Z direction after the unit vector of the sensor unit IMU coordinate system is converted into the world coordinate system. ｗ Z ｓ = (x ２ , y ２ , z ２ ) is supplied to the mounting attitude determination unit 34.

[0031] The mounting attitude determination unit 34 determines whether the imaging camera 11 and the sensor unit 12 are mounted in the Z direction of the imaging camera 11, which is the same axial direction as the imaging camera 11 and the sensor unit 12. ｗ Z ｃ and the Z direction of the sensor unit 12 ｗ Z ｓ The mounting attitude determination unit 34 compares the detected wobble with the detected wobble and supplies the result to the display control unit 35.

[0032] The display control unit 35 generates a control signal for a screen related to the rattle detection process and supplies it to the display device 14. For example, the display control unit 35 generates a control signal for a screen for performing rattle detection and a control signal for a screen showing the results of rattle detection and supplies it to the display device 14. Furthermore, the display control unit 35 instructs the mounting posture determination unit 34 to perform a predetermined process, for example, re-execute rattle detection, based on an operation performed by the user on the screen displayed on the display device 14.

[0033] The display device 14 has, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, and displays a screen showing the rattle detection result, etc., based on the control of the display control unit 35. The display device 14 may be an information processing device equipped with a display device, such as a smartphone, a tablet, or a personal computer. The display device 14 may be incorporated as a display unit in the signal processing device 13 and configured integrally with the signal processing device 13.

[0034] The camera system 1 has the above-described configuration for detecting rattles.

[0035] 3. Detailed Description of Rattle Detection The rattle detection performed by the mounting posture determination unit 34 will be described in detail with reference to FIGS. 5 to 7. FIG.

[0036] The signal processing device 13 has the user perform a preliminary action (hereinafter referred to as a swing action) that can detect the movement of the housing of the imaging camera 11 in three axial directions. After that, the rest determination unit 31 determines that the imaging camera 11 is still. Then, when the imaging camera 11 is still, the mounting attitude determination unit 34 determines the Z direction of the imaging camera 11. ｗ Z ｃ and the Z direction of the sensor unit 12 ｗ Z ｓ The swing motion and comparison process are performed multiple times to compare the mounting attitudes of the imaging camera 11 and the sensor unit 12, in other words, the three movements (3 DoF) around the X-axis, Y-axis, and Z-axis.

[0037] For example, if the swing motion and comparison process are executed five times in total, as shown in FIG. 5, the first time the stationary state is determined after the first swing motion is the time t=T ０ In the Z direction of the imaging camera 11 ｗ Z ｃ and the Z direction of the sensor unit 12 ｗ Z ｓ are compared, and the Z direction ｗ Z ｃ and Z direction ｗ Z ｓ The difference θ ０After the second swing motion, the second stationary state is determined at time t=T １ In the Z direction of the imaging camera 11 ｗ Z ｃ and the Z direction of the sensor unit 12 ｗ Z ｓ are compared, and the Z direction ｗ Z ｃ and Z direction ｗ Z ｓ The difference θ １ After the third swing motion, the time when the third stationary state is determined is calculated as t=T ２ In the Z direction of the imaging camera 11 ｗ Z ｃ and the Z direction of the sensor unit 12 ｗ Z ｓ are compared, and the Z direction ｗ Z ｃ and Z direction ｗ Z ｓ The difference θ ２ The same process is repeated up to five times.

[0038] Next, the mounting posture determination unit 34 performs rattle detection based on the results of multiple (five) comparisons.

[0039] FIG. 6 is an example of a screen displayed on the display device 14 as a result of rattle detection based on the results of multiple (five) comparisons performed by the mounting posture determination unit 34.

[0040] The screen in Figure 6 shows five Z-axis scans as a result of the rattle detection. ｗ Z ｃ and Z direction ｗ Z ｓ The difference θ １ Or θ ５ In the graph, a waveform 51 represents the difference θ １ Or θ ５ The line 52 in the center of the waveform 51 is a line that shows the waveform 51 (difference θ １ Or θ ５ ) average θ AVE The dashed lines 53U and 53L shown above and below the straight line 52 represent the waveform 51 (difference θ １ Or θ５ ) average θ AVE θ is a threshold value used to determine whether a wobble has occurred. This threshold value can be set, for example, based on the rotation angle at which tilting of the sensor unit 12 can be subjectively perceived in the HD image, and can be, for example, ±0.1°. On the graph, portions of the waveform 51 that exceed the threshold dashed lines 53U and 53L and exhibit large fluctuations are clearly indicated in an easily identifiable manner. In the example of FIG. 5 , portions of the waveform 51 that exceed the threshold dashed lines 53U and 53L are surrounded by dashed-dotted ellipses, and are used as a warning indicator 54 indicating the need for wobble adjustment. The warning indicator 54 is not limited to this example. For example, portions of the waveform 51 that exceed the dashed lines 53U and 53L may be displayed in a different color from the other portions of the waveform 51, or may be superimposed and colored in a different color from the waveform 51, thereby making them easily identifiable. Suppose that the average θ AVE If the fluctuation range of the waveform 51 centered on the straight line 52 does not exceed the threshold, it is determined that no backlash occurs.

[0041] The mounting posture determination unit 34 supplies a graph, as shown in Figure 6, in which waveforms 51 of the difference θ obtained multiple times, a straight line 52 indicating the average, and dashed lines 53U and 53L indicating the threshold values ​​are superimposed, to the display control unit 35 as a result of the rattle detection.

[0042] The swing motion may be performed by instructing the user to do so via a screen or voice, or, if the rattle detection process is performed in the background (automatically) without the user being aware of it, the user's normal camera use motion may be regarded as the swing motion. In other words, the swing motion may be performed either explicitly by the user or not.

[0043] As described above, the mounting attitude determination unit 34 determines whether the Z direction of the imaging camera 11 is correct. ｗ Z ｃ and the Z direction of the sensor unit 12 ｗ Z ｓ The Z direction is detected by comparing the ｗ Z ｃ and ｗ Z ｓrepresents the Z direction after transformation in which the Z direction of the camera IMU coordinate system and the sensor unit IMU coordinate system is transformed into the world coordinate system.

[0044] FIG. 7 is a diagram illustrating the conversion from the camera IMU coordinate system and the sensor unit IMU coordinate system to the world coordinate system.

[0045] First, define the unit vector of the camera IMU coordinate system as ( ｃ X, ｃ Y, ｃ Z), the unit vector of the sensor unit IMU coordinate system ( ｓ X, ｓ Y, ｓ Z), and the world coordinate system is ( ｗ X, ｗ Y, ｗ Z).

[0046] World coordinate system ( ｗ X, ｗ Y, ｗ Z) can be defined as in the following equation (1): In equation (1), a virtual X-axis unit vector ｗ X ｔｍｐ Set the gravity direction ｉ g ｗ is -z, and the cross product (×) is used to calculate the world coordinate system ( ｗ X, ｗ Y, ｗ Z) is defined. ｗ X ｔｍｐ = (1,0,0) ｗ Z = - ｉ g ｗ / | ｉ g ｗ | ｗ Y = ｗ Zx ｗ X ｔｍｐ ｗ X = ｗ Y x ｗ Z ・・・・・・・・・・・・・・・(1)

[0047] The unit vector of the camera IMU coordinate system ( ｃ X, ｃ Y, ｃ Z) in the world coordinate system ( ｗ X,​ｗ Y, ｗ Z) expressed as a unit vector ( ｗ X ｃ , ｗ Y ｃ , ｗ Z ｃ ) can be calculated using the dot product (·) in equation (2), and the unit vector ( ｓ X, ｓ Y, ｓ Z) in the world coordinate system ( ｗ X, ｗ Y, ｗ Z) expressed as a unit vector ( ｗ X ｓ , ｗ Y ｓ , ｗ Z ｓ ) can be calculated using the inner product (·) using equation (3). ｗ X ｃ = ｃ X. ｗ X+ ｃ X. ｗ Y+ ｃ X. ｗ Z ｗ Y ｃ = ｃ Y. ｗ Y+ ｃ Y. ｗ Y+ ｃ Y. ｗ Z ｗ Z ｃ = ｃ Z. ｗ Z+ ｃ Z. ｗ Y+ ｃ Z. ｗ Z・・・・・・・・・・・・(2) ｗ X ｓ = ｓ X. ｗ X+ ｓ X. ｗ Y+ ｓ X. ｗ Z ｗ Y ｓ = ｓ Y. ｗ Y+ ｓ Y. ｗ Y+ ｓ Y.ｗ Z ｗ Z ｓ = ｓ Z. ｗ Z+ ｓ Z. ｗ Y+ ｓ Z. ｗ Z・・・・・・・・・・・・(3)

[0048] At this time, the Z direction of the imaging camera 11 ｗ Z ｃ and the Z direction of the sensor unit 12 ｗ Z ｓ The angle θ between these points is expressed as follows: θ = cos －１ ( ｗ Z ｃ ・ ｗ Z ｓ ) ...................................................(4) Z direction of the imaging camera 11 ｗ Z ｃ and the Z direction of the sensor unit 12 ｗ Z ｓ The angle θ formed by the angle θ will be hereinafter also referred to as the rattle component θ.

[0049] 4. Rattling Detection UI Next, a description will be given of the screen displayed on the display device 14 under the control of the display control unit 35 and the user's operation when the rattle detection process is executed.

[0050] FIG. 8 shows an example of an initial screen that is displayed when the rattle detection process is executed.

[0051] The initial screen 71 in FIG. 8 has radio buttons 72A, 72B for selecting an auto mode (first mode) or a manual mode (second mode). The user selects either the auto mode or the manual mode and selects (presses) the start button 73 to execute the selected mode. The auto mode is a mode in which the wobble detection process is automatically executed in the background without the user being aware of it. The manual mode is a mode in which the wobble detection process is executed by having the user perform a predetermined swing motion according to the screen. When the cancel button 74 is selected (pressed), the initial screen 71 is cleared.

[0052] When the user wishes to execute the rattle detection process in the manual mode, he or she selects the radio button 72B and then the start button 73. When execution of the manual mode is instructed, the manual mode start screen shown in FIG.

[0053] FIG. 9 shows an example of a manual mode start screen indicating the start of the manual mode.

[0054] 9, the words "Manual Mode" indicating that the mode is manual are displayed along with a check button 82. When the check button 82 is selected (pressed), the manual mode is initiated and the swing action instruction screen of FIG. 10 is displayed.

[0055] FIG. 10 shows an example of a swing motion instruction screen (first screen) for instructing a swing motion and a stationary state.

[0056] 10 displays an action instruction 92 for instructing the user to perform a swing action, and an end button 93 that the user selects (presses) when the user has finished the swing action and is in a stationary state. The action instruction 92 displays the words "swing camera" and an illustration of a swinging camera.

[0057] The user performs a swinging motion for a predetermined period (e.g., several seconds) in accordance with the action instruction 92, and after the swinging motion is completed, the imaging camera 11 is placed in a stationary state. The user selects the end button 93 at the timing when the imaging camera 11 is placed in a stationary state, thereby instructing the signal processing device 13 to place the imaging camera 11 in a stationary state. If a swinging motion of a certain period or more is detected, the display control unit 35 may change the action instruction 92 to, for example, "stop swinging camera," and instruct the user to place the imaging camera 11 in a stationary state. When the end button 93 is selected, the process transitions to a detection period for detecting the rattle component θ, and the display control unit 35 causes the display device 14 to display the rattle detection screen of FIG. 11 .

[0058] FIG. 11 shows an example of a rattle detection screen (second screen) that is displayed during the detection period in which the rattle component θ is detected.

[0059] 11, the display control unit 35 displays a graph 102 of the difference θ shown in FIG. 6 based on the comparison result supplied from the mounting attitude determination unit 34. The graph 102 shows the difference θ in the Z direction of the imaging camera 11, which is detected multiple times. ｗ Z ｃ and the Z direction of the sensor unit 12 ｗ Z ｓ 11 is a graph of the comparison results for one comparison. When a certain time has passed or the change in the difference θ (vibration) has converged within a predetermined range, one comparison process ends. The signal processing device 13 moves on to the next (e.g., second) comparison process, and the display control unit 35 again displays the swing action instruction screen 91 of FIG. 10 to prompt the user to perform a swing action. As multiple swing actions and comparison processes are performed, the display device 14 repeatedly displays the swing action instruction screen 91 of FIG. 10 and the rattle detection screen 101 of FIG. 11.

[0060] When the comparison process is completed a predetermined number of times, the display control unit 35 displays the rattle detection result screen of FIG. 12 based on the comparison result supplied from the mounting attitude determination unit 34 .

[0061] FIG. 12 shows an example of a rattle detection result screen (third screen) that displays the rattle detection result.

[0062] The rattle detection result screen 111 displays the message "unstable attachment detected" and the graph described in Fig. 6 as the results of a comparison process 112 performed multiple times, which indicates the rattle detection result. The "mean" field displays the average θ of the difference θ calculated multiple times by the attachment posture determination unit 34. AVE is displayed. Average θ AVEcorresponds to the tilt offset between the two housings of the imaging camera 11 and the sensor unit 12. In order to accurately know the position of the imaging camera 11 that is the tracking target, it is necessary to know the offset amount between the imaging camera 11 and the sensor unit 12 in advance, but by detecting wobble, the tilt offset between the imaging camera 11 and the sensor unit 12 can also be calculated automatically, so there is no need to detect the tilt offset between the two housings by a separate means.

[0063] If the comparison process after multiple swings shows a period exceeding the threshold and the rattle detection result is not satisfactory, the message "unstable attachment detected" and the warning display 54 described in FIG. 6 are displayed, and the rattle detection result screen 111 also serves as a screen (fourth screen) prompting the user to adjust the rattle. After the user tightens the attachment of the sensor unit 12 to the imaging camera 11 (for example, by tightening the screws), the user selects the check button 113. When the check button 113 is selected, the manual mode start screen 81 shown in FIG. 9 is displayed again, and rattle detection processing in manual mode is restarted.

[0064] On the other hand, if a satisfactory rattle detection result is obtained, the user selects (presses) the end button 114. This ends the rattle detection process in the manual mode.

[0065] 5. Flowchart of rattle detection processing in manual mode Next, rattle detection processing in manual mode will be described with reference to the flowchart in Fig. 13. This processing is performed after, for example, the check button 82 is pressed on the manual mode start screen 81 in Fig. 9.

[0066] First, in step S1, the display control unit 35 displays the swing motion instruction screen 91 shown in Fig. 10 and instructs the user to perform a swing motion. The user performs a swing motion in accordance with the motion instruction 92 on the swing motion instruction screen 91.

[0067] In step S2, the display control unit 35 determines whether the swing motion has ended, in other words, whether the user has pressed the end button 93 indicating the end of the swing motion on the swing motion instruction screen 91 in Fig. 10. The display control unit 35 repeats the process of step S2 until it is determined that the swing motion has ended, and if it is determined that the swing motion has ended, it supplies this information to the mounting posture determination unit 34 and proceeds to step S3.

[0068] In step S3, the Z direction determination units 32 and 33 start acquiring the sensor signals supplied from the imaging camera 11 and the sensor unit 12, respectively, and ｗ Z ｃ , ｗ Z ｓ is supplied to the mounting attitude determination unit 34. After step S3, the acquisition of the sensor signal is continued until the rattle detection process is completed.

[0069] In step S4, the mounting attitude determination unit 34 determines whether the imaging camera 11 is stationary. The mounting attitude determination unit 34 determines whether the imaging camera 11 is stationary based on whether a determination signal indicating a stationary state is supplied from the stationary state determination unit 31. The processing of step S4 is repeated until it is determined that the imaging camera 11 is stationary, and if it is determined that the imaging camera 11 is stationary, the processing proceeds to step S5. Note that in the manual mode, the user presses the end button 93 when they finish the swing motion and place the imaging camera 11 in a stationary state, so the processing of step S4 may be omitted.

[0070] In step S5, the mounting attitude determination unit 34 determines whether the Z direction of the imaging camera 11 is ｗ Z ｃ and the Z direction of the sensor unit 12 ｗ Z ｓ The difference θ is calculated and supplied to the display control unit 35 .

[0071] In step S6, the display control unit 35 causes the display device 14 to display the rattle detection screen 101 of Fig. 11 based on the difference θ supplied from the mounting attitude determination unit 34. Steps S5 and S6 are executed simultaneously (in parallel) until a certain time has elapsed or until it is determined that the change in the difference θ (vibration) has converged within a predetermined range.

[0072] After steps S5 and S6, in step S7, the mounting posture determination unit 34 determines whether the comparison process has been executed a predetermined number of times. If it is determined in step S7 that the comparison process has not been executed a predetermined number of times, the process returns to step S1, and the above-mentioned process is repeated.

[0073] On the other hand, if it is determined in step S7 that the comparison process has been performed multiple times, the process proceeds to step S8, and the display control unit 35 causes the display device 14 to display the rattle detection result screen 111 of Figure 12, which shows the results of the multiple comparison processes.

[0074] In step S9, the display control unit 35 determines whether or not a command to recheck the rattle detection has been issued on the rattle detection result screen 111 of Fig. 12. That is, if the check button 113 has been pressed on the rattle detection result screen 111 of Fig. 12, the display control unit 35 determines that a command to recheck the rattle detection has been issued. On the other hand, if the end button 114 has been pressed on the rattle detection result screen 111 of Fig. 12, the display control unit 35 determines that a command to recheck the rattle detection has not been issued.

[0075] If it is determined in step S9 that a command to recheck the rattle detection has been issued, the process returns to step S1, and the above-described process is executed again. On the other hand, if it is determined in step S9 that a command to recheck the rattle detection has not been issued, the rattle detection process in manual mode ends.

[0076] 6. Flowchart of rattle detection processing in auto mode Next, rattle detection processing in auto mode will be described with reference to the flowchart in Fig. 14. This processing is started after, for example, the auto mode radio button 72A is selected on the initial screen 71 in Fig. 8 and the start button 73 is pressed.

[0077] In step S31, the Z direction determination units 32 and 33 start acquiring the sensor signals supplied from the imaging camera 11 and the sensor unit 12, respectively, and ｗ Z ｃ , ｗ Z ｓ to the mounting attitude determination unit 34. The sensor signal from the imaging camera 11 is also supplied to the rest state determination unit 31. After step S31, the acquisition of the sensor signal is continued until the rattle detection process is completed.

[0078] In step S32, the mounting attitude determination unit 34 determines whether the imaging camera 11 is stationary or not. The mounting attitude determination unit 34 determines whether the imaging camera 11 is stationary or not based on whether a determination signal indicating a stationary state is supplied from the stationary state determination unit 31. The processing of step S32 is repeated until it is determined that the imaging camera 11 is stationary, and if it is determined that the imaging camera 11 is stationary, the processing proceeds to step S33.

[0079] In step S33, the mounting attitude determination unit 34 determines whether the Z direction of the imaging camera 11 is ｗ Z ｃ and the Z direction of the sensor unit 12 ｗ Z ｓ and calculate the difference θ.

[0080] In step S34, the mounting posture determination unit 34 determines whether the comparison process has been executed a predetermined number of times. If it is determined in step S34 that the comparison process has not been executed a predetermined number of times, the process returns to step S31, and the above-described process is repeated.

[0081] On the other hand, if it is determined in step S34 that the comparison process has been performed multiple times, the process proceeds to step S35, where the mounting posture determination unit 34 determines whether the rattle exceeds the threshold value, i.e., the average θ AVE Calculate the average θ among multiple differences θ AVE It is determined whether there is a point that exceeds a preset threshold value.

[0082] If it is determined in step S35 that the rattle does not exceed the threshold value, the process returns to step S31, and the above-described process is repeated.

[0083] On the other hand, if it is determined in step S35 that the wobble exceeds the threshold value, the process proceeds to step S36, and the mounting posture determination unit 34 supplies the comparison result, including the wobble defect, to the display control unit 35. The display control unit 35 causes the display device 14 to display a screen urging the user to adjust the wobble. For example, a screen displaying the message "There is wobble in the mounting of the sensor unit. Please adjust it" may be displayed. In addition to the warning on the screen (video), the user may be urged to adjust the wobble by audio (buzzer or audio message) or by output of a warning such as an error lamp.

[0084] After step S36, the process returns to step S31, and the above-described process is repeated. The rattle detection process in Fig. 14 ends when the power is turned off by, for example, pressing the power-off button.

[0085] According to the rattle detection process of FIGS. 13 and 14, rattles in the attachment between the imaging camera 11 and the sensor unit 12 can be easily confirmed and adjusted as necessary.

[0086] 13 and 14, the comparison process is executed multiple times, but it does not necessarily have to be executed multiple times, and it is sufficient to execute it at least once. The number of times to execute the comparison process to detect rattle may be set on a setting screen or the like.

[0087] 7. Example of Computer Configuration The series of processes executed by the signal processing device 13 can be executed by hardware or software. When the series of processes are executed by software, the programs constituting the software are installed in a computer. Here, the computer includes a microcomputer incorporated in dedicated hardware, and a general-purpose personal computer, for example, that can execute various functions by installing various programs.

[0088] FIG. 15 is a block diagram showing an example of the hardware configuration of a computer 200 serving as the signal processing device 13 that executes the above-described series of processes by a program.

[0089] In the computer 200 , a CPU (Central Processing Unit) 201 , a ROM (Read Only Memory) 202 , and a RAM (Random Access Memory) 203 are interconnected by a bus 204 .

[0090] An input / output interface 205 is also connected to the bus 204. An input unit 206, an output unit 207, a storage unit 208, a communication unit 209, and a drive 210 are connected to the input / output interface 205.

[0091] The input unit 206 includes a keyboard, mouse, microphone, touch panel, input terminal, etc. The output unit 207 includes a display, speaker, output terminal, etc. The storage unit 208 includes a hard disk, SSD (Solid State Drive), RAM disk, non-volatile memory, etc. The communication unit 209 includes a communication interface for communication via USB, HDMI (registered trademark), Bluetooth, Wi-Fi, LAN, etc. The drive 210 drives removable media 211 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0092] In the computer 200 configured as above, the CPU 201 performs the above-described series of processes by, for example, loading a program stored in the storage unit 208 into the RAM 203 via the input / output interface 205 and the bus 204 and executing the program. The RAM 203 also stores data and the like necessary for the CPU 201 to execute various processes.

[0093] The program executed by the computer 200 (the CPU 201 thereof) can be provided by being recorded on a removable medium 211 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0094] In a computer, the program can be installed in the storage unit 208 via the input / output interface 205 by inserting the removable medium 211 into the drive 210. The program can also be received by the communication unit 209 via a wired or wireless transmission medium and installed in the storage unit 208. Alternatively, the program can be installed in advance in the ROM 202 or the storage unit 208.

[0095] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0096] The steps described in the flowchart may be performed in chronological order in the order described, but they do not necessarily have to be processed in chronological order and may be performed in parallel or at any time required, such as when a call is made.

[0097] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0098] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the technology of the present disclosure.

[0099] For example, it is possible to adopt a configuration in which some of the above-described configurations are combined as appropriate.

[0100] The technology of the present disclosure can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0101] Each step described in the above flowchart can be executed by one device or can be shared and executed by multiple devices. Furthermore, if one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0102] The effects described in this specification are merely examples and are not limiting, and there may be effects other than those described in this specification.

[0103] The technology disclosed herein may employ the following configurations: (1) A camera system including an imaging camera having an inertial sensor, a sensor unit having the inertial sensor, and a signal processing device that detects wobble in the sensor unit attached to the imaging camera, wherein the signal processing device has: a first direction determination unit that determines a predetermined axial direction of the imaging camera from a sensor signal of the inertial sensor of the imaging camera; a second direction determination unit that determines an axial direction of the sensor unit that is the same as the predetermined axial direction from a sensor signal of the inertial sensor of the sensor unit; and an attitude determination unit that compares the axial directions of the imaging camera and the sensor unit to detect wobble. (2) The signal processing device further has a rest determination unit that determines whether the imaging camera is stationary, wherein the attitude determination unit compares the axial directions of the imaging camera and the sensor unit when the imaging camera is stationary to detect wobble. (3) The camera system described in (2), wherein the rest determination unit determines whether the imaging camera is stationary after a swing motion. (4) The camera system according to any one of (1) to (3), wherein the attitude determination unit compares the axial directions of the imaging camera and the sensor unit multiple times and performs rattle detection based on the results of the multiple comparisons. (5) The camera system according to any one of (1) to (4), wherein the signal processing device has a first mode in which the rattle detection is performed in the background and a second mode in which the rattle detection is performed by instructing a user to perform an operation. (6) The camera system according to any one of (1) to (5), wherein the signal processing device further has a display control unit that displays the rattle detection result on a screen.(7) A signal processing device including a display control unit that controls the display of a screen for detecting wobble in the mounting of the sensor unit based on a sensor signal from an inertial sensor of an imaging camera and a sensor signal from an inertial sensor of a sensor unit attached to the imaging camera, wherein the display control unit controls the display of: a first screen that instructs at least one of a swing motion or a stationary state of the imaging camera; and a second screen that displays a comparison result between the sensor signal of the inertial sensor of the imaging camera in the stationary state and the sensor signal of the inertial sensor of the sensor unit. (8) The signal processing device described in (7), wherein the display control unit repeatedly displays the first screen multiple times to allow a user to perform a swing motion and a stationary state of the imaging camera multiple times. (9) The signal processing device described in (7) or (8), wherein the display control unit further displays a third screen that displays a result of the wobble detection based on one or more of the comparison results. (10) The signal processing device described in (9), wherein the display control unit displays, on the third screen, a location where the comparison result is greater than a threshold. (11) The signal processing device according to (9) or (10), wherein the display control unit displays an average value of the comparison results on the third screen. (12) The signal processing device according to any one of (7) to (11), wherein the display control unit displays a fourth screen that prompts adjustment of the jitter when one or more of the comparison results are greater than a threshold value.

[0104] REFERENCE SIGNS LIST 1 camera system, 11 imaging camera, 11m IMU, 12 sensor unit, 12m IMU, 13 signal processing device, 14 display device, 31 stillness determination unit, 32 Z direction determination unit, 33 Z direction determination unit, 34 mounting attitude determination unit, 35 display control unit, 71 initial screen, 81 manual mode start screen, 91 swing operation instruction screen, 101 wobble detection in progress screen, 111 wobble detection result screen, 200 computer, 202 ROM, 204 bus, 205 input / output interface, 206 input unit, 207 output unit, 208 memory unit, 209 communication unit, 210 drive, 211 removable media

Claims

1. A camera system comprising: an imaging camera having an inertial sensor; a sensor unit having an inertial sensor; and a signal processing device which detects wobble of the sensor unit attached to the imaging camera, wherein the signal processing device has: a first direction determination unit which determines a predetermined axial direction of the imaging camera from a sensor signal of the inertial sensor of the imaging camera; a second direction determination unit which determines an axial direction of the sensor unit which is the same as the predetermined axial direction from the sensor signal of the inertial sensor of the sensor unit; and an attitude determination unit which compares the axial directions of the imaging camera and the sensor unit to detect wobble.

2. The camera system according to claim 1, wherein the signal processing device further has a stillness determination unit which determines whether the imaging camera is still, and the attitude determination unit, when the imaging camera is still, compares the axial directions of the imaging camera and the sensor unit to detect wobble.

3. The camera system according to claim 2, wherein the stillness determination unit determines whether the imaging camera is still after a swing motion.

4. The camera system according to claim 1, wherein the attitude determination unit performs a comparison of the axial direction of the imaging camera and the sensor unit multiple times, and detects wobble based on the results of the multiple comparisons.

5. The camera system according to claim 1, wherein the signal processing device has a first mode in which the rattle detection is performed in the background, and a second mode in which the rattle detection is performed by instructing a user to perform an action.

6. The camera system according to claim 1, wherein the signal processing device further has a display control unit that displays the result of the rattle detection on a screen.

7. A signal processing device comprising a display control unit which controls the display of a screen for detecting wobble in the attachment of a sensor unit based on a sensor signal of an inertial sensor of an imaging camera and a sensor signal of an inertial sensor of a sensor unit attached to the imaging camera, wherein the display control unit controls the display of a first screen which indicates at least one of a swing motion or a stationary state of the imaging camera, and a second screen which displays a comparison result between the sensor signal of the inertial sensor of the imaging camera in the stationary state and the sensor signal of the inertial sensor of the sensor unit.

8. The signal processing device according to claim 7, wherein the display control unit repeatedly displays the first screen a plurality of times, and causes the user to execute a swing motion and a still state of the imaging camera a plurality of times.

9. The signal processing device according to claim 7, wherein the display control unit further displays a third screen that displays a result of the rattle detection based on one or more results of the comparison.

10. The signal processing device according to claim 9, wherein the display control unit displays, on the third screen, a portion where the comparison result is greater than a threshold value.

11. The signal processing device according to claim 9, wherein the display control section displays an average value of the comparison results on the third screen.

12. The signal processing device according to claim 7, wherein the display control unit, when one or more of the comparison results are greater than a threshold value, displays a fourth screen that prompts the user to adjust the rattle.

Images