Information processing device and its control method

Abstract

To achieve a convenient trace modification.SOLUTION: An information processing device includes acquisition means for acquiring trace information that is a chronological record of a plurality of operations on an imaging apparatus, display control means for displaying on a display unit a graph showing the movement of the imaging apparatus on the basis of the acquired trace information, reception means for accepting a move operation on any of a plurality of marks corresponding to the plurality of operations displayed by being superimposed on the displayed graph, and changing means for changing the acquired tracing information on the basis of the result of the movement operation accepted by the reception means.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to trace editing.

Background Art

[0002] In recent years, with the expansion of the video distribution market, the number of shooting systems that use cameras capable of remote control and video shooting via a network for shooting at weddings and lectures has been increasing. As remote operations, manual pan-tilt-zoom (PTZ) operations, preset operations, trace operations, etc. are possible.

[0003] Patent Document 1 discloses a system in which a lens and a camera are mounted on a pan-tilt head, the lens and the pan-tilt head are remotely controlled, and the operation state is continuously recorded as trajectory data. Also disclosed is a technique related to a trace operation in which the recorded trajectory data is called and the same operation as the recorded trajectory is reproduced. By the trace operation, the operation can be reproduced, and the convenience is improved when it is desired to make the camera perform the same movement again.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the above-described conventional technology, when memorizing an operation, if a wrong operation is performed, it has to be recorded again from the beginning. Therefore, even if there is a slight mistake in the operation, it has to be recorded again, which takes a lot of time and effort.

[0006] The present invention has been made in view of such problems, and an object thereof is to provide a technology that enables highly convenient trace correction for users. [Means for solving the problem]

[0007] To solve the above-mentioned problems, the information processing apparatus according to the present invention has the following configuration. That is, the information processing apparatus is Acquisition means for acquiring trace information that records multiple operations on an imaging device in chronological order, A display control means that displays a graph showing the movement of the imaging device on a display unit based on the trace information acquired by the acquisition means, A receiving means for receiving a movement operation for any of the multiple marks corresponding to the multiple operations that are displayed superimposed on the graph displayed on the display unit, A modification means that modifies trace information acquired by the acquisition means based on the result of a movement operation received by the reception means, Equipped with 、 The aforementioned operations are a plurality of operations relating to pan, tilt, and zoom on the imaging device, The aforementioned graph is a two-dimensional graph in which one of the pan, tilt, or zoom positions is taken as the first axis and time is taken as the second axis. The display control means is configured to switch between displaying a first graph in which the pan position is taken as the first axis and time as the second axis, and a second graph in which the tilt position is taken as the first axis and time as the second axis. The modification means is configured to operate by switching between a first setting, which outputs first trace information in which only the pan movement is changed when the receiving means receives a move operation for a mark superimposed on the first graph, and a second setting, which outputs second trace information in which both the pan movement and the tilt movement are changed when the receiving means receives a move operation for a mark superimposed on the first graph. . [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a technology that enables users to perform trace corrections in a highly convenient manner. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram shows the overall configuration of the network camera system. [Figure 2] This is a block diagram showing the functional configuration of the imaging device. [Figure 3] This is a block diagram showing the functional configuration of an information processing device. [Figure 4] This figure shows an example of an operation screen in a trace record. [Figure 5] This figure shows an example of a camera information table. [Figure 6] This figure shows an example of a trace information table. [Figure 7] This is a diagram showing an example of a trace correction screen (First Embodiment). [Figure 8] This is a diagram showing a time - pan graph corresponding to trace information. [Figure 9] This is a diagram explaining the modifiable range of a mark. [Figure 10] This is a diagram explaining a change operation. [Figure 11] This is a diagram explaining trace information after a change operation. [Figure 12] This is a diagram explaining the modifiable range of a mark. [Figure 13] This is a diagram explaining a change operation. [Figure 14] This is a diagram explaining trace information after a change operation. [Figure 15] This is a diagram showing a time - tilt graph corresponding to trace information. [Figure 16] This is a diagram showing an interlock setting on a trace correction screen. [Figure 17] This is a diagram explaining the trace information (pan) after change during interlock. [Figure 18] This is a diagram explaining the trace information (tilt) after change during interlock. <able>0000097This is a flowchart of trace correction processing in the first embodiment. [Figure 20] This is a diagram showing an example of a trace correction screen (Second Embodiment). [Figure 21] This is a diagram showing a 3D graph of time - pan - tilt. [Figure 22] This is a flowchart of trace correction processing (Second Embodiment). [Figure 23] This is a block diagram showing the hardware configuration of an information processing apparatus.

Modes for Carrying Out the Invention

[0010] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0011] (First Embodiment) As a first embodiment of the information processing device according to the present invention, a network camera system will be described below as an example.

[0012] <System Configuration and Equipment Configuration> Figure 1 shows the overall configuration of the network camera system. The network camera system includes a camera 101, a terminal 102, a network 103, and a joystick 104. Camera 101 is a camera called a network camera or IP streaming camera. Camera 101 and terminal 102 are connected in a way that allows them to communicate with each other via the network 103. As will be described later, terminal 102 can display video from camera 101 and control commands via the network 103.

[0013] Camera 101 transmits captured video data and information about itself via the network 103 in response to requests from terminal 102. Alternatively, it may proactively transmit data to a pre-connected terminal 102. Terminal 102 is a typical client terminal with a display, such as a PC or tablet / smartphone.

[0014] Network 103 is a network connecting camera 101 and terminal 102. Network 103 is implemented using multiple routers, switches, cables, etc., that comply with a communication standard such as Ethernet®. Network 103 may also be implemented using the Internet, a wired LAN (Local Area Network), a wireless LAN (Wireless LAN), a WAN (Wide Area Network), etc.

[0015] The joystick 104 connects to the terminal 102 via USB, Bluetooth (registered trademark), etc., and is used to remotely control the camera 101. The joystick 104 can be used in particular to achieve smooth PTZ control, which is difficult to do with a GUI on an application.

[0016] Figure 2 is a block diagram showing the functional configuration of the imaging device that constitutes the camera 101. The imaging device includes a system control unit 201, an imaging unit 202, an image processing unit 203, a storage unit 210, a program memory 211, and a communication unit 220. It also has a lens drive unit 204, an imaging angle control unit 205 for zoom control, and a focus control unit 206 for focus control. Furthermore, it has a pan drive unit 207, a tilt drive unit 208, and a pan / tilt control unit 209 for controlling the imaging direction (pan direction and tilt direction).

[0017] The system control unit 201 is a control unit that issues processing instructions to each part of the imaging device. The system control unit 201 analyzes the camera control commands transmitted from the terminal 102 and received by the communication unit 220, and performs processing according to the camera control commands.

[0018] Here, camera control commands can be broadly classified into request commands, which request the acquisition of video data and setting values, and setting commands, which request the setting of setting values. Primarily, the system control unit 201 receives video data request commands from terminal 102 and distributes the video data generated by the image processing unit 203 via the communication unit 220. In addition, the system control unit 201 receives request commands from terminal 102 for zoom, focus, and pan / tilt setting values ​​for camera 101.

[0019] Upon receiving a command requesting a setting value, the system control unit 201 reads the setting values ​​from the image processing unit 203, the imaging angle control unit 205, the focus control unit 206, and the pan / tilt control unit 209, respectively, and distributes them to the terminal 102 via the communication unit 220. Here, the zoom, focus, and pan / tilt setting values ​​include not only the current values ​​but also related information such as the range of values ​​that can be set on the camera. Furthermore, when the system control unit 201 receives a command to set the zoom, focus, and pan / tilt setting values, it instructs each functional unit in the terminal 102 to perform control based on those setting values.

[0020] The image processing unit 203, the imaging angle control unit 205, the focus control unit 206, and the pan / tilt control unit 209 control the imaging unit 202, the lens drive unit 204, the pan drive unit 207, and the tilt drive unit 208 based on commands. As a result, the zoom, focus, and pan / tilt settings set on the terminal 102 are reflected in the camera 101.

[0021] The imaging unit 202 includes a lens and an image sensor, and performs imaging of the subject and conversion to an electrical signal. The image processing unit 203 performs predetermined image processing, resolution conversion processing, and compression encoding processing on the signal captured and photoelectrically converted by the imaging unit 202 to generate video data. The video data captured by the imaging unit 202 is distributed to the terminal 102 via the network 103 by the communication unit 220.

[0022] The lens drive unit 204 consists of a drive system for the focus lens and zoom lens and a motor that is the drive source for them, and is controlled by the image angle of view control unit 205 and the focus control unit 206. The image angle of view control unit 205 commands the lens drive unit 204 to change the zoom lens position based on the zoom setting value output from the system control unit 201. Here, the image angle of view control unit 205 manages the focal length as the zoom setting value. The focus control unit 206 commands the lens drive unit 204 to change the focus lens position based on the focus setting value output from the system control unit 201. The focus position is changed as the position of the focus lens is changed.

[0023] The pan drive unit 207 consists of a mechanical drive system and a motor that performs panning, and its operation is controlled by the pan-tilt control unit 209. The tilt drive unit 208 consists of a mechanical drive system and a motor that performs tilting, and its operation is controlled by the pan-tilt control unit 209. Based on the pan-tilt setting values ​​output from the system control unit 201, the pan-tilt control unit 209 commands the pan drive unit 207 and the tilt drive unit 208 to change the pan-tilt.

[0024] The memory unit 210 stores video data in internal storage and external storage. The program memory 211 is a memory that stores the camera control program, and the system control unit 201 performs various operations based on the camera control program stored in the program memory 211.

[0025] The communication unit 220 distributes video data to the terminal 102 via the network 103. The communication unit 220 also receives camera control commands transmitted from the terminal 102 and outputs them to the system control unit 201. Then, following the instructions of the system control unit 201, it distributes a response to the terminal 102. The camera control commands transmitted from the terminal 102 mainly include commands requesting video data, commands requesting and setting values ​​for the zoom, focus, and pan / tilt of the camera 101. For example, pan can be set within the range of +180° to -180°, and tilt within the range of +90° to -20°.

[0026] Figure 3 is a block diagram showing the functional configuration of the information processing device that constitutes terminal 102. Terminal 102 can display video streamed from camera 101 and remotely control camera 101. The information processing device includes a system control unit 301, a communication unit 302, a display unit 303, an input unit 304, a program memory 305, and a storage unit 306.

[0027] The system control unit 301 is a control unit that issues processing instructions to each part of the information processing device. The system control unit 301 generates camera control commands in response to user operations on the graphical user interface (GUI) and transmits them to the camera 101 via the communication unit 302. By transmitting camera control commands to the camera 101, it is possible to perform operations such as pan, tilt, and zoom (PTZ).

[0028] Furthermore, as will be described later using Figure 4, it is possible to record traces (PTZ operation history) and to play back traces based on the recorded traces. When the communication unit 302 receives a response from the camera 101, the system control unit 301 analyzes the response and performs processing according to the response. The system control unit 301 also instructs the display unit 303 to reflect the video data received from the camera 101 via the communication unit 302, as well as the zoom, focus, and pan / tilt settings.

[0029] The communications unit 302 transmits camera control commands and receives various data distributed from the camera 101. Camera control commands include commands to request live video, commands to request and set the zoom, focus, and pan / tilt settings of the camera 101.

[0030] The display unit 303 uses a liquid crystal display or the like to display video data acquired from the camera 101 and a GUI for controlling the camera. The input unit 304 uses a device such as a keyboard, mouse, or joystick, and the user of the terminal 102 operates the GUI displayed on the display unit 303 via the input unit 304. The display unit 303 and the input unit 304 may also be implemented as an integrated touch panel. Furthermore, while it is desirable to use the joystick 104 for smooth PTZ operation, other means may also be used.

[0031] The program memory 305 is a memory that stores terminal control programs, and the system control unit 301 executes various operations based on the terminal control programs stored in the program memory 305. The storage unit 306 stores camera information, trace information, etc., which will be described later, in the internal storage and external storage.

[0032] Figure 23 is a block diagram showing the hardware configuration of the information processing device 2300 that constitutes terminal 102. As mentioned above, terminal 102 is a computer device such as a PC or tablet / smartphone.

[0033] The CPU (Central Processing Unit) 2301 controls the entire computer device 2300. The ROM (Read Only Memory) 2302 stores programs and parameters that do not require modification. The RAM (Random Access Memory) 2303 temporarily stores programs and data supplied from external devices. The external storage device 2304 is a storage device such as a hard disk or memory card. The external storage device 2304 may include optical discs such as flexible disks (FD) and compact disks (CD) that can be removed from the computer device 2300, magnetic or optical cards, IC cards, and memory cards.

[0034] The input device interface 2305 is an interface with the input device 2309. The input device 2309 is, for example, a pointing device or keyboard that receives user input and inputs data. The output device interface 2306 is an interface with the monitor 2310 for displaying data held by the computer device 2300 or data supplied by it.

[0035] The communication interface 2307 is a communication interface for connecting to a network line 2311, such as the Internet, and connects to the NW camera 112 via the network line 2311, for example. The NW camera is an imaging device that generates video data through shooting. The system bus 2308 is a transmission path that enables communication between each unit in the computer device 2300. The various processes described later with reference to the functional units shown in Figure 3 and Figure 19 are realized by the CPU 2301 reading and executing programs stored in a computer-readable storage medium such as ROM 2302.

[0036] <Trace Record> Figure 4 shows an example of an operation screen for trace recording. Specifically, it shows an example of a GUI displayed on the display unit 303, which is operated by the user when recording a trace in the information processing device in the first embodiment, which corresponds to terminal 102. The screen includes camera ID 401, live video 402, trace ID 403, recording start button 404, and recording stop button 405.

[0037] Camera ID 401 is a UI component that selects a camera ID to identify the camera to be recorded. For example, multiple camera IDs corresponding to multiple cameras are pre-listed, and the user can select the camera to be recorded for tracing. The information of the listed camera IDs is stored in the camera information table in the storage unit 306, which will be described later using Figure 5.

[0038] Live video 402 is a UI component that displays video data streamed from camera 101, which was selected with camera ID 401.

[0039] Trace ID 403 is a UI component that specifies a trace ID that uniquely identifies the trace to be recorded. The trace ID is an ID that uniquely identifies the trace to be recorded in the camera selected by camera ID 401. The information of the trace IDs listed in Trace ID 403 is stored in the trace information table in the storage unit 306, which will be described later using Figure 6. Furthermore, in Trace ID 403, instead of selecting an existing trace ID, it is possible to select to add a trace ID (not shown). If you select to add a trace ID, an ID will be issued that is unique within the camera selected by camera ID 401 and will be displayed in Trace ID 403 in the selected state. Note that Trace ID 403 may have a fixed list of, for example, 10 IDs pre-registered. In this case, the maximum number of IDs listed will be 10, including unregistered ones.

[0040] The recording start button 404 is a UI component that, when pressed by the user, starts the recording of the trace. Before pressing the recording start button 404, the user needs to prepare the camera to the desired starting state for recording camera operations. For example, the PTZ position should be moved to the starting position of the trace. Once the trace recording starts, the user operates the camera to be recorded by using the joystick 104 or similar while viewing the live video 402. Trace information is stored in the program memory 305 as it occurs.

[0041] The stop recording button 405 is a UI component that, when pressed by the user, stops the recording of the trace. When the stop recording button 405 is pressed, the trace information stored in the program memory 305 is stored in the trace information table in the storage unit 306, which will be described later using Figure 6. Note that when the upper limit of the trace recording time or the upper limit of resources such as the recording unit 306 is reached, the trace recording may be stopped without pressing the stop recording button 405, and the trace information up to that point may be stored in the trace information table.

[0042] In this embodiment, trace information is stored on the client terminal 102, but it is also possible to store the trace information on the camera 101 and have the terminal 102 read that information via communication.

[0043] Figure 5 shows an example of a camera information table. The camera information table stores camera ID information for one or more cameras 101 used by terminal 102. The camera information table stores camera information for all cameras that are controlled by the terminal control program. Each record in the camera information table includes a camera ID 501 and an IPv4 address 502.

[0044] Camera ID 501 is an ID that uniquely identifies the camera stored in the camera information table, and is the same as the ID displayed for camera ID 401.

[0045] IPv4 address 502 is the IPv4 address set on the camera and is used when terminal 102 accesses the camera. Note that it is not limited to IPv4 addresses; any address that uniquely identifies and allows access to the camera could be used. For example, an IPv6 address, MAC address, or serial number could also be used.

[0046] There are various methods for registering camera information in the camera information table, and any method is acceptable. For example, you can read a file containing camera information, register camera information via a GUI, or use a terminal control program to search for and register cameras on the network. Depending on the registration method, you may need to provide camera information such as MAC address, username, and password, but a detailed explanation will be omitted.

[0047] Figure 6 shows an example of a trace information table. The trace information table stores one or more trace information entries used by one or more cameras 101 used in terminal 102. For example, trace information can be registered via the GUI described using Figure 4. Each record in the trace information table includes a camera ID 601, a trace ID 602, and a trace operation 603 which is the actual trace.

[0048] Camera ID 601 is the ID corresponding to Camera ID 501 and is recorded to identify the camera being traced. Trace ID 602 ​​is an ID that uniquely identifies the trace to be recorded within a single camera identified by Camera ID 601, and is the ID corresponding to Trace 403.

[0049] Trace operation 603 is a record of camera operations and is referenced during trace playback. Here, an example of recording in JSON (JavaScript Object Notation) format is shown, but other formats are also acceptable. In JSON format, objects can be represented by placing a colon ":" between the "name (key)" and the "value".

[0050] In the example in Figure 6, lines 1-2 record the start and end information of the trace, and lines 3-7 record the command information during the trace period.

[0051] In lines 1 and 2, the name "start" indicates the start of the trace. The corresponding value represents the camera state at the start of the trace. The camera state indicates a pan value of "5(°)", a tilt value of "2(°)", and a zoom value of "60". The name "end" indicates the end of the trace. The corresponding value represents the camera state at the end of the trace. The camera state indicates that the elapsed time required for the trace was "2050(ms)", the pan value was "5(°)", the tilt value was "2(°)", and the zoom value was "60". Here, the elapsed time required for the trace is the time elapsed from when the trace recording actually started.

[0052] In lines 3-7, the name "elapsed time" represents the elapsed time from the start of the trace recording until the setting command (zoom, pan / tilt setting instruction) was sent. Here, the elapsed time is recorded chronologically, ranging from 10ms to 1950ms. The name "command" is information that uniquely represents the setting command sent to camera 101. Here, "pan.speed" represents the pan speed and "pan.dir" represents the pan direction, specifying the pan speed and pan direction as setting values. Similarly, the setting values ​​for tilt and zoom are also specified. Note that settings other than pan, tilt, and zoom can also be set.

[0053] <Trace Correction Screen> Figure 7 shows an example of a trace correction screen in the first embodiment. Specifically, it shows an example of a GUI displayed on the display unit 303 by display control when a user corrects trace information using terminal 102. The trace correction screen includes camera ID 701, display parameter selection tabs 702-704, image display area 705, trace ID 706, linkage checkbox 711, edit save button 712, and undo button 713.

[0054] Camera ID 701 is the camera ID that identifies the camera whose trace is to be modified. The user can select the camera from the listed camera IDs where the trace to be modified is stored. Trace ID 706 is the ID that identifies the trace to be modified within the camera selected by Camera ID 701.

[0055] Image display area 705 displays a two-dimensional graph showing the relationship between time and pan (or tilt, zoom) values ​​in the trace selected by trace ID 706. Whether to display the pan, tilt, or zoom graph in image display area 705 is selected by clicking one of tabs 702-704. The selected tab is highlighted with inverted black and white display. In Figure 7, tab 702 is selected and displayed with inverted black and white display, so image display area 705 shows a graph of pan values ​​against time.

[0056] When the linked checkbox 711 is checked, any modification made in one tab will be linked and modified in other tabs. For example, if the operation is modified on the Time-Pan screen, the tilt operation will also be modified. If it is not checked, even if the pan operation is modified, the tilt operation will not change. In other words, the linked checkbox 711 is for accepting user settings regarding whether or not to perform linked modifications. This operation will be explained later. Note that the description to the right of the linked checkbox 711 will say "Modifications will also link to tilt" when tab 702 is selected, and "Modifications will also link to pan" when tab 703 is selected. Also, when tab 704 is selected, the linked checkbox 711 and the description to the right are removed (not shown), and when the zoom operation is modified, other movements (pan, tilt) will not be changed.

[0057] When the edit / save button 712 is pressed, the changes made are actually saved on the image display area 705. When the undo button 713 is pressed, the changes are cleared from the image display area 705 and the display returns to the saved data. When the close button 714 is pressed, this screen disappears and the editing process ends.

[0058] Figure 8 shows a time-pan graph corresponding to trace information. Specifically, it shows details of what is displayed in the image display area 705 when trace 830 is loaded and tab 702 (i.e., pan correction) is selected.

[0059] The image display area 705 displays graph 800, and the marks 810, 811, 812, and 813, which are superimposed on graph 800, correspond to commands 820, 821, 822, and 823. Furthermore, the shape of marks 810, 811, 812, and 813 changes depending on the type of command. Specifically, marks 810 and 813, which correspond to the instruction to start movement in a stopped state, are displayed as triangular marks. Mark 811, which corresponds to the instruction to change speed in a moving state, is displayed as a square mark. In addition, mark 812, which corresponds to the instruction to stop movement in a moving state, is displayed as a circular mark. The position of marks 810, 811, 812, and 813 within the graph can be moved by the user using mouse (or touch) operation.

[0060] <Change operation for the mark corresponding to the speed change instruction> Figure 9 illustrates the modifiable range of mark 911. Specifically, it shows the range in which mark 911 (corresponding to mark 811) can be moved (i.e., the panning speed can be changed) relative to graph 800 in Figure 8. Figure 9 shows that the range in which mark 911 can be moved is within the shaded area 920. The shaded area 920 is a range calculated considering the positions of two marks adjacent to each other in time (marks 910 and 912) and the maximum panning speed. In the trace correction screen, the user is restricted from moving mark 911 outside the shaded area 920.

[0061] Figure 10 illustrates the modification operation. Specifically, it shows how the panning speed can be changed by moving mark 1011 (corresponding to mark 811) in Figure 10. The user selects mark 1011 by clicking the mouse button, for example, and then drags it to the position of mark 1031 (moving while holding down the mouse button). At this time, mark 1011 in its original position may be displayed transparently. Mark 1031 is a square mark, just like the original mark 1011, and its position changes according to the mouse operation while being dragged.

[0062] Figure 11 illustrates the trace information after a modification operation. Specifically, it shows the state after the mouse operation (drop operation) has ended in Figure 10. Here, the drop operation (ending the mouse button press) has been performed at the position of mark 1131, and the destination position has been determined.

[0063] Specifically, the position of mark 811 (time "4", pan position "45") has been changed to the position of mark 1131 (time "2", pan position "105"). Therefore, the pan speed from the position of mark 1130 (time "0", pan position "5") to the position of mark 1131 is (105-5) / 2 = 50. Also, the pan speed from the position of mark 1131 to the position of mark 1132 (time "8", pan position "165") is (165-105) / (8-2) = 10.

[0064] This modification (mark movement) changes command 820 in Figure 8 to command 1120 in Figure 11. Additionally, command 821 is changed to commands 1121 and 1122. In other words, since the linked checkbox 711 is unchecked, the modification only changes the movement in the pan direction.

[0065] Specifically, command 821 specifies "pan.speed=30&pan.dir=right&tilt.speed=8&tilt.dir=up". The command related to the pan direction has been changed to "pan.speed=10&pan.dir=right" in command 1121 as a result of the modification operation described above. On the other hand, the command related to the tilt direction remains unchanged in command 1122 as "tilt.speed=8&tilt.dir=up".

[0066] Furthermore, command 820 specifies "pan.speed=10&pan.dir=right&tilt.speed=4&tilt.dir=up". The command related to the pan direction has been changed to "pan.speed=50&pan.dir=right" in command 1120 as a result of the above modification operation. On the other hand, the command related to the tilt direction remains unchanged in command 1120 as "tilt.speed=4&tilt.dir=up".

[0067] Note that the above changes will not be saved as is, but will be saved when the edit / save button 712 in Figure 7 is pressed while in the state shown in Figure 11. Also, if the undo button 713 is pressed while in the state shown in Figure 11, the edits made for the above changes will be discarded, and the display will return to the original trace view (Figure 8).

[0068] <Change operation for the mark corresponding to the stop instruction> Figure 12 illustrates the modifiable range of mark 1212. Specifically, it shows the range in which mark 1212 (corresponding to mark 812) can be moved (i.e., the stopping time of pan movement can be changed) relative to graph 800 in Figure 8. As mentioned above, since mark 812 corresponds to the stop command, the stopping position (pan angle) is not changed, and only the time direction can be changed. In Figure 12, the range in which mark 1212 can be moved is the area enclosed by dashed lines 1240 and 1241. The time of dashed line 1240 is calculated considering the position of mark 1211 and the maximum speed of pan movement. The time of dashed line 1241 is calculated considering the position of mark 1213 and the margin time required to ensure that the operation starts at that position. In the trace correction screen, the user is restricted from moving mark 1212 outside the area enclosed by dashed lines 1240 and 1241.

[0069] In the above explanation, the modification operation on mark 1212 was described as being possible within a range where the stopping position (pan angle) does not change, but the stopping position may also be made changeable. For example, the range of changeable parts may be the same as that for the modification operation on the mark corresponding to the speed change instruction (Figure 9) described above. Alternatively, it may be configured to be movable along the line segment between mark 1211 and mark 1212.

[0070] Figure 13 illustrates the modification operation. Specifically, it shows how to change the panning speed by moving mark 1312 (corresponding to mark 812) in Figure 13. The user selects mark 1312 by clicking the mouse button, for example, and then drags it to the position of mark 1322 (moving while holding down the mouse button). At this time, mark 1312 in its original position may be displayed transparently. Mark 1322 is a circular mark, just like the original mark 1312, and its position changes according to the mouse operation while being dragged.

[0071] Figure 14 illustrates the trace information after a modification operation. Specifically, it shows the state after the mouse operation (drop operation) has ended in Figure 13. Here, the drop operation (ending the mouse button press) has been performed at the position of mark 1422, and the destination position has been determined.

[0072] Specifically, the position of mark 1312 (time "8", pan position "165") has been changed to the position of mark 1422 (time "10", pan position "165"). Therefore, the pan speed from the position of mark 1421 (time "4", pan position "45") to the position of mark 1422 is (165-45) / (10-4)=20.

[0073] This modification operation (mark movement) changes command 821 in Figure 8 to commands 1431 and 1432 in Figure 14, and command 822 to command 1433. In other words, since the linked checkbox 711 is not checked here, the modification is made so that only the movement in the pan direction changes.

[0074] Specifically, in the stop command 822, "elapsed time" is specified as 800, but due to the modification operation described above, it is changed to "elapsed time": 1000 in command 1433.

[0075] Furthermore, command 821 specifies "pan.speed=30&pan.dir=right&tilt.speed=8&tilt.dir=up". The command related to the pan direction has been changed to "pan.speed=20&pan.dir=right" in command 1431 due to the above modification operation. On the other hand, since the movement in the tilt direction does not change, it is necessary to stop only the tilt direction at the time "8 (seconds)". For this reason, command 1432 has been added, which stops only the tilt direction (speed zero) "tilt.speed=0&tilt.dir=up".

[0076] <Tilt direction movement when not linked> Figure 15 shows a time-tilt graph corresponding to trace information. Specifically, it shows the graph displayed in the image display area 705 when tilt (i.e., tab 703) is selected in the trace correction screen of Figure 7. As mentioned above, since the linkage checkbox 711 is not checked, even if the pan direction changes from the state in Figure 8 to the state in Figure 11 or Figure 14, the tilt direction operation does not change. In other words, when not linked, even if a change operation is performed in the pan direction, the pan direction graph remains unchanged as shown in Figure 15.

[0077] <Linking Settings> Next, we will explain the operation when the linked checkbox 711 in Figure 7 is checked.

[0078] Figure 16 shows the linkage settings in the trace correction screen. Specifically, it is an example screen when the linkage checkbox 711 is checked in the trace correction screen of Figure 7. That is, in screen 1601, the linkage checkbox 1611 is checked.

[0079] Figure 17 illustrates the trace information (pan) after a speed change operation during synchronized operation. Specifically, it shows the graph (time-pan) and trace when the pan direction changes from the state in Figure 8 to the state in Figure 11. The graph is the same as in Figure 11, but some of the commands in trace 1726 differ from trace 1126 when the system is not synchronized.

[0080] Figure 18 illustrates the trace information (tilt) after a speed change operation during synchronization. Specifically, it shows the graph (time-tilt) and trace when the pan direction changes from the state in Figure 8 to the state in Figure 11. The graph in Figure 18 shows the graph displayed in the image display area 1605 when tilt (i.e., tab 1603) is selected in the trace correction screen of Figure 16. Note that trace 1825 is a re-posting of trace 1726 and has the same content. It can be seen that the graph in Figure 18 is different from the graph (time-tilt) shown in Figure 15 when synchronization is not enabled. Specifically, the position of mark 1811 (time "2", tilt position "32") is different from the position of the corresponding mark in Figure 15 (time "4", tilt position "18"). In other words, in trace 1825 after pan operation correction when synchronization is enabled, unlike trace 1436 after pan operation correction when synchronization is not enabled, changes are made in both the pan direction and the tilt direction.

[0081] The specific linking method in the examples of Figures 17 and 18 above will now be explained. The user edits the position of mark 1711 (time "2", pan position "105"). The two marks adjacent in time are mark 1712 (time "8", pan position "165") and mark 1710 (time "0", pan position "5").

[0082] This section explains how editing the position of mark 1711 affects the trace in the tilt direction. Marks 1810, 1811, and 1812 in the tilt direction (Figure 18) correspond to marks 1710, 1711, and 1712 in the pan direction (Figure 17). Since the user has not edited marks 1712 and 1710, the positions of the corresponding marks 1812 and 1810 remain unchanged. On the other hand, since editing has been done to mark 1711, the position (time and pan position) of mark 1811 is changed in conjunction with mark 1711.

[0083] In the pan direction, there is a 160° pan movement in 8 seconds between mark 1710 and mark 1712. However, there is a 100° pan movement in 2 seconds between mark 1710 and mark 1711. In other words, the proportion of the pan movement from mark 1710 to mark 1711 to the total pan movement from mark 1710 to mark 1712 is 100 / 160 = 5 / 8.

[0084] On the other hand, in the tilt direction, there is a 48° tilt movement in 8 seconds between mark 1810 and mark 1812. Therefore, the amount of tilt movement from mark 1810 to mark 1811 is calculated as 5 / 8 of 48° (i.e., 48 * 5 / 8 = 30), and as a result, the tilt position of mark 1811 is corrected to "32". The command in trace 1825 is changed accordingly.

[0085] Specifically, the tilt speed in command 1820 is calculated by subtracting the tilt position of mark 1810 (2°) from the tilt position of mark 1811 (32°), and then dividing this value by the time between the two marks (2 seconds), resulting in (32-2) / 2 = 15. As a result, the tilt direction part of command 1820 is changed to "tilt.speed=15&tilt.dir=up". Similarly, the tilt speed in command 1821 is calculated by subtracting the tilt position of mark 1811 (32°) from the tilt position of mark 1812 (50°), and then dividing this value by the time between the two marks (6 seconds), resulting in (50-32) / 6 = 3. As a result, the tilt direction part of command 1821 is changed to "tilt.speed=3&tilt.dir=up".

[0086] <Device Operation> Figure 19 is a flowchart of the trace correction process in the first embodiment. This process starts, for example, when the user operates terminal 102 and displays the trace correction screen (Figure 7). The process ends when the user presses the close button 714.

[0087] In step S1900, terminal 102 reads the trace command described in Figure 6. In step S1901, terminal 102 sets the trace correction mode to pan-time mode. This corresponds to the state where tab 702 in Figure 7 is selected and the time-pan graph is displayed in the image display area 705.

[0088] In step S1902, terminal 102 turns on the linkage flag in response to the user checking the linkage checkbox 711. When the linkage setting is ON, as explained in Figures 17 and 18, when the user corrects the pan direction, the tilt direction operation is also changed accordingly.

[0089] In step S1903, terminal 102 begins accepting user edit operations on the trace correction screen. Specifically, if the linkage setting is OFF, the trace correction screen shown in Figure 7 is displayed; if the linkage setting is ON, the trace correction screen shown in Figure 16 is displayed, and user operations begin to be accepted. Once any user operation is detected, the process proceeds to S1904.

[0090] In step S1904, terminal 102 determines whether or not the user has performed a move operation on the mark. Whether or not a move operation has been performed is determined, for example, by determining whether or not the user has performed a drag-and-drop operation on the mark. If it is determined that a move operation has been performed, proceed to S1905; otherwise, proceed to S1908.

[0091] In step S1905, terminal 102 determines whether the linkage flag is ON or OFF. This determines whether the check for 711 in Figure 7, as explained earlier, has been performed. If the linkage flag is ON, proceed to S1906; otherwise, proceed to S1907.

[0092] In step S1906, terminal 102 performs a trace change that links pan and tilt and returns to S1904. A trace change that links pan and tilt is a trace change as explained in Figures 17 and 18. On the other hand, in step S1907, terminal 102 performs a trace change that does not link pan and tilt and returns to S1904. A trace change that does not link pan and tilt is a trace change as explained in Figures 10 and 11.

[0093] In step S1908, terminal 102 determines whether or not a display mode change operation has been performed by the user. This is determined by whether or not a tab corresponding to a mode other than the currently displayed mode has been selected from tabs 702 to 704 in Figure 7. If it is determined that a display mode change operation has been performed, the terminal proceeds to S1909; otherwise, it proceeds to S1910.

[0094] In step S1909, terminal 102 modifies the display of the trace correction screen. For example, the explanatory text for the linked checkbox 711 and the graph displayed in the image display area 705 are changed. After that, the process returns to S1904.

[0095] In step S1910, terminal 102 determines whether or not a save command has been issued by the user. For example, this is determined by whether or not the edit / save button 712 has been pressed. If it is determined that a save command has been issued, the process proceeds to S1911; otherwise, it proceeds to S1912.

[0096] In step S1911, terminal 102 saves the current (modified) trace settings. For example, if a save command is issued in the state shown in Figure 11, trace 1126 is saved. Then, the process returns to S1904.

[0097] In step S1912, terminal 102 determines whether or not the user has given an undo command. For example, this is determined by whether or not the undo button 713 has been pressed. If it is determined that an undo command has been given, the process proceeds to S1913; otherwise, it returns to S1904.

[0098] In step S1913, terminal 102 restores the trace information to its most recently saved state (the trace information read in S1900 or the trace saved in S1911). This restores the trace to the state shown in Figure 8, even if editing as shown in Figure 11 has been performed (if S1911 has not been executed). After that, the process returns to S1904.

[0099] As described above, in the first embodiment, the user can easily edit trace information. Therefore, the user can make desired corrections to already saved traces. For example, even if the user has saved an incorrect operation as a trace, they can easily make the desired correction. In other words, the user does not need to perform the cumbersome task of operating the camera again and saving the trace again, improving user convenience.

[0100] In the above explanation, it was assumed that the graph would accept user input for modification and confirmation. However, for example, when saving the trace as explained in Figure 4, the video at that time may be saved, and the system may be configured to accept modification operations by allowing the user to confirm a preview display that modifies the saved video according to the trace information.

[0101] Furthermore, in the above explanation, for the sake of simplicity, the motor for pan-tilt drive was described as one that performs the desired drive instantaneously according to the command. However, generally, some time lag or drive lag may occur when the motor starts and stops. Therefore, the trace correction process described above may be performed taking these lags into consideration.

[0102] (Second Embodiment) In the second embodiment, the basic configuration is the same as in the first embodiment (Figures 1 to 6). Below, only the parts that differ from the first embodiment will be described.

[0103] <Trace Correction Screen> Figure 20 shows an example of the trace correction screen in the second embodiment. Specifically, it shows an example of the GUI displayed on the display unit 303 when a user corrects trace information using terminal 102. The trace correction screen includes camera ID 2001, display parameter selection tabs 2002 and 2004, image display area 2005, trace ID 2006, edit save button 2012, and undo button 2013. Except for display parameter selection tabs 2002 and 2004 and image display area 2005, the screen is the same as in the first embodiment (Figure 7), so a description is omitted.

[0104] Image display area 2005 displays a 3D graph showing the relationship between time and pan / tilt (or zoom) values ​​in the trace selected by trace ID 2006. The type of graph displayed in image display area 2005 is selected by clicking either tab 2002 or 2004. The selected tab is highlighted with a black and white inversion. In Figure 20, tab 2002 is selected and displayed with a black and white inversion, and image display area 2005 displays a 3D graph of pan / tilt values ​​against time. Note that when tab 2002 is selected, it displays a 3D time-pan / tilt display, and when tab 2004 is selected, it displays a 2D time-zoom display. In other words, in time-pan / tilt mode, the user can simultaneously edit the pan and tilt movements.

[0105] Figure 21 shows a 3D graph of time-pan-tilt corresponding to trace information. Direction 2120 represents pan, direction 2121 represents tilt, and direction 2122 represents time. Mark 2110 corresponds to the start of the trace, mark 2111 corresponds to a speed change, and mark 2112 corresponds to a stop. Marks 2110, 2111, etc., can be moved within the 3D graph by the user using mouse (or touch) operation, and the trace can be modified by this operation. In other words, the position of the marks moved by the operation is converted into a command from the position on the time axis, pan axis, and tilt axis.

[0106] Below, we consider mark 2110 (time "0", pan position "5", tilt position "2") and mark 2111 (time "5", pan position "50", tilt position "7"). The pan movement speed is (50-5) / 5=9, and the tilt movement speed is (7-2) / 5=1. Therefore, the first command in the trace is "pan.speed=9&pan.dir=right&tilt.speed=1&tilt.dir=up". Although the operation method is different, the concept of how to calculate is basically the same as in the first embodiment, so a detailed explanation is omitted.

[0107] <Device Operation> Figure 22 is a flowchart of the trace correction process in the second embodiment. This process starts, for example, when the user operates terminal 102 and displays the trace correction screen (Figure 20). The process ends when the user presses the close button 2014.

[0108] In step S2200, terminal 102 reads the trace command described in Figure 6. In step S2201, terminal 102 sets the trace correction mode to time-pan-tilt mode. This is when tab 2002 in Figure 20 is selected and the time-pan-tilt 3D graph is displayed in the image display area 2005.

[0109] In step S2202, terminal 102 begins accepting user edit operations on the trace correction screen. Specifically, it begins accepting user operations on the trace correction screen shown in Figure 20. Once any user operation is detected, the process proceeds to S2203.

[0110] In step S2203, terminal 102 determines whether or not the user has moved the mark. Whether or not a move operation has occurred is determined, for example, by whether or not the user has performed a drag-and-drop operation on the mark. If it is determined that a move operation has occurred, proceed to S2204; otherwise, proceed to S2205.

[0111] In step S2204, terminal 102 makes trace changes corresponding to the movement process and returns to S2203.

[0112] In step S2205, terminal 102 determines whether or not a display mode change operation has been performed by the user. This is determined by whether or not a tab corresponding to a mode other than the currently displayed mode has been selected from tabs 2002 and 2004 in Figure 20. If it is determined that a display mode change operation has been performed, the terminal proceeds to S2206; otherwise, it proceeds to S2208.

[0113] In step S2206, terminal 102 modifies the display of the trace correction screen. For example, the graphs displayed in the image display area 2005 are changed. After that, the process returns to S2203.

[0114] In step S2207, terminal 102 determines whether or not a save command has been issued by the user. For example, this determination is made by whether or not the edit / save button 2012 has been pressed. If it is determined that a save command has been issued, the terminal proceeds to S2008; otherwise, it proceeds to S2009.

[0115] In step S2208, terminal 102 saves the current (modified) trace settings. For example, if a save command is issued in the state shown in Figure 21, the corresponding trace is saved. Then, the process returns to S2203.

[0116] In step S2209, terminal 102 determines whether or not the user has given an undo command. For example, this is determined by whether or not the undo button 2013 was pressed. If it is determined that an undo command has been given, the process proceeds to S2210; otherwise, it returns to S2203.

[0117] In step S2210, terminal 102 restores the trace information to its most recently saved state (the trace read in S1900 or the trace saved in S2208). This restores the trace to the state shown in Figure 21, even if editing has been performed (if S2208 has not been executed). The process then returns to S2203.

[0118] As described above, in the second embodiment, the user can easily edit the trace information. In particular, by editing in 3D view, it becomes possible to edit the pan and tilt movements simultaneously.

[0119] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0120] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0121] 101 Camera; 102 Terminal; 103 Network; 104 Joystick

Claims

1. Acquisition means for acquiring trace information that records multiple operations on an imaging device in chronological order, A display control means that displays a graph showing the movement of the imaging device on a display unit based on the trace information acquired by the acquisition means, A receiving means for receiving a movement operation for any of the multiple marks corresponding to the multiple operations that are displayed superimposed on the graph displayed on the display unit, A modification means that modifies trace information acquired by the acquisition means based on the result of a movement operation received by the reception means, Equipped with, The aforementioned operations are a plurality of operations relating to pan, tilt, and zoom on the imaging device, The aforementioned graph is a two-dimensional graph in which one of the pan, tilt, or zoom positions is taken as the first axis and time is taken as the second axis. The display control means is configured to switch between displaying a first graph in which the pan position is taken as the first axis and time as the second axis, and a second graph in which the tilt position is taken as the first axis and time as the second axis. The modification means is configured to switch between a first setting, which outputs first trace information in which only the pan movement is changed when the receiving means receives a move operation for a mark superimposed on the first graph, and a second setting, which outputs second trace information in which both the pan movement and the tilt movement are changed when the receiving means receives a move operation for a mark superimposed on the first graph. An information processing device characterized by the following:

2. The system further includes a setting acceptance means for accepting the specification of either the first setting or the second setting. The information processing apparatus according to feature 1.

3. Acquisition means for acquiring trace information which records multiple operations on an imaging device in chronological order, A display control means that displays a graph showing the movement of the imaging device on a display unit based on the trace information acquired by the acquisition means, A receiving means for receiving a movement operation for any of the multiple marks corresponding to the multiple operations that are displayed superimposed on the graph displayed on the display unit, A modification means that modifies trace information acquired by the acquisition means based on the result of a movement operation received by the reception means, Equipped with, The aforementioned operations are a plurality of operations relating to pan, tilt, and zoom on the imaging device, The graph described above is a three-dimensional graph in which the pan position is plotted on the first axis, the tilt position on the second axis, and time on the third axis. An information processing device characterized by the following:

4. Each of the operations included in the aforementioned plurality of operations corresponds to one of the following: starting movement, changing speed, or stopping movement, relating to at least one of the pan, tilt, or zoom movements of the imaging device. The information processing apparatus according to any one of claims 1 to 3.

5. The system further comprises control means for controlling the movement of the imaging device based on the trace information modified by the modification means. The information processing apparatus according to any one of claims 1 to 4.

6. A method for controlling an information processing device, An acquisition process to obtain trace information that records multiple operations on the imaging device in chronological order, A display control step, in which a graph showing the movement of the imaging device is displayed on the display unit based on the trace information acquired in the acquisition step, A receiving step of receiving a movement operation for any of the multiple marks corresponding to the multiple operations that are displayed superimposed on the graph displayed on the display unit, A modification step which modifies the trace information acquired in the acquisition step based on the result of the movement operation received in the reception step, Includes, The aforementioned operations are a plurality of operations relating to pan, tilt, and zoom on the imaging device, The aforementioned graph is a two-dimensional graph in which one of the pan, tilt, or zoom positions is taken as the first axis and time is taken as the second axis. In the aforementioned display control process, a first graph in which the pan position is taken as the first axis and time as the second axis, and a second graph in which the tilt position is taken as the first axis and time as the second axis are switched and displayed. In the modification process, the system operates by switching between a first setting, which outputs first trace information in which only the pan movement is changed when a movement operation is received for a mark superimposed on the first graph by the reception process, and a second setting, which outputs second trace information in which both the pan movement and the tilt movement are changed when a movement operation is received for a mark superimposed on the first graph by the reception process. A control method characterized by the following:

7. A control method for an information processing device, An acquisition process to obtain trace information that records multiple operations on the imaging device in chronological order, A display control step, in which a graph showing the movement of the imaging device is displayed on the display unit based on the trace information acquired in the acquisition step, A receiving step of receiving a movement operation for any of the multiple marks corresponding to the multiple operations that are displayed superimposed on the graph displayed on the display unit, A modification step which modifies the trace information acquired in the acquisition step based on the result of the movement operation received in the reception step, Includes, The aforementioned operations are a plurality of operations relating to pan, tilt, and zoom on the imaging device, The graph described above is a three-dimensional graph in which the pan position is plotted on the first axis, the tilt position on the second axis, and time on the third axis. A control method characterized by the following:

8. A program for causing a computer to execute the control method described in claim 6 or 7.

Images