Information processing equipment, information processing system, information processing method

Abstract

To notify a camera operator of information regarding the change status of the settings, when the camera operator performs an operation to control an imaging device from a remote location.SOLUTION: An information processing device for setting an external device that controls an imaging device that tracks a subject according to a target position of the subject to be tracked in a captured image accepts a setting of a first target position of the subject to be tracked in an image captured by the imaging device, acquires a second target position of the subject to be tracked, which is held in the external device, from the external device, and controls so as to display information regarding the setting of the target position on a display unit. On the basis of the first target position and the second target position, information regarding the setting of the target position is controlled so as to be displayed in a first aspect or a second aspect different from the first aspect.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus for operating an imaging device remotely.

Background Art

[0002] In recent years, in the field of video production, a method called remote production, which performs imaging and production by controlling an imaging device from a remote location without going to the site, has been spreading.

[0003] According to this method, a camera operator can remotely perform imaging-related settings on an imaging device or a control device that controls the imaging device by operating a controller or the like via a network.

[0004] Patent Document 1 discloses a method of controlling an imaging device in which a camera operator designates the arrangement (composition) of a subject in an image captured by the imaging device via a network.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, the longer the distance between the controller operated by the camera operator and the imaging device, the longer the communication delay time of the network. As a result, the reflection of the result of changing the imaging-related settings for the imaging device or the control device that controls the imaging device is delayed.

[0007] In such a situation, the camera operator may misrecognize that the imaging-related settings have not been changed and may perform excessive operations.

[0008] Therefore, the present invention aims to provide an information processing device that can notify a camera operator of information regarding changes in settings when the camera operator performs operations to control the imaging device from a remote location. [Means for solving the problem]

[0009] One aspect of the present invention is, An information processing device, The system includes: a receiving means for receiving the setting of a first target position of a subject to be tracked in an image captured by an imaging device; a communication means for receiving position information indicating a second target position of the subject in the tracking, which is held by a control device that controls the imaging device that tracks the subject in the image, and transmitting position information indicating the first target position to the control device; and a display control means for controlling the display unit to display a mark indicating the first target position. A calculation means for calculating the communication time between the control device and the information processing device, The display control means displays the mark regardless of the distance between the first target position and the second target position, and controls the display of the mark in a first manner or a second manner different from the first manner, depending on the distance between the first target position and the second target position. Then, depending on the communication time, the display control means decides whether to enable control to change the display manner of the mark. do. [Effects of the Invention]

[0010] According to the present invention, when a camera operator performs operations to control the imaging device from a remote location, it becomes possible to notify the camera operator of information regarding the status of setting changes. [Brief explanation of the drawing]

[0011] [Figure 1] This is a diagram showing the remote control system of the first embodiment. [Figure 2] This is a block diagram showing the configuration of the camera 100, edge AI device 200, and PC 300 in the first embodiment. [Figure 3] (a) is a flowchart showing the control operation of the AI ​​device 200 in the first embodiment. (b) is a flowchart showing the operation of the camera 100 in the first embodiment. [Figure 4] This diagram shows the operation flow when an operation is performed on the PC300 of the first embodiment. [Figure 5] (a) is a diagram showing an example of a target position mark in the PC300 of the first embodiment. (b) is a diagram showing an example of a target position mark in the PC300 of the first embodiment. [Figure 6] (a) is a diagram showing the operation flow of the PC300 in the first embodiment. (b) is a diagram showing the operation flow of the PC300 in the first embodiment. [Figure 7] This figure shows how the display mode of the target position mark in the first embodiment is changed. [Figure 8] This is a diagram showing a remote control system according to a second embodiment. [Figure 9] This is a block diagram showing the configuration of each device in the second embodiment. [Figure 10] (a) is a diagram showing the operation flow of the controller 800 in the second embodiment. (b) is a diagram showing the operation flow of the controller 800 in the second embodiment. [Figure 11] (a) is a diagram showing an example of a target position mark in the controller 800 of the second embodiment. (b) is a diagram showing an example of a target position mark in the controller 800 of the second embodiment. (c) is a diagram showing an example of a target position mark in the controller 800 of the second embodiment. [Figure 12] This figure shows the operation flow for changing the display mode according to the communication latency in the third embodiment. [Figure 13] This figure shows the operation flow for changing the display mode according to the communication latency in the fourth embodiment. [Figure 14] This figure shows how the display mode of the target position mark changes in the fourth embodiment. [Modes for carrying out the invention]

[0012] <First Embodiment> (Configuration of the remote control system) FIG. 1 is a diagram showing an example of the configuration of a remote control system for implementing the present invention.

[0013] The remote control system is composed of a camera 100, an edge AI device 200, a PC 300 (Personal Computer), and a controller 400. The camera 100 is connected to the edge AI device 200 via a LAN (Local Area Network) 500, but is not limited thereto. For example, it may be connected by SDI (Serial Digital Interface) or HDMI (High-Definition Multimedia Interface) (registered trademark). Similarly, the controller 400 is connected to the PC 300 (information processing device) via a LAN 600, but is not limited thereto. For example, it may be connected by SDI, HDMI, or UVC (Universal Serial Bus Video Class).

[0014] Furthermore, the LAN 500 and the LAN 600 are connected via the Internet 700, and each device is configured to be able to communicate with each other.

[0015] Also, the camera 100 and the edge AI device 200 are installed at physically close positions, and the PC 300 is installed at a remote location away from them.

[0016] Therefore, communication between the PC 300 (information processing device) and the camera 100 or the edge AI device 200 is in a situation where latency, that is, propagation delay of communication data (communication delay) occurs. Also, communication between the camera 100 and the edge AI device 200 is in a situation where no communication delay occurs.

[0017] In this embodiment, the camera 100 and the edge AI device 200 are described as separate components, but they may also be configured as an integrated unit. Similarly, the PC 300 and the controller 400 are described as separate components, but they may also be configured as an integrated unit.

[0018] (Camera 100 configuration) Figure 2 is a block diagram showing the configuration of the camera 100, edge AI device 200, and PC 300 according to this embodiment.

[0019] The camera 100 includes a CPU 101, RAM 102, ROM 103, imaging unit 104, drive unit 105, imaging optical system 106, image processing unit 107, image output interface (I / F) 108, network I / F 109, and an internal bus 110 that enables interoperability.

[0020] Camera 100 can output images captured by the imaging unit 104 to external devices via various networks and image cables (not shown). Camera 100 also has a tracking function that automatically tracks a subject.

[0021] The CPU 101 is a central processing unit that provides overall control over the camera 100.

[0022] RAM102 is a storage device such as DRAM (Dynamic Random Access Memory) that temporarily stores computer programs executed by CPU101. RAM102 also provides a work area where the OS, various programs, and various data are loaded, and which CPU101 uses to execute processing. It is also used as a workspace for the OS and various programs.

[0023] ROM103 is a non-volatile storage device, such as flash memory, HDD, SSD (Solid State Drive), or SD card. ROM103 is used as a persistent storage area for the CPU101 to store the OS, various programs, and data, as well as for short-term data storage.

[0024] The imaging unit 104 includes a drive unit 105, an imaging optical system 106, and an image processing unit 107, which will be described later. The imaging direction can be changed by driving the pan-tilt (PT) mechanism with the drive unit 105. In addition, zoom (Z) operation to change the imaging angle of view is possible by driving the imaging optical system 106 in the optical axis direction with the drive unit 105.

[0025] The drive unit 105 drives the imaging unit 104 to achieve pan, tilt, and zoom values ​​corresponding to the PTZ drive commands from the CPU 101.

[0026] The imaging optical system 106 is a lens that focuses light from the subject onto the imaging surface of the image sensor, and is composed of, for example, a zoom lens, a focus lens, and an image stabilization lens. Furthermore, the zoom value can be changed by driving the imaging optical system 106 in the direction of the optical axis. The image sensor (not shown) of the imaging unit 104 captures an image of the subject and generates an image. The image sensor (not shown) of the imaging unit 104 converts the light from the subject focused by the imaging optical system 106 into an electrical signal for each pixel. Furthermore, an amplifier (not shown) amplifies the electrical signal converted by the image sensor and outputs it to the image processing unit 107, which will be described later. The image sensor is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0027] The image processing unit 107 converts the electrical signal amplified by the amplifier (not shown) into a predetermined format, compresses it as needed, and transfers it to the RAM 102. The image processing unit 107 also performs image quality adjustment when acquiring captured images and cropping to extract only a predetermined area of ​​the image data. These operations are performed according to instructions received from external devices such as the edge AI device 200 and PC 300 via the network interface 109.

[0028] In this embodiment, the imaging unit 104 is described as being integrated with the drive unit 105, but the imaging unit 104 and the drive unit 105 may be separate and detachable, like a camera mounted on a tripod head to change the imaging direction. Furthermore, although the imaging unit 104 and the imaging optical system 106 are described as being integrated, the imaging unit 104 and the imaging optical system 106 may be separate and detachable, like a camera with interchangeable lenses.

[0029] The image output I / F108 is an interface for outputting captured images to an external source and consists of SDI (Serial Digital Interface) and HDMI (High-Definition Multimedia Interface) (registered trademark). Here, the image output I / F108 is connected to the image input I / F208 of the edge AI device 200, which will be described later.

[0030] Network I / F 109 is an interface for connecting to the aforementioned LAN 500 and is responsible for communication with external devices such as the edge AI device 200 and PC 300 via a communication medium such as Ethernet (registered trademark). Here, it was explained that remote camera control of camera 100 is performed via network I / F 105, but it may also be performed via another interface such as a serial communication interface (not shown).

[0031] (Configuration of 200 edge AI devices) Figure 2 is a block diagram showing the configuration of the camera 100, edge AI device 200, and PC 300 according to this embodiment.

[0032] The edge AI device 200 includes a CPU 201, RAM 202, ROM 203, detection unit 204, user input I / F 205, image output I / F 206, network I / F 207, image input I / F 208, and an internal bus 209 that enables mutual communication.

[0033] CPU201 is a central processing unit that provides overall control over the edge AI device 200.

[0034] RAM202 is a storage device such as DRAM that temporarily stores computer programs executed by CPU201. RAM202 also provides a work area where the OS, various programs, and various data are loaded, and which CPU201 uses to execute processing. It is also used as a workspace for the OS and various programs.

[0035] ROM203 is a non-volatile storage device, such as flash memory, HDD, SSD (Solid State Drive), or SD card. ROM203 is used as persistent storage for the OS, various programs, and data, including programs used by the CPU201 to control the edge AI device 200. It is also used as storage for short-term data.

[0036] The detection unit 204 estimates the location and presence of a subject from an image received from an image input I / F 208, etc., which will be described later. The detection unit 204 is composed of a computing device specialized for image processing and detection processing, such as a GPU (Graphics Processing Unit). The detection unit 207 also has a trained model created using machine learning methods such as deep learning. While a GPU is generally effective for training, equivalent functionality may be achieved with a reconfigurable logic circuit such as an FPGA (Field Programmable Gate Array). Alternatively, the CPU 201 may handle the processing of the detection unit 204. In this embodiment, the subject is described as a person, but it is not limited to this. For example, it may be an object other than a person, such as a car or a dog.

[0037] The User Input I / F205 (reception mechanism) is an interface for connecting to a mouse, keyboard, and other input devices, and is configured using USB (Universal Serial Bus), etc.

[0038] Image output I / F206 is an interface for outputting settings information screens and other information from edge AI devices.

[0039] Network I / F207 is an interface for connecting to the aforementioned LAN500 and is responsible for communication with external devices such as camera 100 and PC300 via a communication medium such as Ethernet.

[0040] The image input interface 208 is an interface for receiving images captured from the aforementioned camera 100, and consists of SDI and HDMI.

[0041] The edge AI device 200 performs AI-based detection on the captured images transmitted from the camera 100 to detect a subject. Furthermore, based on the results of detecting a subject in the acquired captured images, the edge AI device 200 transmits control signals via the LAN 500 to track the subject, such as changing the imaging direction of the camera 100.

[0042] Furthermore, the control functions performed by the edge AI device 200 described above may be integrated into the camera 100.

[0043] (PC300 configuration) Figure 2 is a block diagram showing the configuration of the camera 100, edge AI device 200, and PC 300 according to this embodiment.

[0044] The PC300 includes a CPU 301, RAM 302, ROM 303, network I / F 304, display unit 305, operation unit 306, device I / F 307, and an internal bus 308 that enables interoperability.

[0045] The CPU 301 controls the entire PC 300. It also controls the display of the display unit 305, which will be described later (display control).

[0046] RAM302 is a storage device such as DRAM that temporarily stores computer programs executed by CPU301. RAM302 also provides a work area where the OS, various programs, and various data are loaded, and which the CPU301 uses to execute processing. It is also used as a workspace for the OS and various programs.

[0047] ROM303 is a non-volatile storage device such as flash memory, HDD, SSD, or SD card. ROM303 is used as a persistent storage area for the CPU301 to store programs for controlling the PC300, including the OS, various programs, and various data. It is also used as a storage area for short-term data. In this embodiment, ROM303 is assumed to be an SSD, but it is not limited to this.

[0048] Network I / F304 is an interface for connecting to the aforementioned LAN600 and is responsible for communication with external devices such as camera 100 and edge AI device 200 via a communication medium such as Ethernet. Here, communication refers to sending and receiving control commands to camera 100 and PC300, and receiving captured images from camera 100.

[0049] The display unit 305 is a display unit for displaying images captured by the camera 100 and the settings screen of the PC 300. In this embodiment, an example is shown where the PC 300 has a display unit, but the configuration is not limited to this. For example, there may be a configuration in which a display monitor and a controller exist, each solely for displaying captured images.

[0050] The control unit 306 is an interface for receiving user input to the PC 300, and examples include buttons, dials, joysticks, and touch panels. For example, the control unit 306 receives input of instructions for panning and tilting the camera 100 based on user input.

[0051] Device I / F307 is an interface for connecting to an input device and is configured as a USB (Universal Serial Bus) or the like. In this embodiment, it is used as the connection interface to the controller 400 shown in Figure 1.

[0052] The PC300 acquires and displays captured images output from the camera 100 via the internet 700. Furthermore, it receives setting operations related to subject tracking from the user and transmits control instructions for the edge AI device 200 via the internet 700 based on the operation content.

[0053] Settings related to subject tracking specifically include determining the target to track, the field of view and direction of the captured image, and the position of the subject in the captured image.

[0054] Settings can be configured by directly operating the PC300 or by operating it via the controller 400 shown in Figure 1. Specifically, settings related to subject tracking can be configured using the operation unit 306 of the PC300, or using the operation unit 401 of the controller 400, which will be described later.

[0055] In this embodiment, the user can input tracking settings, including various settings for tracking and imaging a target, by operating the controller 400, which will be described later. The controller 400 will also be described as outputting the input tracking settings to the PC 300.

[0056] The method for selecting a person to track is not limited to any particular method, as long as it allows selecting the face of the person to be tracked from one or more detected faces. For example, the PC300 or edge AI device 200 may select the face located at the coordinates closest to the target position in the captured image, or it may select a person that has been set in advance as the target of tracking. Furthermore, although this embodiment describes the case where the target of tracking is a person, the target of tracking is not limited to a person, and the following description is also applicable when the target of tracking is something other than a person.

[0057] Next, I will explain the controller 400.

[0058] The controller 400 is a controller equipped with an operation unit 401, and is a user interface that allows operation of the camera 100 using, for example, a keyboard or a multi-directional input stick controller. In this embodiment, the user uses the joystick of the controller 400 to set up subject tracking. The controller 400 is also connected to the device I / F 307 and transmits information corresponding to the user's operation to the PC 300.

[0059] In this embodiment, the controller 400 and the PC 300 are described as separate entities, but the operation unit 306 of the PC 300 may also handle the processing of the controller 400.

[0060] With the above configuration, the remote control system allows the camera 100 to automatically track a subject using the edge AI device 200. Furthermore, the user can check the captured image displayed on the PC 300 and operate the PC 300 to make various settings related to subject tracking.

[0061] This section outlines the operation of the remote control system (information processing system). Details of the basic operation of each device and the characteristic operation of this invention will be described later.

[0062] <Basic Operation Description of Each Device> (The camera 100 tracks the subject detected by the edge AI device 200.) Next, we will explain the operation of each device in relation to the basic operation of the remote control system mentioned above, referring to the flowchart.

[0063] First, we will explain the operation in which the camera 100 tracks a subject detected by the edge AI device 200 using Figure 3.

[0064] Figure 3(a) shows the operation flow of the edge AI device 200, and Figure 3(b) shows the operation flow of the camera 100. These figures illustrate a series of steps in which the edge AI device 200 controls the camera 100 according to the subject position detected from the captured image and the tracking settings. In this embodiment, the tracking settings refer to information about the subject to be tracked, selected from one or more subjects, and the placement of the subject to be tracked in the captured image (target position).

[0065] First, we will explain the operation of the edge AI device 200 based on the flow chart in Figure 3(a).

[0066] Figure 3(a) shows a loop process in which the edge AI device 200 receives sequentially transmitted images from the camera 100 via the image input I / F 208, identifies the position of the subject within the captured image, and controls the device to automatically track the target subject. This control flow starts when the edge AI device 200 connects to the camera 100 via LAN 500 and acquires the captured images from the camera 100.

[0067] In step S101, the CPU 201 of the edge AI device 200 acquires the image captured by the camera 100 via the image input I / F 208. The CPU 201 writes the acquired image to RAM 202 and proceeds to step S202.

[0068] The captured images are transmitted sequentially from the image output I / F 104 of the camera 100 at a predetermined frame rate, and the CPU 201 sequentially writes the received captured images to the RAM 202. Alternatively, the captured images may be received via the network I / F 207 and loaded into the internal RAM 202.

[0069] In step S102, the detection unit 207 detects the position of the subject in the captured image. The CPU 201 reads the captured image from the RAM 202 and inputs it to the detection unit 207. The detection unit 207 writes output data, including the detected position of the subject, back to the RAM 202, and then proceeds to step S103.

[0070] Specifically, the detection unit 207 receives the captured image as input data and outputs the type of tracking target object such as a person, the positional information of the subject in the captured image, and a score indicating the likelihood of detection as output data.

[0071] In step S103, the edge AI device 200 obtains tracking settings from the PC 300 via the network 700 and stores them in the RAM 202. Here, tracking settings refer to information about the subject to be tracked from among the one or more subjects detected in step S102, as well as information about the target position. For simplicity, the subject to be tracked will be referred to as the subject below.

[0072] In step S104, the CPU 201 determines whether the subject's position in the captured image matches the target position, based on the tracking settings stored in RAM 202 and the subject's position (target position) in the captured image detected in step S102. While information regarding the target position is acquired in step S103, it is not limited to this. It may also be loaded into RAM 202 by the CPU 102 when the edge AI device 200 is started. For example, the subject's position (target position) in the captured image at startup may be automatically set to the center of the captured image, or the position at the time the edge AI device 200 was last shut down.

[0073] If CPU201 determines that the target position and the detected subject's position are the same, it skips steps S105 through S107 and returns to the beginning of the loop. If CPU201 determines that the target position and the detected subject's position are not the same, it proceeds to step S105.

[0074] In step S105, the CPU 201 calculates the difference between the position of the subject detected in step S102 and the target position. Furthermore, it calculates at least one angular velocity (control value) in the pan / tilt (and zoom) direction corresponding to the calculated difference so that the position of the subject matches the target position. The calculated result is written to the RAM 202, and the process proceeds to step S105.

[0075] Regarding the calculation of angular velocity, one method involves multiplying the distance (the difference between the coordinate values ​​in the pan and tilt directions) by a predetermined coefficient, and then determining the drive direction based on whether the calculated value is positive or negative. Since these techniques are well-known, a more detailed explanation will be omitted.

[0076] Furthermore, the method of tracking a subject using the technology described above, which calculates and controls the direction and speed of movement, is just one example. For example, any method can be used to track a subject, such as a method that calculates the position to drive toward a target position and tracks the subject.

[0077] In step S106, the CPU 201 converts the result calculated in step S105 into a control command according to a predetermined protocol for controlling the camera 100, writes it to RAM 202, and then proceeds to step S107.

[0078] In step S107, the CPU 201 reads the control command that was written to RAM 202 in step S106. Furthermore, the CPU 201 sends the control command to camera 100 via network I / F 207 and returns to the beginning of the loop process.

[0079] Next, based on Figure 3(b), the control flow of the camera 100 when it receives a control command from the edge AI device 200 will be explained. This control flow starts when the camera 100 connects to the edge AI device 200 via LAN 500 and then receives a control command from the edge AI device 200.

[0080] In step S201, CPU101 receives a control command via network I / F105.

[0081] In step S202, the CPU 101 is notified that it has received communication data, in this case a control command, from the network I / F 105. Upon receiving this notification, the CPU 101 reads a value corresponding to the angular velocity in the pan-tilt direction from the control command written by the network I / F 105 to RAM 102. After the CPU 101 reads the control command, the process transitions to step S203. At this point, it is sufficient to read at least one value of the angular velocity in the pan-tilt (and zoom) direction, either for the pan direction or the tilt (and zoom) direction.

[0082] In step S203, the CPU 101 calculates drive parameters for panning and tilting in the desired direction and at the desired speed based on the values ​​read in step S202, and then proceeds to step S204. Specifically, these are parameters for controlling the respective motors (not shown) for the pan direction and tilt direction included in the drive unit 106. The operation amounts included in the received control command may be converted into drive parameters by referring to a conversion table previously stored in the ROM 103.

[0083] In step S204, the CPU 101 controls the drive unit 106 based on the drive parameters calculated in step SS203, and then proceeds to step S205. The drive unit 106 drives the camera 100 based on these parameters, causing the camera 100 to change its imaging direction, i.e., to perform pan and tilt operations.

[0084] As described above, the camera 100 can change the imaging direction based on control commands acquired from the edge AI device 200, according to the control flow shown in Figure 3(b). In this embodiment, the case in which control commands related to the angular velocity in the pan and tilt directions are acquired from the edge AI device 200 has been described, but the acquired control commands are not limited to these. For example, the zoom value may be changed by the drive unit 106 driving the imaging optical system 105 based on a control command related to the zoom value.

[0085] Next, the operation flow when PC300 configures subject tracking settings for the edge AI device 200 will be explained using Figures 4 and 5(a)(b).

[0086] Figure 4 shows the operation flow of the PC300. Here, we will explain the flow in which the PC300 transmits coordinate information (target position) indicating where to place the subject in the captured image when tracking a subject. This control flow starts when it receives an operation from the user.

[0087] In step S301, the CPU 301 of the PC 300 detects the joystick operation of the controller 400 operated by the user via the device I / F 307, and obtains the direction and amount of joystick operation from the device I / F 307. After detecting and obtaining the user's operation, the system proceeds to step S302.

[0088] In this embodiment, the joystick is described as having an analog output specification that uses voltages output from variable resistors provided for the pan and tilt directions, respectively, but it is not limited to this. For example, operation may be performed by clicking or touching the GUI (Graphical User Interface) displayed on the display unit using a mouse or keyboard. The controller 400 quantizes the voltage input from the joystick using an A / D conversion unit (not shown) at a predetermined sampling period, and outputs the result as information corresponding to the manipulated variable to the device I / F 307.

[0089] In the A / D conversion unit, values ​​within a predetermined range, such as 0 to 1023, are quantized as components in the pan and tilt directions, respectively, according to the manipulated variable.

[0090] In step S302, the CPU 301 calculates the amount of movement of the coordinate position (target position) that indicates where the subject should be placed in the captured image, based on the joystick operation direction and quantized operation amount information acquired in step S301. The calculated result is written to RAM 302, and the process proceeds to step S303.

[0091] Here, we will explain the target position when tracking a subject using Figure 5(a).

[0092] Figure 5(a) shows the captured image displayed on the display unit 305. Figure 5(a) shows the subject indicated by the diagonal lines and the target position mark 900, which is the position (target position) of the subject in the captured image. The PC 300 periodically receives the captured image from the camera 100 via the network I / F 304 and can display the image on the display unit 305. The target position mark 900 is a mark indicating the target position when tracking the subject as described above. When the subject moves, the edge AI device 200 sends control information to the camera 100 so that the subject is positioned at the target position mark 900 on the captured image, and the drive unit 106 of the camera 100 is driven. In Figure 5(a), the target position mark 900 is set in the center of the image.

[0093] In this embodiment, the user can move the target position mark 900 by operating the joystick of the controller 400. Specifically, the PC 300 calculates the amount of movement of the target position mark 900 from the amount and direction of joystick operation. The PC 300 transmits the calculated result to the edge AI device 200, and the edge AI device 200 controls the imaging direction of the camera 100 so that even if the target position mark 900 is moved, the subject remains at the position of the target position mark 900.

[0094] Figure 5(b) shows the captured image after moving the target position mark 900 to the left from the position shown in Figure 5(a). When the target position mark 900, which is set in the center of the image in Figure 5(a), is moved to the left, the imaging direction of the camera 100 is changed so that the subject is positioned to the left in the captured image.

[0095] In step S303, the CPU 301 calculates coordinate information for the target position based on the value calculated in step S302 and transmits this coordinate information to the edge AI device 200 via the network I / F 105. After the PC 300 transmits the coordinate information to the edge AI device 200, the process proceeds to step S304. The coordinate information transmitted here is stored in the RAM 202 of the edge AI device 200 and is referenced by the CPU 201 as the target position in step S103 of Figure 3(a). In this embodiment, it is explained that the coordinate information for the target position is calculated and transmitted continuously while the user is operating the joystick. This makes it possible to adjust the target position for subject tracking to the edge AI device 200.

[0096] In this embodiment, since the edge AI device 200 and the PC 300 are under conditions where communication latency (communication delay) occurs, after the transmission of the coordinate information described above, transmission and reception will occur after a delay time due to the communication delay.

[0097] Therefore, the display unit 305 of the PC300 is updated immediately in response to the operation of the controller 400. However, after the operation, there is a lag (delay time) until the setting (target position) for the tracking position of the edge AI device 200 is updated and the imaging direction of the camera 100 begins to change.

[0098] In step S304, the CPU 301 calculates the coordinate values ​​on which the target position mark will be superimposed on the captured image based on the values ​​calculated in step S302, and instructs the display unit 305 to superimpose the target position mark on the coordinate values ​​on the captured image and display it.

[0099] In step S305, the CPU301 writes the coordinate values ​​superimposed with the target position marker sent to the edge AI device to the RAM302, updating the target position information where the subject is located.

[0100] As described above, the operation flow of PC300 and camera100 makes it possible to change the tracking position of camera100 according to user operation.

[0101] Note that the processes from steps S301 to S305 are executed periodically while the program is running. For example, they may be executed periodically at intervals of several tens of milliseconds.

[0102] In this case, the user responds immediately to the joystick operation of the controller 400, and the target position is transmitted and the display is updated sequentially based on the direction and amount of operation.

[0103] <Explanation of how to change the display of the target location marker> As shown in Figures 5(a) and 5(b), changing the target position causes a communication delay between the edge AI device 200 and the PC 300. Therefore, the transmission of new target position information and the change in imaging direction occur after a delay due to the communication delay. When such a delay occurs, the user may mistakenly believe that the target position has not been changed, potentially leading to excessive operation.

[0104] In situations where such a delay occurs, the display unit 305 indicates that the user has sent a target position change instruction to the edge AI device 200. In other words, in this embodiment, the display mode is changed based on the target position information stored in the RAM 302 of the PC 300 and the target position information stored in the RAM 202 of the edge AI device 200. Specifically, in this embodiment, if the difference between the target position coordinates in the RAM 302 of the PC 300 and the target position coordinates in the RAM 202 of the edge AI device 200 is greater than a set threshold, the color of the target position mark is changed to red; otherwise, the color of the target position mark is changed to white. Here, if the difference between the target position coordinates in the RAM 302 of the PC 300 and the target position coordinates in the RAM 202 of the edge AI device 200 is less than or equal to a predetermined threshold, the target positions are considered to match. In this embodiment, as long as the user continues to operate the controller 400, the CPU 301 continues to update the target position coordinates held by the RAM 302. Similarly, the CPU 301 continues to send the updated target position coordinates to the edge AI device 200, and when the edge AI device 200 obtains the target position coordinates, it updates the target position coordinates held in the RAM 202. Also, if the edge AI device 200 can communicate with the PC 300 via the network 700, it continues to send the target position coordinates held in the RAM 202 to the PC 300.

[0105] In this embodiment, the set threshold is described as being set to a circular area centered on the target position stored in RAM 302. For example, the threshold is set to an area of ​​2% of the display unit 305, centered on the target position stored in RAM 302. The color of the target position mark is changed when the coordinates of the target position stored in RAM 202 fall within an area of ​​2% of the display unit 305 from the coordinates of the target position stored in RAM 302, but this is not limited to this. For example, the threshold may be set centered on the coordinates of the target position stored in RAM 202. Furthermore, the threshold does not have to be based on the area of ​​the display unit 305; the threshold may be changed according to the actual distance of the imaged object by decreasing the threshold according to the zoom value.

[0106] Using Figures 6(a) and 6(b), we will explain the operation of changing the display mode according to the coordinates of the target position stored in RAM 202 when displaying information about the target position (target position mark 900) on the display unit 305.

[0107] First, let's explain the operation flow of the edge AI device 200 shown in Figure 6(a). Figure 6(a) shows the processing flow in which the edge AI device 200 sends the coordinates of the target position, which are stored in RAM 202, to PC 300. The CPU 201 within the edge AI device 200 reads the target position stored in RAM 202 within the edge AI device 200, which is referenced in step S103 of Figure 3(a), and sends it to PC 300 at all times.

[0108] In step S401, the CPU 201 reads the coordinates of the target position for subject tracking, which are stored in the RAM 202.

[0109] In step S402, the CPU201 transmits the coordinates of the target position read out above to the PC300 via the network I / F207.

[0110] In step S403, CPU201 sleeps for a predetermined time and then returns to step S401.

[0111] If the PC300 and the edge AI device 200 can communicate via the network 700, the above process is repeated to sequentially transmit the coordinates of the target position held by the RAM 202 to the PC300.

[0112] Next, we will explain the operation flow of the PC300 when changing the display mode of the target position information (target position mark 900) displayed on the display unit 305 according to the coordinates of the target position held by the RAM202. Figure 6(b) shows the process by which the PC300 receives the coordinates of the target position from the edge AI device 200 and displays the result of the target position operation performed by the user via the controller 400 on the display unit 305 of the PC300.

[0113] In step S501, the CPU 301 receives the coordinates of the target position transmitted by the edge AI device 200 in step S402 of Figure 6(b) via the network I / F 304, and then proceeds to step S502.

[0114] In step S502, the CPU 301 reads the coordinates of the target position that were written to the RAM 302 in step S305 of Figure 4, and then proceeds to step S503. That is, it reads the coordinates of the target position held by the RAM 302.

[0115] In step S503, the CPU 301 compares the target position coordinates held by the RAM 202 received in step S501 with the target position coordinates held by the RAM 302 in step S502. If they match, the process proceeds to step S504, where the CPU 301 instructs the display unit 305 to display the target position mark in white. If they do not match, the process proceeds to step S505, where the CPU 301 instructs the display unit 305 to display the target position mark in red.

[0116] Here, we will explain how the image displayed on the display unit 305 looks when the display color of the target position mark is changed, using Figures 7(a) to (e).

[0117] Figures 7(a) to 7(e) show the contents displayed on the display unit 305 in chronological order, and Figures 7(b) to 7(c) are explained assuming that the user is operating the controller 400 to change the target position.

[0118] Figure 7(a) shows the display content when the subject is captured at the position of the target position mark 900, and Figure 7(b) shows the display content when the user operates the target position mark 900. Furthermore, Figure 7(c) shows the display content while the camera's imaging direction is being changed in accordance with the movement of the target position mark 900, and Figure 7(d) shows the display content when the operation to change the target position mark 900 is completed. Figure 7(e) shows the display content when the change in imaging direction is completed and the subject is captured again at the position of the target position mark.

[0119] Figures 7(a) and 7(e) show the display content when the target position information matches in step S504, with the target position mark 900a displayed in white.

[0120] Figures 7(b), 7(c), and 7(d) show the display content when the target position information does not match in step S505, and the target position mark 900b is displayed in red (as a diagonal line pattern in the diagram).

[0121] While they do not match, PC300 can understand that the edge AI device 200 is unable to acquire the target position due to communication latency.

[0122] The aim of the process described above is to utilize the difference in target position caused by this communication latency to change the color of the target position marker during that time. By making the user aware that communication latency is occurring, the process helps to prevent them from perceiving that their actions have not been registered.

[0123] On the other hand, under conditions where communication latency is sufficiently low, or when the target location is slightly changed, the time during which the target location coordinates do not match is reduced.

[0124] If the process of changing the color of the target position marker 900 is performed while the conditions are not matched, the color of the target position marker 900 may rapidly alternate and appear to be flashing.

[0125] Therefore, in this embodiment, the control is set to change the color of the target position mark when the difference between the target position coordinates held by RAM302 and the target position coordinates held by RAM202 is greater than a predetermined threshold, but it is not limited to this. For example, the control may be changed to change the color of the target position mark only when the situation of not matching continues for a predetermined time. Alternatively, the control may be set without a predetermined threshold, and change the color of the target position mark only when the target position coordinates held by RAM302 and the target position coordinates held by RAM202 perfectly match.

[0126] In step S506, CPU301 sleeps for a predetermined time and then returns to the beginning of the process.

[0127] As described above, by changing the color of the target location marker according to the comparison result of the target location coordinates, users can visually identify when communication latency is occurring.

[0128] In this embodiment, we have described the situation in which communication latency occurs as an example, but it is also effective when applied to operating systems where latency is high due to processes other than communication, such as edge device detection processing or image signal processing.

[0129] In this embodiment, an example of changing the display method of the target position mark was shown as changing the color, but it is also possible to use other methods to provide notification, such as changing the shape or fill pattern.

[0130] Furthermore, in addition to target position markers, it is also possible to inform the user of communication latency through methods such as communication latency time information, warning messages, voice messages, and controller vibration. In this embodiment, an example was described in which the edge AI device 200 controls the camera 100 to track a subject and transmits target position information to the PC 300, but this is not the only example. For example, the functions of these edge AI devices 200 may be configured to be built into the camera 100 itself.

[0131] <Second Embodiment> In the first embodiment, an example was described in which the display mode of the target position mark is changed using an edge AI device 200 and a PC 300.

[0132] In this embodiment, unlike the first embodiment, a controller 800 equipped with a joystick and monitor is used instead of the PC 300. Furthermore, in this embodiment, an example is described in which the display mode of the rectangular frame (subject frame) indicating the subject detected by the AI ​​is changed instead of the target position mark. In the first embodiment, joystick operation was accepted via the device I / F 307 of the PC 300, but in this embodiment, the controller 800 directly reads the amount of joystick operation it has and changes the target position.

[0133] In this embodiment, components that perform the same processing as in the first embodiment are given the same reference numerals, and their descriptions are omitted.

[0134] Figure 8 is a diagram showing the configuration of the remote control system in the second embodiment.

[0135] In the first embodiment, the PC300 was connected to the LAN600, but here the controller 800 is connected to the LAN600.

[0136] Next, the internal configuration of each device in the second embodiment will be explained with reference to Figure 9. Figure 9 is a block diagram showing the configuration of each device in the second embodiment.

[0137] Note that the internal configurations of the camera 100 and the edge AI device 200 are the same as those of the first embodiment and are therefore omitted.

[0138] The controller 800 includes a CPU 301, RAM 302, ROM 303, network I / F 304, display unit 305, operation unit 801, and an internal bus 308 that enables mutual communication.

[0139] The control unit 801 is, for example, a component such as a joystick provided in the controller body. A specific example of the joystick is an analog output specification that uses voltages output from variable resistors provided for the pan direction and tilt direction, respectively, as in the first embodiment.

[0140] Next, the operation of the controller 800 in this embodiment will be explained with reference to Figure 10.

[0141] Figure 10(a) shows the operation flow of the controller 800 when the user operates the joystick of the controller 800.

[0142] In the first embodiment, the joystick input was received via the PC300's device I / F307, but in this embodiment, the controller 800 directly reads the joystick input provided by the controller 800. The method for doing so is described below.

[0143] In step S601, the controller 800 accepts input via the joystick.

[0144] Specifically, the control unit 806 receives a voltage input from the joystick, and the A / D conversion unit (not shown) quantizes it at a predetermined sampling period, writing the result and information on the direction of operation to the internal RAM 302.

[0145] The CPU 301 reads the quantized value corresponding to the operation amount and the operation direction information written to the RAM 302.

[0146] In step S602, the CPU 301 calculates the amount by which the coordinate position (target position) that keeps the subject in focus while tracking the subject should be moved, based on the joystick operation direction and amount read from RAM 302 in step S601, and writes this calculation back to RAM 302.

[0147] This allows target position information to be transmitted to the edge AI device 200 in response to joystick operation, similar to the first embodiment. Steps S303, 304, and 305 are the same as in the first embodiment, so their explanation is omitted.

[0148] Next, using Figure 10(b), an example of changing the manner of tracking position information used to inform the user of the communication latency status on the display unit 305 of the controller 800 will be explained. Steps S501, 502, 503, and 506 are the same as in the first embodiment, so their explanation will be omitted.

[0149] In step S701, the CPU 301 instructs the display unit 305 to display the subject frame shown on the display unit 305 in white.

[0150] In step S702, the CPU 301 instructs the display unit 305 to display the subject frame shown on the display unit 305 in red.

[0151] The controller 800 periodically receives information regarding a subject frame, which is a rectangle representing the detected subject, from the edge AI device 200 via the network interface 304. Furthermore, the controller 800 can overlay this subject frame onto the image periodically received from the camera 100 and display it on the display unit 305.

[0152] Here, we will explain how the image displayed on the display unit 305 looks when the display color of the subject frame is changed, using Figures 11(a) to (c). Figures 11(a) to (c) are shown in chronological order.

[0153] In Figure 11(a), the user is not operating the joystick of controller 800, and the target position mark 900a is in the center of the image.

[0154] In Figure 11(a), 1000a is the subject frame, and the subject frame received from the edge AI device 200 described above is superimposed on the captured image.

[0155] Furthermore, since the coordinates of the target position received in step S501 and the coordinates of the target position read from RAM 302 in step S502 match, the subject frame 1000a is displayed in white.

[0156] Next, Figure 11(b) shows that 900b is the target position marker, and that the target position marker 900b is moving as the user operates the joystick on the controller 800.

[0157] In Figure 11(b), 1000b is the subject frame, and when the target position mark 900b moves, it is determined in step S503 that the coordinates of the target position do not match, and the process transitions to step S702, showing how the subject frame is displayed in red.

[0158] Finally, in Figure 11(c), the target position mark 900b is located at the same coordinates as in Figure 11(b), indicating that the control command has been reflected after communication latency. Since the coordinates of the target position match those of the target position received in step S501 and those read from RAM 302 in step S502, the subject frame 1000a is displayed in white.

[0159] As described above, by changing the color of the subject frame according to the comparison result of the target position information, the user can visually recognize that communication latency is occurring. Furthermore, in the second embodiment, by displaying not only the target position mark but also the detection frame in which the subject has been detected, the visibility of the subject being tracked is improved, which is useful when switching the subject being tracked.

[0160] In this embodiment, an example of changing the color of the subject frame was shown, but other methods such as changing the shape of the frame or the fill pattern may also be applied.

[0161] <Third Embodiment> Communication latency can change dynamically depending on the communication environment. Considering this, it is also useful to enable or disable the display mode change control shown in the first or second embodiment depending on the communication latency situation.

[0162] Figure 12 shows the flow of how the controller 800 switches between enabling and disabling the display mode change control based on the results of measuring the communication latency. The internal configuration of the remote control system and each device is the same as in the second embodiment.

[0163] In step S801, the CPU 301 of the controller 800 sends a message to the edge AI device 200 via the network I / F 304 using ping, which allows the response time to be determined.

[0164] In step S802, the CPU 301 receives a response from the edge AI device 200 via the network interface 304.

[0165] In step S803, the CPU 301 calculates the RTT (Round Trip Time), which corresponds to the round-trip time of communication, i.e., the communication time, and in step S804, it determines whether the RTT is equal to or greater than a predetermined time. In this embodiment, the CPU 301 calculates the communication time, but this is not limited to that. For example, the communication time obtained from the edge AI device 200 may be used.

[0166] In step S804, if the CPU 301 determines that the RTT is greater than a predetermined time, it proceeds to step S805 and enables the display mode change control. Specifically, it dynamically changes the process to execute the steps from step S503 onwards in Figure 10(b).

[0167] In step S804, if the CPU 301 determines that the RTT is less than or equal to a predetermined time, it proceeds to step S806 and disables the display mode change control. Specifically, it dynamically changes the process to skip the processing from step S503 onwards in Figure 10(b).

[0168] As described above, we have shown an example of measuring the communication latency between the controller 800 and the edge AI device 200 and switching whether or not to change the display mode based on the result, that is, an example of switching between an operating mode that takes communication time into consideration and an operating mode that does not. By adding such processing, it is possible to automatically suppress mode changes in situations where a change in the display mode is unnecessary, for example, in situations where no communication latency occurs.

[0169] Furthermore, while this embodiment shows an example of dynamically switching the display mode change control on and off according to RTT, this switching could, for example, be made configurable by the user.

[0170] Specifically, one method involves the edge AI device 200 having a web server function, and enabling it to be configured to operate in a mode that takes communication latency into consideration, based on the main unit settings and other information running on the server function.

[0171] The appearance of the target position marker described above will only be changed when the user checks the checkbox for the mode that takes communication latency into consideration in the device settings. In this way, it is also possible to configure the system to switch modes according to the user's instructions.

[0172] <Fourth Embodiment> In the first embodiment, an example was described in which the display mode of the target position mark is changed based on the target position stored in the RAM 202 of the edge AI device 200 and the RAM 302 of the PC 300.

[0173] In this embodiment, an example is described in which a second target position mark is displayed based on the target position stored in the RAM 202 of the edge AI device 200, in addition to a first target position mark based on the target position stored in the RAM 302 of the PC 300. In the fourth embodiment, when the first target position mark and the second target position mark match, the display mode of the target position marks is changed. Specifically, the first target position mark 1300 stored in RAM 302 and the second target position mark 1400 stored in RAM 202 are superimposed on the image captured by the camera 100.

[0174] This embodiment will be explained using the flowchart in Figure 13 and Figures 14(a) to (d) to illustrate how the image displayed on the display unit 305 changes when the display color of the target position mark is changed.

[0175] In the flow chart of Figure 13, the processing from steps S301 to S305 is the same as in the flow chart described above, so the explanation is omitted.

[0176] In step S901, the PC300 obtains the coordinates of the target position stored in the RAM202 from the edge AI device200.

[0177] In step S902, the CPU 301 controls the display unit 305 to display a mark indicating the target position based on the coordinates of the target position acquired in step S901.

[0178] Figures 14(a) to (d) show the contents displayed on the display unit 305 in chronological order, and Figures 14(b) to (c) are explained assuming that the user is operating the controller 400 to change the target position. Furthermore, the first target position mark 1300 indicates the target position changed by the user, i.e., the target position stored in RAM 302, and the second target position mark 1400 indicates the target position stored in RAM 202.

[0179] Figure 14(a) shows the display content when the positions of the first target position mark 1300 and the second target position mark 1400 coincide. In this embodiment, as shown in Figure 14(a), the second target position mark 1400 is not displayed when the first target position mark 1300 and the second target position mark 1400 coincide, but this is not limited to this. For example, the first target position mark may be displayed so as to overlap with the second target position mark 1400. Figure 14(b) shows the display content when the user is operating the first target position mark 1300. Figure 14(c) shows the display content when the camera's imaging direction has been changed according to the second target position mark 1400 stored in RAM 202, and the user has completed the operation to move the first target position mark 1300. Figure 14(d) shows the display content when the position of the first target position mark 1300 and the position of the second target position mark 1400 coincide due to the change in the camera's imaging direction.

[0180] Here, target position marks 1300a and 1300b are marks of different colors, and they are displayed in red if the position coordinates in the image of the first target position mark 1300 and the position coordinates in the image of the second target position mark 1400 do not match.

[0181] As described above, an example was shown in which two target position markers are displayed when the target positions do not match, based on the target positions stored in the RAM 202 of the edge AI device 200 and the RAM 302 of the PC 300. By adding such processing, the user can visually see how much communication latency is occurring on the display unit 305.

[0182] Furthermore, while this embodiment shows an example where the target position matches, it is not limited to this. For example, the color may change when the target position is approximately below a set threshold, as in the first embodiment. In addition, the system may be controlled so that the target position mark 1300 or target position mark 1400 is not displayed when the target position is approximately matched. [Explanation of symbols]

[0183] 100 Cameras 200 Edge AI Devices 300 PC 301 CPU 302 RAM 303 ROM 400 Controllers 305 Display section

Claims

1. An information processing device, An imaging device captures an image, and includes a receiving means for receiving the setting of a first target position of the subject to be tracked, A communication means that receives position information indicating a second target position of the subject in the tracking, which is held by a control device that controls an imaging device that tracks the subject in the captured image, and transmits position information indicating the first target position to the control device. A display control means that controls the display unit to display a mark indicating the first target position, It has a calculation means for calculating the communication time between the control device and the information processing device, The display control means is The mark is displayed regardless of the distance between the first target position and the second target position, and the mark is controlled to be displayed in a first manner or a second manner different from the first manner, depending on the distance between the first target position and the second target position. An information processing apparatus characterized in that, depending on the communication time, the display control means determines whether to enable control to change the display manner of the mark.

2. The display control means If the distance between the first target position and the second target position is greater than a predetermined threshold, the mark is controlled to be displayed in the first manner. If the distance to the first target position or the second target position is less than or equal to the predetermined threshold, the mark is controlled to be displayed in the second manner. The information processing apparatus according to feature 1.

3. The information processing apparatus according to claim 1, characterized in that the display control means superimposes the mark onto the captured image.

4. The aforementioned receiving means receives operations from the user, The display control means controls the display so that the mark moves in the captured image based on the user's operation. The information processing apparatus according to feature 1.

5. The system further includes a calculation means for calculating the amount of movement of the mark according to the amount of operation performed by the user, The display control means moves and displays the mark in the captured image based on the amount of movement calculated by the calculation means. The information processing apparatus according to feature 4.

6. The display control means is If the communication time is longer than a predetermined time, and the distance between the first target position and the second target position is greater than a predetermined threshold, the system is controlled to display the mark in the first manner. If the communication time is longer than the predetermined time, and the distance to the first target position or the second target position is less than or equal to the predetermined threshold, the system controls the display of the mark in the second manner. The information processing apparatus according to claim 1, characterized in that, when the communication time is less than or equal to the predetermined time, the information is displayed in the same manner regardless of the distance between the first target position and the second target position.

7. The information processing apparatus according to claim 1, characterized in that the first and second embodiments differ in shape.

8. The information processing apparatus according to claim 1, characterized in that the first and second embodiments have different colors.

9. The information processing apparatus according to claim 1, characterized in that the information indicating the second target position is not displayed.

10. An information processing method performed by an information processing device, In the captured image, a reception process is performed to accept the setting of a first target position of the subject to be tracked, A receiving step of receiving position information indicating a second target position of the subject in the tracking, which is held by a control device that controls an imaging device that tracks the subject in the captured image, A transmission step of transmitting position information indicating the first target position to the control device, A display control step that controls the display unit to display a mark indicating the first target position, A calculation step for calculating the communication time between the control device and the information processing device, The process includes a step of determining whether to enable control to change the display manner of the mark according to the communication time, An information processing method characterized in that, in the display control step, the mark is displayed regardless of the distance between the first target position and the second target position, and the mark is controlled to be displayed in a first manner or a second manner different from the first manner, depending on the distance between the first target position and the second target position.

11. A program for causing a computer to execute the information processing method described in claim 10.

Images