Synchronization control method and synchronization control system

Abstract

In the synchronization control method, time synchronization based on a reference time is performed; a processor that executes a first process in which a center time of an exposure period during which the camera is exposed is determined on the basis of the reference time, and an image of a subject is captured by exposure on the basis of the determined center time, and a second process in which the position of the servo motor at the center time is acquired on the basis of the reference time; detecting the subject from the frame on the basis of the position of the servo motor at the central time and the frame in which the subject is captured; and driving a servo motor based on the detected position of the subject.

Description

Technical Field

[0001] This disclosure relates to a synchronization control method and a synchronization control system. Background Technology

[0002] Patent Document 1 discloses a robot control device that uses a second command value obtained by adding a first command value to move the endpoint of the robot's movable part to control the movable part. The robot control device performs target tracking based on images acquired from a camera device.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2015-074063 Summary of the Invention

[0006] The problem the invention aims to solve

[0007] However, in the structure of Patent Document 1, since there is no control to ensure that the timing of the frame capture (capture time) is consistent with the timing of the encoder value readout, the position of the robot arm at the capture time is unknown, and position deviation may occur during movement control to move the current position to the target position. Furthermore, in conventional robot control devices that perform target tracking based on the position of the endpoint captured in the frame and the position of the target object, there is also no control to ensure that the timing of the frame capture (capture time) is consistent with the timing of the encoder value readout. Therefore, the position of the endpoint captured in the frame may differ from the position of the endpoint indicated by the readout encoder value.

[0008] This disclosure is made in view of the above-mentioned existing situation, and its purpose is to provide a synchronization control method and a synchronization control system for synchronizing a camera with a servo motor to improve the tracking accuracy of an object.

[0009] Solution for solving the problem

[0010] This disclosure provides a synchronization control method, which is a synchronization control method performed by a synchronization control system. The synchronization control system includes at least one image processing device and at least one controller. The at least one image processing device is capable of acquiring frames of the subject captured by a camera that captures the subject. The at least one controller is capable of communicating with the image processing device and controlling a servo motor that moves the position of the camera. In the synchronization control method, time synchronization based on a reference time is performed; a first process and a second process are performed. In the first process, the center time of the exposure period of the camera is determined based on the reference time, and the subject is captured by exposure based on the determined center time. In the second process, the position of the servo motor at the center time is acquired based on the reference time; the subject is detected from the frames based on the position of the servo motor at the center time and the frames of the subject captured; and the servo motor is driven based on the position of the detected subject.

[0011] Additionally, this disclosure provides a synchronization control system comprising at least one image processing device and at least one controller. The at least one image processing device is capable of acquiring frames of the subject captured by a camera that captures the subject. The at least one controller is capable of communicating with the image processing device to control a servo motor that moves the position of the camera. The image processing device and the controller perform time synchronization based on a reference time. The image processing device performs a first process in which it determines the center time of the exposure period of the camera based on the reference time, acquires frames captured by the camera based on the determined center time exposure, and detects the subject from at least a portion of the detected pixels in the frames. The controller performs a second process in which it acquires the position of the servo motor at the center time based on the reference time and sends it to the image processing device. The image processing device detects the position of the subject from the frames based on the position of the servo motor at the center time and sends it to the controller. The controller drives the servo motor based on the received position of the subject.

[0012] Invention Effects

[0013] According to this disclosure, it is possible to synchronize the camera with the servo motor to improve the tracking accuracy of objects. Attached Figure Description

[0014] Figure 1 This is a block diagram illustrating an example of the overall structure of the motor control system according to Embodiment 1.

[0015] Figure 2 This is a block diagram illustrating an example of the internal structure of the image processing apparatus and servo controller in Embodiment 1.

[0016] Figure 3 This is a sequence diagram illustrating an example of the operation process of the motor control system according to Embodiment 1.

[0017] Figure 4 This is a sequence diagram illustrating an example of the operation process of the image processing device and the servo controller in Embodiment 1.

[0018] Figure 5 This is a timing diagram showing the operation process of the image processing device and the servo controller in Embodiment 1.

[0019] Figure 6 This is a comparison diagram illustrating conventional image processing devices and motion processing examples of servo motors.

[0020] Figure 7 This is a comparative diagram illustrating the image processing apparatus and motion processing example of the servo motor of this disclosure.

[0021] Figure 8 This is a block diagram illustrating an example of the overall structure of the motor control system according to Embodiment 2.

[0022] Figure 9 This is a sequence diagram illustrating an example of the operation process of the motor control system according to Embodiment 2. Detailed Implementation

[0023] (The process of this disclosure)

[0024] In recent years, a control system has emerged that utilizes an image processing device and a motor control device. The image processing device uses a camera to capture images of an object and measures the position of the object within the captured frames. The motor control device drives servo motors to enable a robotic arm or similar device to perform a predetermined task on the object. This control system calculates the movement of the servo motors based on the object's position measured by the image processing device and the encoder position information read from the encoder, thereby correcting the position of the robotic arm relative to the object. The control system then executes the drive control of the servo motors by the motor control device based on the calculated movement, thus enabling the predetermined task to be performed on the object.

[0025] However, conventional motor control devices consist of a servo motor and a dedicated controller (control unit) for controlling the servo motor. These devices drive the servo motor by specifying position information through the controller, and output a completion notification after the movement to the specified position is completed. Therefore, it is difficult to obtain the servo motor's position (encoder value) within a specified time. Furthermore, conventional image processing devices do not use the servo motor parameters output from the external motor control device to perform image processing. Consequently, conventional motor control devices do not know the actual time information of the encoder position information, nor do they know the time information corresponding to the position of the object obtained through image processing. This results in problems such as reduced tracking accuracy or tracking delay for the motor control device.

[0026] Therefore, examples of synchronization control methods and synchronization control systems for improving the tracking accuracy of objects by synchronizing the camera with the servo motor are described in the various embodiments shown below.

[0027] The following detailed description, with appropriate reference to the accompanying drawings, details various embodiments of the structure and function of the synchronization control method and synchronization control system involved in this disclosure. However, sometimes unnecessary detailed descriptions are omitted. For example, detailed descriptions of already known matters and repetitive descriptions of substantially the same structures are sometimes omitted. This is to avoid making the following description overly lengthy and to facilitate understanding by those skilled in the art. Furthermore, the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand this disclosure and are not intended to limit the subject matter of the claims.

[0028] Furthermore, in the following description, the term "image processing apparatus" is sometimes referred to as an apparatus that includes a camera and lighting connected to each image processing apparatus in a manner capable of transmitting and receiving data.

[0029] (Implementation Method 1)

[0030] Reference Figure 1 Here, an example of the overall structure of the motor control system 100 according to Embodiment 1 will be described. Figure 1 This is a block diagram illustrating an example of the internal structure of the motor control system 100 according to Embodiment 1.

[0031] As an example of a system, image processing system, or synchronous control system, a motor control system 100 is configured to include N (N: an integer greater than or equal to 1) servo controllers (specifically, first servo controller S1 to Nth servo controller SN) and K (K: an integer greater than or equal to 1) image processing devices (specifically, first image processing device Im1 to Kth image processing device ImK). Furthermore, the number of servo controllers N and the number of image processing devices K can be the same or different values.

[0032] The motor control system 100 performs timing through any one of the following devices (i.e., a servo controller or an image processing device): the first servo controller S1 to the Nth servo controller SN and the first image processing device Im1 to the Kth image processing device ImK. The motor control system 100 synchronizes and cooperates with the first servo controller S1 to the Nth servo controller SN and the first image processing device Im1 to the Kth image processing device ImK at the synchronization time. Based on the image processing results (i.e., the position of the object) of each of the first image processing devices Im1 to the Kth image processing device ImK, the first servo controller S1 to the Nth servo controller SN respectively execute the object tracking.

[0033] As an example of a controller, each of the first servo controllers S1 to Nth servo controllers SN is connected to other servo controllers besides itself and to each of the first image processing devices Im1 to Kth image processing devices ImK in a manner that enables the transmission and reception of data.

[0034] Each servo motor includes an encoder, a motor driver, and a servo motor. That is, the motor control system 100 includes N encoders (first encoder En1 to Nth encoder EnN), N motor drivers (first motor driver Md1 to Nth motor driver MdN), and N servo motors (first servo motor Sm1 to Nth servo motor SmN).

[0035] Each of the first image processing devices Im1 to the Kth image processing device ImK is connected to other image processing devices besides itself, as well as to each of the first servo controller S1 to the Nth servo controller SN, in a manner that enables the transmission and reception of data.

[0036] Each image processing device includes a camera and lighting. That is, the motor control system 100 includes K cameras (first camera C1 to Kth camera CK) and K lights (first lighting L1 to Kth lighting LK).

[0037] Next, refer to Figure 2The internal structure of the image processing apparatus and the servo controller in Embodiment 1 will be described. Figure 2 This is a block diagram illustrating an example of the internal structure of the image processing apparatus and servo controller in Embodiment 1. Furthermore, in Figure 2 In order to simplify the explanation, the internal structure of the first image processing device Im1 and the first servo controller S1, which are the minimum structures of the motor control system 100, will be described as an example.

[0038] The first image processing device Im1 includes a communication unit 10, a processor 11, and a memory 12.

[0039] The communication unit 10 is connected to the first camera C1, the first illumination L1, and the first servo controller S1 in a manner capable of transmitting and receiving data. The communication unit 10 is constructed using a communication interface circuit for transmitting and receiving data. The communication unit 10 uses CameraLink (registered trademark) cables, Ethernet cables, EtherCAT (registered trademark), etc., to connect to the first camera C1, the first illumination L1, and the first servo controller S1 in a wired communication manner to achieve data transmission and reception.

[0040] The processor 11 is constructed using a real-time operating system (hereinafter referred to as "OS") such as a Field Programmable Gate Array (FPGA) and works in conjunction with the memory 12 to perform various processing and control tasks. Specifically, the processor 11 refers to the programs and data stored in the memory 12 and executes the programs to implement the functions of each part.

[0041] The processor 11 determines the start and end times of exposure for the first camera C1, as well as the on and off times of the first illumination, and controls the first camera C1 and the first illumination respectively. Specifically, during the exposure period from the start to the end of exposure for the first camera C1, the processor 11 determines the start and end times of exposure for the first camera C1, as well as the on and off times of the first illumination, in a manner that the exposure time corresponding to half of the exposure period (hereinafter referred to as the "exposure midpoint") coincides with the readout time of the servo motor position information sent from the first servo controller S1.

[0042] Processor 11 acquires encoder position information of the first encoder En1 and drive electrical value information of the first servo motor Sm1 sent from the first servo controller S1. Processor 11 estimates (predicts) the position of the captured object in the frame using a Kalman filter based on the encoder position information of the first encoder En1, and acquires pixel data corresponding to the estimated position from the first camera C1. Processor 11 performs image processing using the Hill Climbing Method on the acquired pixel data to detect the position of the object.

[0043] The processor 11 sends information about the determined readout area of ​​the pixel data to the first camera C1. Instead of storing the pixel data sent from the first camera C1 into the memory 12, the processor 11 directly performs image processing and measures the position of the object. The processor 11 then sends the image processing result, i.e., the position information of the object, to the first servo controller S1. Thus, the processor 11 can shorten the time required for image processing, achieving high-speed image processing.

[0044] The memory 12 includes, for example, random access memory (hereinafter referred to as "RAM") used as working memory during the execution of various processes of the processor 11, and read-only memory (hereinafter referred to as "ROM") for storing programs and data that define the actions of the processor 11. Data or information generated or acquired by the processor 11 is temporarily stored in the RAM. Programs defining the actions of the processor 11 are written into the ROM.

[0045] The first camera C1 is configured to have at least a lens (not shown) and an image sensor (not shown). The image sensor is, for example, a solid-state imaging element of a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS), which converts the optical image imaged on the imaging surface into an electrical signal. The first camera C1 captures images of the object at a high frame rate of 960 fps or higher, for example.

[0046] The first camera performs exposure control based on the shutter timing (exposure start command and exposure end command) sent from the processor 11. Additionally, after converting the image sensor into an electrical signal, the first camera C1 sequentially executes the readout areas requested by the processor 11 (e.g., ...). Figure 5The reading of pixel data contained in the reading areas AR1, AR2, AR3, AR4, etc. shown, and the transmission of the read pixel data.

[0047] The first illumination L1 is implemented by at least one light-emitting diode (LED). The first illumination L1 controls the start and end of illumination based on the illumination start and end instructions sent from the processor 11, thereby illuminating the object.

[0048] The first servo controller S1 includes a communication unit 20, a processor 21, a memory 22, a first motor driver Md1, a first servo motor Sm1, and a first encoder En1.

[0049] The communication unit 20 is connected to the first image processing device Im1 in a manner capable of transmitting and receiving data. The communication unit 20 is configured using a communication interface circuit for transmitting and receiving data. The communication unit 20 is connected to the first image processing device Im1 via a wired communication method using an HDMI cable (registered trademark) or similar cable to achieve data transmission and reception.

[0050] The processor 21 is constructed using, for example, a central processing unit (CPU), an FPGA, etc., and works in conjunction with the memory 22 to perform various processing and control tasks. Specifically, the processor 21 refers to the program stored in the memory 22 and executes the program to implement the functions of the first motor driver Md1, the first servo motor Sm1, the first encoder En1, etc.

[0051] Processor 21 acquires encoder feedback information from first encoder En1 at the center of exposure of first camera C1. Furthermore, the encoder feedback information mentioned here is the rotation angle of first servo motor Sm1. Processor 21 measures the position of first servo motor Sm1 based on the rotation angle of first servo motor Sm1.

[0052] The processor 21 calculates a correction amount for the position of the first servo motor Sm1 based on the image processing result (i.e., the position information of the object) sent from the first image processing device Im1 and the current position information of the first servo motor Sm1. Based on the calculated correction amount, the processor 21 generates a current control command to drive the first servo motor Sm1 and outputs the current control command to the first motor driver Md1.

[0053] The memory 22 includes, for example, RAM used as working memory during the execution of various processes of the processor 21, and ROM for storing programs and data that define the actions of the processor 21. Data or information generated or acquired by the processor 21 is temporarily stored in the RAM. Programs defining the actions of the processor 21 are written into the ROM.

[0054] The first motor driver Md1 drives (rotates) the first servo motor Sm1 based on current control commands output from the processor 21. The first motor driver Md1 acquires current feedback information through feedback control. This current feedback information represents the difference between the target movement value of the first servo motor Sm1 corresponding to the current control command and the actual control amount of the first motor driver Md1. The first motor driver Md1 outputs the acquired current feedback information to the processor 21.

[0055] The first encoder En1 can detect the rotation angle of the first servo motor Sm1 and output (feedback) the rotation angle information of the first servo motor Sm1 (encoder feedback information) to the processor 21.

[0056] Next, refer to Figure 3 An example of the operation process of the motor control system 100 according to Embodiment 1 will be described. Figure 3 This is a sequence diagram illustrating an example of the operation process of the motor control system 100 according to Embodiment 1.

[0057] In addition, Figure 3 The example shown illustrates a first image processing device Im1 configured as a host to synchronize the timing information of other image processing devices (second image processing devices Im2 to Kth image processing devices ImK) and the first servo controller S1 to the Nth servo controller SN at a synchronization time (i.e., to perform clock alignment). However, the device configured as the host can be any other device.

[0058] The motor control system 100 executes step St11 and the first loop process Lp11. Furthermore, the number of cycles for the first loop process Lp11 can be set arbitrarily based on the time synchronization interval of step St11.

[0059] The first image processing device Im1 sends synchronized timing information to other image processing devices (second image processing devices Im2 to Kth image processing device ImK) and the first servo controller S1 to the Nth servo controller SN, and repeatedly performs time synchronization (St11) for each device at predetermined intervals. Here, the motor control system 100 can determine that the time synchronization process has failed if the timing information in step St11 cannot be received by each device simultaneously, and thus transfer to the first loop process Lp11, delaying the time synchronization process to the next step St11.

[0060] Additionally, as shown by the dashed line, when the first servo controller S1 is set as the host, the first servo controller S1 sends synchronization time information to the first image processing device Im1 to the Kth image processing device ImK and the second servo controller (not shown) to the Nth servo controller SN respectively, and performs time synchronization (St11A) for each device.

[0061] The first image processing device Im1 to the Kth image processing device ImK each use the first camera C1 to the Kth camera CK and the first illumination L1 to the Kth illumination LK to perform object image processing (St100).

[0062] Each of the first servo controllers S1 to the Nth servo controller SN performs servo motor position acquisition processing (St200) at the exposure center time T0 of the first camera C1 to the Kth camera CK to acquire the position information of the first servo motor Sm1 to the Nth servo motor SmN.

[0063] Each of the first servo controllers S1 to the Nth servo controller SN sends the acquired position information (St12, St13) of the servo motors to other servo controllers and the first image processing device Im1 to the Kth image processing device ImK.

[0064] For example, the first servo controller S1 sends the position information (St12) of the first servo motor Sm1 to the second servo controller (not shown) to the Nth servo controller SN and the first image processing device Im1 to the Kth image processing device ImK.

[0065] Additionally, for example, the Nth servo controller SN sends the position information (St13) of the Nth servo motor SmN to the first servo controller S1 to the (N-1)th servo controller (not shown) and the first image processing device Im1 to the Kth image processing device ImK, respectively.

[0066] In addition, Figure 3In the sequence shown, for ease of understanding, step St12 shows the process of the first servo controller S1 sending the position information of the first servo motor Sm1, and step St13 shows the process of the Nth servo controller SN sending the position information of the Nth servo motor SmN. However, the processes of steps St12 and St13 can be performed at approximately the same time.

[0067] The motor control system 100 enables the second loop processing Lp21 executed by each of the first image processing device Im1 to the Kth image processing device ImK and the third loop processing Lp22 executed by each of the first servo controller S1 to the Nth servo controller SN to execute in parallel.

[0068] In the second loop processing Lp21, each of the first image processing device Im1 to the Kth image processing device ImK performs image processing approximately simultaneously (St300) based on the frames captured in the processing in step St100 and the position information of the first servo motor Sm1 to the Nth servo motor SmN obtained in steps St12 to St13.

[0069] In the third loop processing Lp22, each of the first servo controllers S1 to the Nth servo controller SN executes the driving processing of the first servo motor Sm1 to the Nth servo motor SmN and the acquisition processing of the position information of the first servo motor Sm1 to the Nth servo motor SmN (St400). The first servo controllers S1 to the Nth servo controller SN send the acquired position information of the servo motors to each other (St14).

[0070] The first image processing device Im1 to the Kth image processing device ImK each send the image processing results obtained through step St300, i.e., the position information of the object (St15, St16), to other image processing devices and the first servo controller S1 to the Nth servo controller SN, respectively.

[0071] For example, the first image processing device Im1 performs image processing on the frames (pixel data) captured by the first camera C1, and sends the image processing result, i.e. the position information of the object, as the position information of the object to the second image processing device (not shown) ~ the Kth image processing device ImK and the first servo controller S1 ~ the Nth servo controller SN (St15).

[0072] Additionally, for example, the Kth image processing device ImK performs image processing on the frame (pixel data) captured by the Kth camera CK, and sends the image processing result, i.e. the position information of the object, as the position information of the object to the first image processing device Im1 to the (K-1)th image processing device (not shown) and the first servo controller S1 to the Nth servo controller SN (St16).

[0073] Next, refer to Figure 4 An example of the operation process of the image processing device and the servo controller is explained. Figure 4 This is a sequence diagram illustrating an example of the operation process of the image processing apparatus (specifically, the first image processing apparatus Im1 to the Kth image processing apparatus ImK) and the servo controller (specifically, the first servo controller S1 to the Nth servo controller SN) in Embodiment 1.

[0074] In addition, Figure 4 In the example shown, for in Figure 3 The actions performed in the first loop process Lp11 shown are explained in detail. Additionally, in Figure 4 In order to simplify the explanation, the operation process performed between the first image processing device Im1, which is the minimum structure of the motor control system 100, and the first servo controller S1 will be described as an example, and the processes of steps St13 and St16 will be omitted.

[0075] First, the camera processing in step St100 will be explained. The camera processing in step St100 is performed, for example, at a period of 1 kHz.

[0076] The first image processing device Im1 determines the timing of the start and end of the exposure of the first camera C1, and the timing of the start and end of the illumination of the first illumination L1 (St101).

[0077] Furthermore, the first image processing device Im1 determines the exposure center time T0, as well as the timing of the start and end of emission, based on the synchronization time after time synchronization. For example, the exposure center time T0 can be determined at intervals of a predetermined period (e.g., 1 kHz).

[0078] Therefore, each of the first image processing unit Im1 to the Kth image processing unit ImK can prevent deviations in the shooting time caused by shutter opening and closing times, etc. Thus, each of the first image processing unit Im1 to the Kth image processing unit ImK can ensure that the exposure center time T0 of all cameras (first camera C1 to Kth camera CK) is consistent.

[0079] The first image processing device Im1 generates an exposure start command for the first camera C1 based on the determined exposure center time T0, and sends the generated exposure start command to the first camera C1 based on the timing of the exposure start of the first camera C1 (St102). Additionally, the first image processing device Im1 generates a light emission start command for the first illumination L1 based on the determined exposure center time T0, and sends the generated light emission start command to the first illumination L1 based on the timing of the light emission start of the first illumination L1 (St103). Furthermore, the processing order of steps St102 and St103 can also be reversed.

[0080] The first illumination L1 starts to emit light based on the light emission start command sent from the first image processing device Im1 (St104).

[0081] The first camera C1 begins exposure (St105) based on the exposure start command sent from the first image processing device Im1.

[0082] The first image processing device Im1 generates an exposure end command for the first camera C1 and sends the generated exposure end command (St106) to the first camera C1 based on the timing of the exposure end of the first camera C1. The first camera C1 ends the exposure based on the exposure end command sent from the first image processing device Im1.

[0083] In addition, the first image processing device Im1 generates a light emission termination command for the first illumination L1, and sends the generated light emission termination command (St107) to the first illumination L1 based on the timing of the light emission termination of the first illumination L1. The first illumination L1 stops emitting light based on the light emission termination command sent from the first image processing device Im1.

[0084] also, Figure 4 The order of processing steps St102 and St103, and the order of processing steps St106 and St107 shown are examples and are not limited to this. In addition, the illumination period of the first illumination L1 can be approximately the same as the exposure period of the first camera C1, or it can be a longer time.

[0085] Next, the servo motor position acquisition process in step St200 will be described. The servo motor position acquisition process in step St200 is performed at a period corresponding to the camera processing in step St100, for example, a period of 1 kHz.

[0086] The first servo controller S1 generates an A / D conversion start command to request the position information of the first servo motor Sm1 from the first encoder En1, and outputs the A / D conversion start command (St201) to the first encoder En1.

[0087] The first encoder En1 acquires an electronic signal (analog signal) representing the position information of the first servo motor Sm1 at the exposure center time T0 of the first image processing device Im1, and converts the electronic signal into a digital signal (St202). The first encoder En1 outputs the position information of the first servo motor Sm1 to the first servo controller S1 (St203).

[0088] Therefore, each servo controller from the first servo controller S1 to the Nth servo controller SN can also acquire the position information of the servo motors (first servo motor Sm1 to Nth servo motor SmN) at the same time as the exposure center time T0, based on the synchronization time after time synchronization. In other words, the motor control system 100 can align the recording time of all cameras and the acquisition time of the position information of all servo motors to the same time (= exposure center time T0).

[0089] The first servo controller S1 sends the position information of the first servo motor Sm1 obtained by the first encoder En1 to the first image processing device Im1 (St12).

[0090] Next, the image processing in step St300 will be described. The image processing in step St300 is performed, for example, at a period of 1 kHz.

[0091] The first image processing device Im1 uses a Kalman filter to estimate the position of an object captured in a subsequent frame based on the frame rate of the first camera C1, the position information of the first servo motor Sm1 sent from the first servo controller S1, and the position of an object detected in the previous frame. The first image processing device Im1 requests the first camera C1 to read out and transmit pixel data containing at least one pixel corresponding to the estimated position of the object.

[0092] The first camera C1 performs A / D conversion processing, which replaces the light incident from the lens during exposure with an electrical signal (analog signal) and then converts it into a digital signal. The first camera C1 reads out the pixel data and sends it to the first image processing device Im1 (St301).

[0093] The first image processing device Im1 performs image processing (St302) before recording the pixel data sent from the first camera C1 into the memory 12. The first image processing device Im1 sequentially performs object detection processing based on a hill-climbing method on the pixel data sent from the first camera C1. The first image processing device Im1 calculates an evaluation value representing the probability that the detected unit from the pixel data is an object. The first image processing device Im1 requests the first camera C1 to read pixel data containing at least one pixel located near a pixel with a higher calculated evaluation value using the hill-climbing method.

[0094] The first camera C1 reads pixel data and sends it to the first image processing device Im1 (St301) based on a request sent from the first image processing device Im1.

[0095] The first image processing device Im1 performs image processing (St302) before recording the pixel data sent from the first camera C1 into the memory 12. The first image processing device Im1 repeatedly performs the pixel data reading process of the first camera C1 in step St301 and the object detection process of the first image processing device Im1 in step St302 until the improvement of the evaluation value based on the hill-climbing method disappears, that is, until it is determined that an object has been detected.

[0096] The first image processing device Im1 sends the image processing result, namely the position information (St15) obtained by detecting the object, to the first servo controller S1. In addition, the first image processing device Im1 records the frames acquired from the first camera C1, the address numbers indicating the shooting order of the frames by the first camera C1, the position information of the detected object, and the position information of the servo motor in the memory 12.

[0097] Therefore, the first image processing device Im1 can calculate the amount of movement of the servo motor between two frames based on the position information of the object captured in the previous frame, the position information of the first servo motor Sm1 at the shooting time (= exposure center time T0) of the previous captured frame, and the position information of the first servo motor Sm1 at the shooting time (= exposure center time T0) of the next captured frame. The first image processing device Im1 can estimate the position of the object captured in the next frame with higher accuracy based on the amount of movement of the servo motor between two frames.

[0098] Therefore, compared with the past, the first image processing device Im1 can estimate the position of the object captured in the next frame with higher accuracy, thus effectively selecting the pixels to be processed and reducing the number of pixels processed by the hill-climbing method, thereby further shortening the object detection time.

[0099] Furthermore, the first image processing device Im1 can process the two frames as frames captured by the stereo camera, based on two consecutive frames captured by the first camera C1 at a high frame rate and the amount of movement of the servo motor between the two frames. Therefore, the first image processing device Im1 can measure the position of the object in a direction approximately perpendicular to the image sensor of the first camera C1 (that is, the distance or depth between the object and the image sensor). Thus, the first image processing device Im1 can measure the distance between the first camera C1 and the object.

[0100] Furthermore, when the object is moving, the first image processing device Im1 can measure the object's speed and distance of movement based on two consecutive frames captured by the first camera C1 at a high frame rate and the amount of movement of the servo motor between the two frames. Additionally, when the object is not moving, the first image processing device Im1 can measure (determine) the object's amount of movement, speed, etc., based on two consecutively captured frames and the amount of movement of the servo motor between the two frames, and use this as interference acting on the object. Furthermore, the determination of interference will be discussed later. Figure 6 and Figure 7 The details are explained in the text.

[0101] Furthermore, when the object is not moving at a constant linear speed, the first image processing device Im1 can measure changes in the object's movement, acceleration, etc., based on two consecutively captured frames and the movement of the servo motor between the two frames.

[0102] Next, the motor drive processing and servo motor position acquisition processing in step St400 will be described. The motor drive processing and servo motor position acquisition processing in step St400 are executed, for example, at a cycle of tens of kHz.

[0103] The first servo controller S1 generates a current control command and outputs it to the first motor driver Md1 (St401). The first motor driver Md1 drives the first servo motor Sm1 (St402) based on the current control command output from the first servo controller S1.

[0104] The first servo controller S1 generates an A / D conversion start command and outputs it to the first encoder En1 (St403). Based on the A / D conversion start command output from the first servo controller S1, the first encoder En1 acquires an electronic signal (analog signal) representing the position information of the first servo motor Sm1 and converts this electronic signal into a digital signal (St404). The first encoder En1 outputs the position information of the first servo motor Sm1 to the first servo controller S1 (St405).

[0105] Here, Figure 3 The processing in step St400 is repeatedly executed at an earlier cycle than other processing steps. Based on the position information of the object obtained in the processing of steps St100 to St300, the motor control system 100 corrects the current position of the first servo motor Sm1 in the processing of step St400, or moves the position of the first servo motor Sm1 in accordance with the position of the object (following).

[0106] Next, refer to Figure 5 Timing diagrams are used to illustrate the operation of the image processing device and the servo controller.Figure 5 This is a timing diagram illustrating the operation of the image processing device and the servo controller in Embodiment 1. Furthermore, Figure 5 The readout cycle and number of times shown for the encoder are examples and are not limited to these.

[0107] In addition, Figure 5 In the example shown, for the sake of simplicity, an example of the action process performed between the first image processing device Im1, which is the minimum structure of the motor control system 100, and the first servo controller S1 will be described.

[0108] The first camera C1 exposes to capture the object at a central exposure time t=0 (zero), and sequentially reads out the pixel data contained in the readout region AR1 of the first frame Img1 starting from time t=t1. The first image processing unit Im1 acquires the pixel data read from the first camera C1 and sequentially performs image processing. The first image processing unit Im1 sends the detected object's position information to the first servo controller S1.

[0109] During the object imaging process performed by the first camera C1 and the object detection process performed by the first image processing device Im1, the first servo controller S1 drives the first motor driver Md1 and repeatedly performs the acquisition (readout) of the position information of the first servo motor Sm1 by the first encoder En1 at a cycle of tens of kHz. In addition, the image Ps1 represents the state of the first motor driver Md1 at time t=0 (zero).

[0110] The first servo controller S1 acquires the position information of the first servo motor Sm1 at the exposure center time t=1 and sends it to the first image processing device Im1.

[0111] The first camera C1 exposes the image of the object in such a way that the exposure center time t=1. The first image processing unit Im1 estimates the position of the object captured in the second frame Img2 based on the position information of the object detected in the first frame Img1 and the position information of the first servo motor Sm1 sent from the first servo controller S1, and determines the readout area AR2 from which pixel data is to be read from the second frame Img2. The first image processing unit Im1 requests the first camera C1 to read out the readout area AR2.

[0112] Starting from time t=t2, the first camera C1 sequentially reads the pixel data contained in the readout region AR2 of the second frame Img2. The first image processing device Im1 acquires the pixel data read from the first camera C1 and sequentially performs image processing. The first image processing device Im1 sends the position information of the detected object to the first servo controller S1.

[0113] During the object imaging process performed by the first camera C1 and the object detection process performed by the first image processing device Im1, the first servo controller S1 drives the first motor driver Md1 based on the object position information detected from the first frame Img1, and repeatedly performs the acquisition (readout) of the position information of the first servo motor Sm1 by the first encoder En1 at a cycle of tens of kHz. Furthermore, image Ps2 represents the state of the first motor driver Md1 at time t=1.

[0114] The first servo controller S1 acquires the position information of the first servo motor Sm1 at the exposure center time t=2 and sends it to the first image processing device Im1.

[0115] The first camera C1 exposes the object in such a way that the exposure center time t=2. The first image processing device Im1 measures the movement amount and speed of the object, the movement amount and speed of the first servo motor Sm1, etc., during the exposure center time t=1~2, based on the position information of the object detected in the first frame Img1 and the second frame Img2, and the position information of the first servo motor Sm1 corresponding to each frame sent from the first servo controller S1.

[0116] The first image processing unit Im1 uses the calculated movement and speed of the object during the exposure center time t=1~2, as well as the movement and speed of the first servo motor Sm1, to estimate the position of the object captured in the third frame Img3. The first image processing unit Im1 determines the readout area AR3 from the third frame Img3 from which pixel data is to be read. The first image processing unit Im1 requests the first camera C1 to read out the readout area AR3.

[0117] Starting from time t=t3, the first camera C1 sequentially reads the pixel data contained in the readout region AR3 of the third frame Img3. The first image processing device Im1 acquires the pixel data read from the first camera C1 and sequentially performs image processing. The first image processing device Im1 sends the position information of the detected object to the first servo controller S1.

[0118] During the object imaging process performed by the first camera C1 and the object detection process performed by the first image processing device Im1, the first servo controller S1 drives the first motor driver Md1 based on the object's position information corresponding to the second frame Img2, and repeatedly performs the acquisition (readout) of the first servo motor Sm1's position information by the first encoder En1 at a cycle of tens of kHz. Furthermore, image Ps3 represents the state of the first motor driver Md1 at time t=2. Image Ps4 represents the state of the first motor driver Md1 at time t=3.

[0119] The first servo controller S1 acquires the position information of the first servo motor Sm1 at the exposure center time t=3 and sends it to the first image processing device Im1.

[0120] The first camera C1 exposes the object at a central exposure time t=4 to capture the image. The first image processing unit Im1 calculates the object's movement amount, movement speed, and the movement amount and movement speed of the first servo motor Sm1 during the central exposure time t=2~3, based on the position information of the object detected in the second frame Img2 and the third frame Img3, and the position information of the first servo motor Sm1 corresponding to each frame sent from the first servo controller S1.

[0121] The first image processing unit Im1 uses the calculated movement and speed of the object during the exposure center time t=2~3, as well as the movement and speed of the first servo motor Sm1, to estimate the position of the object captured in the fourth frame Img4. The first image processing unit Im1 determines the readout area AR4 from the fourth frame Img4 from which pixel data will be read. The first image processing unit Im1 requests the first camera C1 to read out the readout area AR4.

[0122] Based on the above description, the motor control system 100 of Embodiment 1 can follow the object in real time through each of the first servo motor Sm1 to the Nth servo motor SmN.

[0123] Next, refer to respectively Figure 6 and Figure 7 This will explain how to identify and handle interference. Figure 6 This is a comparison diagram illustrating conventional image processing devices and motion processing examples of servo motors. Figure 7 This is a comparative diagram illustrating the image processing apparatus and motion processing example of the servo motor of this disclosure.

[0124] In addition, Figure 6 and Figure 7In the example shown, a camera is mounted on a trolley Tr driven by a servo motor. The servo motor causes the trolley Tr to move in a manner that follows the position of an object Tg, captured and measured by the camera, while a glove Gr captures the approaching object Tg. Additionally, in Figure 6 and Figure 7 In the example shown, illustrations of the image processing device, lighting, servo controller, encoder, servo motor, etc., are omitted.

[0125] In addition, Figure 6 and Figure 7 In the example shown, the interference is a wind blowing towards the right side of the paper, which moves the object Tg 20 cm to the right side of the paper during the period from the exposure center time t=11 to the exposure center time t=12. Furthermore, for the sake of simplicity, an example will be given here where the interference only affects the object Tg and not the trolley Tr.

[0126] First, refer to Figure 6 The image processing device and the motion processing of the servo motor are explained in the case where the imaging time (exposure center time t=11, 12, 13) of the object being photographed is not synchronized with the acquisition time of the position information of the servo motor.

[0127] The camera captures an image of the object Tg at a central exposure time t=11. The image processing unit detects the object Tg from the frame Img11 captured by the camera. Figure 6 In frame Img11 shown, the image processing device detects the object Tg at a position offset 1m to the right of the paper relative to the camera's viewpoint center, that is, relative to the estimated position Tg11 of the object Tg, which is the center position of frame Img11 captured by the camera. The image processing device sends the position information of the object Tg to the servo controller.

[0128] The servo controller generates a current control command based on the position information of the object Tg sent from the image processing device, which moves the trolley Tr 1m to the right of the paper. The servo controller outputs the generated current control command to the motor driver. The motor driver drives the servo motor based on the current control command, causing the trolley Tr to move 1m to the right of the paper.

[0129] The object Tg moves 20cm to the right of the paper due to interference during the period from the exposure center time t=11 to the exposure center time t=12.

[0130] The camera captures an image of the object Tg at a central exposure time t=12. The image processing unit detects the object Tg from the frame Img12 captured by the camera. Figure 6In the frame Img12 shown, the image processing device detects the object Tg at a position that is 70cm offset to the right of the paper relative to the estimated position Tg12 of the object Tg.

[0131] However, the image processing device does not know the exact position of the carriage Tr at the center of the camera exposure time t=12. Therefore, conventional image processing devices cannot transmit precise information such as how much the carriage Tr should move to the servo controller.

[0132] The servo motor at the exposure center time t=12 causes the trolley Tr to continue moving based on the position of the object Tg detected at the exposure center time t=11, that is, to continue moving 1m to the right of the paper.

[0133] The camera captures an image of the object Tg at a central exposure time t=13. The image processing unit detects the object Tg from the frame Img13 captured by the camera. Figure 6 In frame Img13 shown, the image processing device detects the object Tg at a position that is offset 20cm to the right of the paper relative to the estimated position Tg13 of the object Tg.

[0134] When the carriage Tr completes its movement based on the position of the object Tg detected at the exposure center time t=11, that is, its movement 1m to the right of the paper, the servo motor at the exposure center time t=13 stops. The servo controller generates a notification indicating that the movement of the carriage Tr to the right of the paper 1m by the servo motor has been completed and sends this notification to the image processing device.

[0135] As described above, without synchronizing the camera capture time (exposure center time t=11, 12, 13) with the acquisition time of the servo motor's position information, the servo controller continues to execute servo motor drive control based on the position information of the object Tg obtained from the image processing device until the movement of the carriage Tr is completed. However, when interference is applied to the object Tg, during the period from exposure center time t=11 to exposure center time t=13 when the carriage Tr is moving, the object Tg moves 20cm to the right of the paper due to the interference.

[0136] Therefore, because the object Tg moved 20cm to the right of the paper due to interference at the exposure center time t=13, the servo controller had difficulty keeping the positions of the carriage Tr and glove Gr following the object Tg. Furthermore, during the period until the servo motor completed its 1m movement to the right of the paper at the exposure center time t=13, the image processing unit did not receive position information from the servo controller for the servo motor. Consequently, the image processing unit's estimation accuracy for the object Tg captured in the next frame decreased. Because of the reduced accuracy in estimating the object Tg's position, the image processing unit had to perform more image processing for detecting the object Tg in the next frame, resulting in a longer image processing time.

[0137] Next, refer to Figure 7 The image processing device and the motion processing of the servo motor are explained when the imaging time (exposure center time t=11, 12, 13) of the subject is synchronized with the acquisition time of the position information of the servo motor.

[0138] The camera captures the object Tg at the center of the exposure time t=11. The image processing device detects the object Tg from the frame Img21 captured by the camera. Figure 7 In frame Img21 shown, the image processing device detects the object Tg at a position offset 1m to the right of the paper relative to the camera's viewpoint center, that is, relative to the estimated position Tg21 of the object Tg, which is the center position of frame Img21 captured by the camera. The image processing device sends the position information of the object Tg to the servo controller.

[0139] The servo controller generates a current control command based on the position information of the object Tg sent from the image processing device, which moves the trolley Tr 1m to the right of the paper. The servo controller outputs the generated current control command to the motor driver. The motor driver drives the servo motor based on the current control command, causing the trolley Tr to move 1m to the right of the paper.

[0140] The object Tg moves 20cm to the right of the paper due to interference during the period from the exposure center time t=11 to the exposure center time t=12.

[0141] The camera captures the object's Tg at the center of the exposure time t=12. The image processing unit detects the object's Tg from the frame Img22 captured by the camera. Figure 7 In frame Img22 shown, the image processing device detects the object Tg at a position offset 70cm to the right of the paper relative to the estimated position Tg22 of the object Tg. The image processing device sends the position information of the object Tg to the servo controller.

[0142] At the center of exposure time t=12, the servo controller reads the encoder value from the encoder. The servo controller then notifies the image processing unit that the carriage Tr has moved 50cm to the right of the paper at the center of exposure time t=12.

[0143] Additionally, based on the object position information Tg sent from the image processing unit, the servo controller performs a correction to further adjust the position of the carriage Tr (i.e., the target position) at the next exposure center time t=13 to the right of the paper by 20cm. The servo controller generates a current control command to move the carriage Tr 70cm to the right of the paper. The servo controller outputs the generated current control command to the motor driver. The motor driver drives the servo motor based on the current control command, causing the carriage Tr to move 70cm to the right of the paper.

[0144] The camera captures an image of object Tg at time t=13. The image processing device detects object Tg from frame Img23 captured by the camera. Figure 7 In the frame Img23 shown, the image processing device detects the object Tg at the estimated position Tg23 of the object Tg.

[0145] When the movement of the carriage Tr based on the position of the object Tg detected at time t=11, that is, the movement to the right of the paper by 1.2m, is completed, the servo motor at time t=13 stops. The servo controller generates a notification indicating that the movement of the carriage Tr to the right of the paper by the servo motor has been completed, and sends the notification to the image processing device.

[0146] As described above, when the camera capture time (exposure center time t=11, 12, 13) is synchronized with the acquisition time of the servo motor's position information, the servo controller sends (notifies) the position of the carriage Tr to the image processing device at each exposure center time, and acquires the position information of the object Tg from the image processing device. Therefore, the servo controller can correct the target position of the carriage Tr based on the latest object Tg position. The servo controller can then drive the servo motor based on the corrected target position of the carriage Tr.

[0147] Therefore, even when interference is applied to the object Tg, the servo controller can drive the servo motor in response to the positional offset of the object Tg caused by the interference (moving 20cm to the right of the paper), so that the trolley Tr and the glove Gr can follow the movement of the object Tg and capture the object Tg using the glove Gr.

[0148] Furthermore, the image processing device can calculate the amount of carriage movement (Tr) between the midpoints of two consecutive exposures based on encoder values ​​fed back from the servo controller. This allows it to determine the impact of interference on the object's Tg (e.g., the amount of movement, speed, and acceleration of the object's Tg caused by the interference). By determining the impact of interference on the object's Tg, the image processing device can improve the accuracy of estimating the position of the object's Tg captured in the next frame. Improving the accuracy of the estimated position further reduces the image processing area for detecting the object's Tg in the next frame, thereby further shortening the image processing time required.

[0149] (A variation of Implementation Method 1)

[0150] Regarding the motor control system 100 according to Embodiment 1, the following example is shown: any one of the first servo controllers S1 to Nth servo controllers SN or the first image processing device Im1 to Kth image processing device ImK is set as the host, and time synchronization of the first servo controllers S1 to Nth servo controllers SN and the first image processing device Im1 to Kth image processing device ImK is performed. Regarding the motor control system 200 according to a variation of Embodiment 1, the following example is described: an external host M1 that is communicatively connected to the first servo controllers S1 to Nth servo controllers SN and the first image processing device Im1 to Kth image processing device ImK is set as the host, and time synchronization of the first servo controllers S1 to Nth servo controllers SN and the first image processing device Im1 to Kth image processing device ImK is performed.

[0151] Furthermore, the motor control system 200 according to the variation of Embodiment 1 has a structure that is substantially the same as that of the motor control system 100 according to Embodiment 1. Therefore, in the following description of the motor control system 200 according to the variation of Embodiment 1, the same reference numerals are used to refer to structures that are the same as those in the motor control system 100 according to Embodiment 1, and the description is omitted.

[0152] Reference Figure 8 The overall structure of the motor control system 200 involved in the variation of Embodiment 1 will be described. Figure 8 This is a block diagram illustrating an example of the internal structure of a motor control system 200 according to a variation of Embodiment 1.

[0153] The motor control system 200 is configured to include an external host M1, N servo controllers (specifically, the first servo controller S1 to the Nth servo controller SN), and K image processing devices (specifically, the first image processing device Im1 to the Kth image processing device ImK).

[0154] The motor control system 200 performs time synchronization via an external host M1. Through time synchronization performed by the external host M1, the motor control system 200 enables each of the first servo controllers S1 to Nth servo controllers SN to synchronize and cooperate with each of the first image processing devices Im1 to Kth image processing devices ImK. Based on the image processing results (i.e., the position of the object) of each of the first image processing devices Im1 to Kth image processing devices ImK, the first servo controllers S1 to Nth servo controllers SN respectively perform object tracking.

[0155] Each of the first servo controllers S1 to the Nth servo controller SN is connected to the external host M1, other servo controllers besides itself, and the first image processing devices Im1 to the Kth image processing devices ImK in a manner that enables the transmission and reception of data.

[0156] Each of the first image processing devices Im1 to the Kth image processing device ImK is connected to the external host M1, other image processing devices besides this image processing device, and the first servo controller S1 to the Nth servo controller SN in a manner that enables data transmission and reception.

[0157] As an example of an external device, the external host M1 is connected to each of the first servo controllers S1 to Nth servo controllers SN and each of the first image processing devices Im1 to Kth image processing devices ImK in a manner that enables data transmission and reception. At the same time, it sends synchronization time information to each of the first servo controllers S1 to Nth servo controllers SN and each of the first image processing devices Im1 to Kth image processing devices ImK to perform time synchronization.

[0158] Next, refer to Figure 9 An example of the operation process of the motor control system 200 involved in a variation of Embodiment 1 will be described. Figure 9 This is a sequence diagram illustrating an example of the operation process of the motor control system 200 according to a variation of Embodiment 1. Furthermore, Figure 9 The operation process of the motor control system 200 shown is as follows: Figure 3 The operation process of the motor control system 100 shown is roughly the same, so only the different processing (step St21) will be described.

[0159] The motor control system 200 executes step St21 and the first loop process Lp11. Furthermore, the number of cycles for the first loop process Lp11 can be set arbitrarily based on the time synchronization interval of step St21.

[0160] The external host M1 sends synchronized timing information to other image processing devices (first image processing device Im1 to Kth image processing device ImK) and first servo controller S1 to Nth servo controller SN, and repeatedly performs time synchronization (St21) on each device at predetermined intervals. Here, the motor control system 200 can determine that the time synchronization process has failed if the synchronization timing information in step St21 cannot be received by each device simultaneously, and thus transfer to the first loop process Lp11, delaying the time synchronization process to the next step St21.

[0161] Based on the above description, the motor control system 200 involved in the modified example of embodiment 1 can follow the object in real time through each of the first servo motor Sm1 to the Nth servo motor SmN.

[0162] Based on the above description, the motor control system 100 and 200 (an example of a synchronous control system) involved in Embodiment 1 and its variations include at least one image processing device (first image processing device Im1 to Kth image processing device ImK) and at least one servo controller. The at least one image processing device is capable of acquiring frames of the object captured by a camera (first camera C1 to Kth camera CK) that captures the object (an example of a subject). The at least one servo controller is capable of communicating with the image processing device to control the servo motor (first servo motor) that moves the position of the camera. The motor control system 100 and 200 control the Nth servo motor (Sm1 to SmN) and perform the following processes: performing time synchronization based on the synchronization time; performing a first process and a second process, in which the first process determines the exposure center time (an example of the center time) of the camera exposure period based on the synchronization time, and captures the object based on the determined exposure center time; in the second process, the position of the servo motor at the center time is obtained based on the synchronization time; the object is detected from the frame based on the position of the servo motor at the center time and the frame of the captured object; and the servo motor is driven based on the position of the detected object.

[0163] Therefore, even for frames that are blurred, the motor control systems 100 and 200 according to Embodiment 1 and its variations can obtain the time of image capture of the object at the center of the frame and the position of the servo motor at the time of image capture, thus enabling more accurate measurement of the object's position. Furthermore, the motor control systems 100 and 200 can control the servo motor to follow the object based on the measured position of the object.

[0164] Furthermore, the motor control systems 100 and 200 of Embodiment 1 and its variations calculate the position correction amount of the servo motor based on the difference between the position of the object at the central moment and the position of the servo motor. Therefore, the motor control systems 100 and 200 of Embodiment 1 and its variations can control the servo motor to follow the object based on the position correction amount of the servo motor calculated at the moment of image capture of the object.

[0165] Furthermore, the motor control systems 100 and 200 of Embodiment 1 and its variations measure the moving speed of the object based on the difference between the position of the object and the position of the servo motor at the central moment of exposure. Therefore, by capturing an image and acquiring the position of the servo motor at the central moment of exposure, the motor control systems 100 and 200 of Embodiment 1 can obtain information such as the moving speed, amount of movement, and acceleration of the object. Thus, the motor control systems 100 and 200 can control the servo motor to follow the position of the moving object.

[0166] Furthermore, the motor control systems 100 and 200 of Embodiment 1 and its variations measure the interference acting on the object based on the difference between the position of the object and the position of the servo motor at the central exposure time. Therefore, by capturing an image and acquiring the position of the servo motor at the central exposure time, the motor control systems 100 and 200 of Embodiment 1 can obtain information such as the object's moving speed, amount of movement, and acceleration caused by the interference acting on the object. Thus, even when interference acts on the object, the motor control systems 100 and 200 can control the servo motor to follow the object's position.

[0167] Furthermore, in Embodiment 1 and its variations, either the image processing device or the servo controller of the motor control system 100 and 200 sends a synchronization time to an image processing device or servo controller other than the device itself. Thus, the motor control system 100 and 200 in Embodiment 1 and its variations are capable of performing time synchronization between the first image processing device Im1 to the Kth image processing device ImK and the first servo motor Sm1 to the Nth servo motor SmN.

[0168] Furthermore, the motor control system 100 according to Embodiment 1 also includes an external host M1 (an example of an external device), which is communicatively connected to the image processing device and the servo controller. The external host M1 sends synchronization times to the image processing device and the servo controller respectively. Thus, the motor control system 100 according to Embodiment 1 is capable of performing time synchronization between the first image processing device Im1 to the Kth image processing device ImK and the first servo motor Sm1 to the Nth servo motor SmN.

[0169] Furthermore, the camera of the motor control system 200 involved in the variation of Embodiment 1 has a frame rate of 960fps or higher. Therefore, the motor control system 200 involved in the variation of Embodiment 1 can adjust the position of the servo motor at a period of 1kHz based on the position of the detected object, thus enabling the servo motor to be adjusted to follow the position of the object.

[0170] Furthermore, according to the above description, the motor control system 100 and 200 (an example of a synchronous control system) involved in Embodiment 1 and its variations include at least one image processing device (first image processing device Im1 to Kth image processing device ImK) and at least one servo controller (an example of a controller, namely first servo controller S1 to Nth servo controller SN). The at least one image processing device is capable of acquiring frames of the subject (first camera C1 to Kth camera CK) captured by a camera that captures the subject (an example of a photographic object). The at least one servo controller is capable of communicating with the image processing device to control the servo motors (first servo motor Sm1 to Nth servo motor SmN) that move the position of the camera. The image processing device and the servo controller perform time synchronization based on the synchronization time. The image processing device performs a first process in which the center time of the camera's exposure period is determined based on the synchronization time, a frame captured by exposure based on the determined center time is acquired from the camera, and the subject is detected from at least a portion of the detection pixels in the frame. The servo controller performs a second process in which it acquires the position of the servo motor at a central time based on the synchronization time and sends it to the image processing device. The image processing device, based on the servo motor position at the central time and the captured frame of the object, detects the position of the object from the frame and sends it to the servo controller. The servo controller then drives the servo motor based on the received position of the object.

[0171] Therefore, even for frames that are blurred, the motor control systems 100 and 200 according to Embodiment 1 and its variations can obtain the time of image capture of the object at the center of the frame and the position of the servo motor at the time of image capture, thus enabling more accurate measurement of the object's position. Furthermore, the motor control systems 100 and 200 can control the servo motor to follow the object based on the measured position of the object.

[0172] Based on the above description, the motor control system 100 and 200 (an example of a system) involved in Embodiment 1 and its variations include at least one image processing device (first image processing device Im1 to Kth image processing device ImK) and at least one servo controller (an example of a controller, namely first servo controller S1 to Nth servo controller SN). The at least one image processing device is capable of acquiring frames of the object captured by a camera (first camera C1 to Kth camera CK) that captures the object (an example of a subject). The at least one servo controller is capable of communicating with the image processing device and controlling the servo motors (first servo motor Sm1 to Nth servo motor SmN) that move the position of the camera. The motor control system 100 and 200 perform the following processes: performing time synchronization based on a synchronization time (an example of a reference time); capturing the object at the exposure center time (an example of a predetermined time) based on the synchronization time; acquiring the position information of the servo motor at the exposure center time based on the synchronization time; and detecting the object from at least a portion of the pixels of the frame based on the captured object frame and the position information of the servo motor.

[0173] Therefore, the motor control systems 100 and 200 of Embodiment 1 and its variations can control the servo motor to follow the object based on the position of the object detected from the frame captured by the camera and the position of the servo motor when the frame was captured.

[0174] Furthermore, the motor control systems 100 and 200 of Embodiment 1 and its variations enable the reading of at least a portion of pixels based on the position information of the servo motor and the detection of objects based on the read-out at least a portion of pixels to be performed in parallel. Thus, the motor control systems 100 and 200 of Embodiment 1 and its variations can, for example, perform the reading of pixels to be detected next while detecting objects from at least a portion of pixels, thereby enabling object detection to be performed without waiting for the reading of all pixels in the frame, and shortening the time required for object detection (image processing). Additionally, since the motor control systems 100 and 200 perform pixel reading and object detection based on the position information of the servo motor, the total number of pixels processed before object detection can be reduced, thereby shortening the time required for object detection (image processing) and reducing the processing burden of image processing.

[0175] Furthermore, the motor control systems 100 and 200 of Embodiment 1 and its variations estimate the position of an object captured in a frame based on the difference between the position of the servo motor corresponding to the frame and the position of the servo motor corresponding to the previous frame captured consecutively with that frame. Based on the estimated position of the object, a subset of pixels to be detected is determined. Therefore, the motor control systems 100 and 200 of Embodiment 1 and its variations can estimate the position of an object captured in the latest frame of two consecutively captured frames with higher accuracy based on the amount of movement of the servo motor between the two frames captured consecutively. Since the motor control systems 100 and 200 select pixels to be detected (image processing) based on the estimated position of the object, the total number of pixels processed before object detection can be reduced, thus shortening the time required for object detection (image processing) and reducing the processing burden of image processing.

[0176] Furthermore, the motor control systems 100 and 200 of Embodiment 1 and its variations perform image processing using a hill-climbing method on pixels. If it is determined that no object is detected from a pixel, the hill-climbing method is used to repeatedly determine at least a portion of the pixels to be read next, and to detect the object based on that portion of the pixels until the object is detected. Therefore, the motor control systems 100 and 200 of Embodiment 1 select pixels near pixels with a higher probability of being detected as objects from among the pixels that have undergone image processing as pixels to be processed for object detection (image processing). This reduces the total number of pixels processed before object detection, shortens the time required for object detection (image processing), and reduces the processing burden of image processing.

[0177] Furthermore, the motor control systems 100 and 200 of Embodiment 1 and its variations measure the distance between the camera's image sensor and the object based on the position of the servo motor corresponding to the frame and the position of the servo motor corresponding to the previous frame captured consecutively with the frame. Thus, the motor control systems 100 and 200 of Embodiment 1 and its variations can measure the position of the object in the direction perpendicular to the camera's image sensor based on the positions of the servo motors in two consecutively captured frames, using two frames captured instead of two frames captured by the stereo camera.

[0178] Furthermore, in Embodiment 1 and its variations, either the image processing device or the servo controller of the motor control system 100 and 200 sends a synchronization time to an image processing device or servo controller other than the device itself. Thus, the motor control system 100 and 200 in Embodiment 1 and its variations are capable of performing time synchronization between the first image processing device Im1 to the Kth image processing device ImK and the first servo motor Sm1 to the Nth servo motor SmN.

[0179] Furthermore, the motor control system 100 according to Embodiment 1 also includes an external host M1 (an example of an external device), which is communicatively connected to the image processing device and the servo controller. The external host M1 sends synchronization times to the image processing device and the servo controller respectively. Thus, the motor control system 100 according to Embodiment 1 is capable of performing time synchronization between the first image processing device Im1 to the Kth image processing device ImK and the first servo motor Sm1 to the Nth servo motor SmN.

[0180] Furthermore, the camera of the motor control system 200 involved in the variation of Embodiment 1 has a frame rate of 960fps or higher. Therefore, the motor control system 200 involved in the variation of Embodiment 1 can adjust the position of the servo motor at a period of 1kHz based on the position of the detected object, thus enabling the servo motor to be adjusted to follow the position of the object.

[0181] Furthermore, according to the above description, the motor control system 100 and 200 (an example of an image processing system) involved in Embodiment 1 and its variations includes at least one image processing device (first image processing device Im1 to Kth image processing device ImK) and at least one servo controller (an example of a controller, namely first servo controller S1 to Nth servo controller SN). The at least one image processing device is capable of acquiring frames of the subject (first camera C1 to Kth camera CK) captured by a camera photographing the subject (an example of a photographed object). The at least one servo controller is capable of communicating with the image processing device to control the servo motors (first servo motor Sm1 to Nth servo motor SmN) that move the position of the camera. The image processing device and the servo controller perform time synchronization based on a synchronization time (an example of a reference time). The image processing device photographs the subject at the exposure center time (an example of a predetermined time) based on the synchronization time. The servo controller acquires the position information of the servo motor at the exposure center time based on the synchronization time and sends it to the image processing device. The image processing device detects the position of the object from at least a portion of the pixels in the captured subject frame and sends this information to the servo controller. The servo controller then drives the servo motor based on the received position of the object.

[0182] Therefore, the motor control systems 100 and 200 of Embodiment 1 and its variations can control the servo motor to follow the object based on the position of the object detected from the frame captured by the camera and the position of the servo motor when the frame was captured.

[0183] Various embodiments have been described above with reference to the accompanying drawings, but this disclosure is not limited to these examples. Those skilled in the art will appreciate that various modifications, alterations, substitutions, additions, deletions, and equivalents can be conceived within the scope of the claims, and these also fall within the technical scope of this disclosure. Furthermore, the constituent elements in the above-described embodiments can be arbitrarily combined without departing from the spirit of the invention.

[0184] Furthermore, the contents of Japanese patent applications filed on March 29, 2023 (Japanese Patent Application No. 2023-054259 and Japanese Patent Application No. 2023-054260) are incorporated herein by reference.

[0185] Industrial availability

[0186] This disclosure is useful as a method and system for synchronizing a camera with a servo motor to improve the tracking accuracy of an object.

[0187] Explanation of reference numerals in the attached figures

[0188] 10, 20: Ministry of Communications

[0189] 11, 21: Processor

[0190] 12, 22: Memory

[0191] 100, 200: Motor control system

[0192] C1: First Camera

[0193] CK: Camera K

[0194] En1: First encoder

[0195] EnN: Nth encoder

[0196] Im1: First image processing device

[0197] ImK: Kth image processing device

[0198] L1: First Illumination

[0199] LK: Lighting of the Kth Generation

[0200] M1: External host

[0201] Md1: First motor driver

[0202] MdN: Nth motor driver

[0203] S1: First Servo Controller

[0204] SN: Nth servo controller

[0205] Sm1: First servo motor

[0206] SmN: The Nth servo motor.

Claims

1. A synchronization control method, which is a synchronization control method performed by a synchronization control system, wherein the synchronization control system includes at least one image processing device and at least one controller, wherein, The at least one image processing device is capable of acquiring frames of the subject captured by a camera that captures the subject, and the at least one controller is capable of communicating with the image processing device to control the servo motor that moves the position of the camera. In the synchronization control method... Perform time synchronization based on the reference time; The process involves performing a first process and a second process. In the first process, the central moment of the exposure period of the camera is determined based on the reference moment, and the subject is photographed based on the determined central moment. In the second process, the position of the servo motor at the central moment is obtained based on the reference moment. The subject is detected from the frame based on the position of the servo motor at the central moment and the frame in which the subject is captured. as well as The servo motor is driven based on the position of the detected subject.

2. The synchronization control method according to claim 1, wherein, The position correction amount of the servo motor is calculated based on the difference between the position of the subject and the position of the servo motor at the central moment.

3. The synchronization control method according to claim 1, wherein, The moving speed of the subject is measured based on the difference between the position of the subject and the position of the servo motor at the central moment.

4. The synchronization control method according to claim 1, wherein, The interference acting on the subject is measured based on the difference between the position of the subject and the position of the servo motor at the central time.

5. The synchronization control method according to claim 1, wherein, Either the image processing device or the controller sends the reference time to the image processing device or the controller outside of this device.

6. The synchronization control method according to claim 1, wherein, The synchronization control system also includes an external device, which is communicatively connected to the image processing device and the controller. The external device sends the reference time to the image processing device and the controller, respectively.

7. The synchronization control method according to claim 1, wherein, The camera has a frame rate of 960fps or higher.

8. A synchronous control system, comprising at least one image processing device and at least one controller, wherein, The at least one image processing device is capable of acquiring at least one frame of the subject captured by the camera, and the at least one controller is capable of communicating with the image processing device to control the servo motor that moves the position of the camera. The image processing device and the controller perform time synchronization based on a reference time. The image processing device performs a first process, in which it determines the center time of the exposure period of the camera based on the reference time, acquires a frame captured by the camera based on the determined center time, and detects the subject from at least a portion of the detection pixels in the frame. The controller performs a second process, in which the position of the servo motor at the central time is obtained based on the reference time and sent to the image processing device. The image processing device detects the position of the subject from the frames captured at the central moment based on the position of the servo motor and sends the position to the controller. The controller drives the servo motor based on the position of the subject being transmitted.

Images