Image processing system, and image processing apparatus

The image processing apparatus addresses inefficiencies in optimizing parameters by using real-time and recording modes to synchronize and process image data efficiently, enhancing the generation of high-quality virtual viewpoint images without requiring system reconfiguration.

JP2024159105APending Publication Date: 2024-11-08CANON KK
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2023074879
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-04-28
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing image processing systems face inefficiencies in optimizing image processing parameters due to changing shooting conditions and subject conditions, making it difficult to evaluate and adjust these parameters consistently across multiple workflows, and require reconfiguration when switching between imaging and processing steps.

Method used

An image processing apparatus that acquires and processes image data from multiple imaging devices, utilizing real-time and recording modes to efficiently manage and synchronize data, allowing for flexible operation without reassembly, and includes features for time synchronization and image processing optimization.

Benefits of technology

Enables efficient image processing tuning across multiple imaging devices by synchronizing and optimizing parameters, reducing the need for system reconfiguration and improving the generation of high-quality virtual viewpoint images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024159105000001_ABST
    Figure 2024159105000001_ABST
Patent Text Reader

Abstract

To improve the efficiency of an image processing tuning step in an image processing system that processes a plurality of pieces of image data based on a plurality of pieces of imaging data.SOLUTION: An image processing apparatus, which supplies image data to an image processing system that processes a plurality of pieces of image data based on a plurality of pieces of imaging data, acquires first imaging data from imaging means that picks up an image of an imaging object. The image processing apparatus acquires second imaging data from storage means that stores second imaging data picked up by the imaging means. The image processing apparatus performs predetermined image processing on the acquired first imaging data or the acquired second imaging data according to the settings of a mode, to acquire the image data. The image processing apparatus transmits the image data to an external device.SELECTED DRAWING: Figure 4
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to an image processing system and an image processing device that process a plurality of image data based on a plurality of imaging data. [Background technology]

[0002] There is a technology in which a plurality of imaging devices are installed at different positions, and images are captured synchronously from multiple viewpoints, and an image (hereinafter referred to as a "virtual viewpoint image") viewed from any viewpoint designated by a user is generated using a plurality of images (hereinafter referred to as "captured images") obtained by the imaging. The virtual viewpoint image allows a user to view highlight scenes of, for example, soccer or basketball from various angles, and therefore can provide a user with a higher sense of realism than a normal captured image. In order to generate a highly accurate virtual viewpoint image, it is necessary to generate three-dimensional shape data based on captured images captured at the same time by each imaging device, and therefore it is necessary to synchronously acquire the captured images captured at the same time by each imaging device. Also, in the field of surveillance camera technology, in order to display captured images captured at the same time by a plurality of imaging devices side by side, it is necessary to synchronously acquire the captured images captured at the same time by each imaging device.

[0003] Patent Document 1 discloses a technology related to a system for generating a virtual viewpoint image, and Patent Document 2 discloses a technology for synchronously playing back a plurality of images captured by a plurality of imaging devices. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-50593 A [Patent Document 2] JP 2023-3096 A Summary of the Invention [Problem to be solved by the invention]

[0005] In an image processing system, in order to obtain a good virtual viewpoint image, it is necessary to vary image processing according to shooting conditions (illumination conditions and sunlight conditions), subject conditions (number / size of subjects, arrangement of still / moving objects), etc. Specifically, it is necessary to optimize image processing parameters in each image processing device connected to each of a plurality of imaging devices. For example, in the system described in Patent Document 1, a foreground / background separation process is performed as a preprocessing for generating a virtual viewpoint image, and a threshold value required for the process corresponds to this. In addition, parameters required for a blur correction process for reducing image blur caused by camera vibration also correspond to the image processing parameters of the image processing device. In optimizing the image processing parameters, there are cases where a workflow from shooting the subject to image processing, image generation, and image evaluation is executed multiple times (image processing tuning process). However, if the subject conditions / sunlight conditions change while the workflow is executed multiple times, the output image is also changed each time, making it difficult to evaluate the image processing parameters of the image processing device. On the other hand, there are also cases where it is practically difficult to keep / reproduce the shooting conditions / subject conditions constant during multiple workflows.

[0006] Furthermore, it is desirable in terms of system operation that there be no need to rearrange the system when switching between the image processing tuning process and the actual shooting process, such as changing or removing equipment, or reconnecting cables, etc. The technology disclosed in Patent Document 2 discloses a synchronous playback technology for captured images that can be applied to each of the actual shooting process and the image processing tuning process, but does not disclose a synchronous playback technology required for each of the actual shooting process and the image processing tuning process using the same system / device.

[0007] The present invention has been made in consideration of at least one of the above-mentioned problems, and aims to make the image processing tuning process in an image processing system that processes multiple image data based on multiple imaging data more efficient. [Means for solving the problem]

[0008] In order to achieve the above-mentioned object, an image processing device according to one embodiment of the present invention is an image processing device that supplies image data to an image processing system that processes multiple image data based on multiple imaging data, and has a first acquisition means that acquires first imaging data from an imaging means that images an imaging target, a second acquisition means that acquires the second imaging data from a storage means that stores second imaging data captured by the imaging means, an image processing means that acquires image data by performing a predetermined image processing on the first imaging data acquired by the first acquisition means or the second imaging data acquired by the second acquisition means depending on a mode setting, and an image data transmission means that transmits the image data acquired by the image processing means to an external device. Effect of the Invention

[0009] According to the present invention, it is possible to make the image processing tuning process in an image processing system having a plurality of image capturing devices more efficient. [Brief description of the drawings]

[0010] [Figure 1] An example of an image processing system to which the present invention can be applied [Diagram 2] An example of sensor system layout and connection [Diagram 3] Functional block diagram of the image processing device [Figure 4] Flow chart diagram for control station [Diagram 5] Sequence diagram of the image processing system in real-time playback mode [Figure 6] Sequence diagram of the image processing system in recording and playback mode DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

[0012] An example of an image processing system according to the present invention will be described. FIG. 1 is a block diagram showing an example of the configuration of an image processing system. The image processing system 1 includes a plurality of sensor systems 10 (10a-10x), an image computing server 13, a controller 16, a switching hub 18, an end user terminal 17, and a time server 12. The controller 16 includes a control station 14 and a virtual camera operation UI 15. Each sensor system 10 (10a-10x) includes an image processing device 100 (100a-100x), an imaging device 11 (11a-11x), and a storage device 110 (110a-110x). In the image processing system 1, each imaging device 11 is disposed at a different position in a facility such as a stadium or a concert hall, and captures an image of an object to be imaged.

[0013] Each sensor system 10 will be described as having one image processing device 100, one imaging device 11, and one storage device 110, but the number of image processing devices 100, imaging devices 11, and storage devices 110 in each sensor system 10 is not limited to this. For example, the number of image processing devices 100 in each sensor system 10 may be two or more. The sensor system 10 may be configured to have other devices in addition to the imaging device 11, the image processing device 100, and the storage device 110. The imaging device 11, the image processing device 100, and the storage device 110 will be described as separate devices exteriorly mounted in different housings, but the imaging device 11, the image processing device 100, and the storage device 110 may be configured as an integrated device exteriorly mounted in the same housing. Each sensor system 10 will be described as having the same configuration, but each sensor system 10 may have a configuration in which some or all of the multiple sensor systems 10 are different from each other. Hereinafter, unless otherwise specified, the image processing devices 100a-100x, imaging devices 11a-11x, and storage devices 110a-110x of the sensor systems 10a-10x will be described without distinction as the imaging device 11, image processing device 100, and storage device 110. Also, the image processing system will be described as having 24 sensor systems 10, but the number of sensor systems 10 may be less than 24 or more than 24, so long as it is two or more. Hereinafter, unless otherwise specified, the 24 sensor systems 10a-10x will be described without distinction, simply described as sensor systems 10.

[0014] The imaging device 11 has a lens and an image sensor, and outputs an image (hereinafter referred to as a "captured image") obtained by capturing an image of an object with the image sensor as an image signal. The imaging device 11 may have a microphone in addition to the lens and image sensor, and may output sound collected by the microphone as an audio signal in addition to the image signal. The imaging device 11 may be fixed to a camera platform that can change the imaging direction of the imaging device 11 by rotating the imaging device 11 in a pan, tilt, or roll direction. Hereinafter, unless otherwise specified, the term "image" will be described as including the concepts of moving images and still images. In other words, the image processing system 1 according to the present invention is capable of processing both still images and moving images.

[0015] The image processing device 100 can perform processing in two modes. The first mode is referred to as a real-time playback mode and will be described here. In the real-time playback mode, the image processing device 100 receives an image signal output by the imaging device 11, generates image information corresponding to the image signal, and records the image information in the storage device 110. In parallel with the previous recording process, the image processing device 100 transmits processed image information obtained by performing a predetermined image processing on the image information to the image computing server 13 as a communication packet. This is the real-time playback mode. The second mode is referred to as a recording / playback mode and will be described here. In the recording / playback mode, the imaging device 11 does not necessarily need to be operating. The image information recorded in advance in the storage device 110 in the real-time playback mode or the like is read, and the processed image information obtained by performing a predetermined image processing is transmitted to the image computing server 13 as a communication packet. This is the recording / playback mode. The detailed functions of the image processing device 100 and the details of the two modes will be described later.

[0016] The block diagram shown in FIG. 1 shows an example of the connection form of each block constituting the image processing system 1. That is, in the image processing system 1, a plurality of image processing devices 100 are daisy-chained using cables 19a-19w, so that they can communicate with each other. Also, the image processing device 100a, the image computing server 13, the time server 12, and the control station 14 are connected to each other via a switching hub 18 and cables 18a, 12a, 13a, and 14a so that they can communicate with each other. Note that the image processing system 1 is not limited to the connection form shown in FIG. 1. For example, all the sensor systems 10 may be directly connected to the switching hub. In FIG. 1, the cables 18a, 19a-w are described below as being configured by IEEE standard-compliant GbE (Gigabit Ethernet) or 10GbE, which is Ethernet (registered trademark), but other communication methods may be used. For example, they may be configured by an interconnect such as Infiniband, or industrial Ethernet, or may be configured by a combination of these. Furthermore, the configuration is not limited to these, and may be other types of configuration.

[0017] Next, an example of the arrangement and connection method of the sensor system 10 will be described with reference to FIG. 2(a) and FIG. 2(b). The imaging devices 11 of each sensor system 10 are arranged at different positions so as to surround a field for soccer or the like. Each imaging device 11 is arranged so as to be able to image an object to be imaged, such as a player playing on the field, from a plurality of different directions. The image processing devices 100 of the plurality of sensor systems 10 are connected to each other by a daisy chain. By connecting the image processing devices 100 to each other by a daisy chain, the number of connection cables can be reduced, or the labor of wiring work can be saved. FIG. 2(a) shows a form in which all the image processing devices 100a-100x are cascade-connected to form one daisy chain, but the connection form of the image processing system is not limited to this. For example, the plurality of sensor systems 10 may be divided into two groups of 24 sensor systems 10 as shown in FIG. 2(b), and the sensor systems 10, i.e., the image processing devices 100, may be connected to each other by a daisy chain in units of the divided groups. The number of groups into which the sensor systems 10 are divided is not limited to two, and may be three or more. The connection form in which the sensor systems 10 are divided into several groups is particularly effective in a stadium or the like in which spectator seats are arranged in a cone shape around the field. For example, it is possible to consider a case in which spectator seats are arranged across several floors, and a sensor system 10 is arranged on each floor on which the spectator seats are arranged. In this case, for example, the sensor systems 10 arranged on each floor are treated as one group, and the sensor systems 10 in the same group are connected to each other in a daisy chain. This connection form can simplify wiring and make the system more flexible. Furthermore, this connection form can divide data flowing on the network, and it is possible to increase the amount of image information output by the image processing device 100. The form in which the sensor systems 10 are divided into several groups and connected in a daisy chain is not limited to the form in which the sensor systems 10 arranged on each floor are treated as one group. For example, the sensor systems 10 may be arranged so as to surround the periphery of the field with two groups.In this case, each group may be composed of sensor systems 10 arranged at positions corresponding to half the circumference of the field, and the sensor systems 10 in the same group may be connected to each other in a daisy chain. This type of connection can simplify wiring and make the system more flexible. In the following description, the multiple image processing devices 100 are connected to each other in a daisy chain in the same connection form as shown in FIG. 2(a).

[0018] Returning to the description of FIG. 1, the virtual camera operation UI 15 receives a signal indicating the position and attitude of the virtual camera based on the input of the operator from an operation input device (not shown), and outputs information corresponding to the signal as viewpoint information of the virtual camera to the image computing server 13. Note that the viewpoint information of the virtual camera is not designated only based on the input of the operator of the controller 16. For example, it may be designated automatically based on the analysis result of the image based on the captured image acquired by the imaging device 11. The control station 14 manages the operation state of each block constituting the image processing system 1, and sets or controls parameters. The virtual camera operation UI 15 provides a user interface for designating the position and attitude of the virtual camera used when generating a virtual viewpoint image. Note that the virtual camera is an imaging device that acquires an image of an object (virtual viewpoint image) from a virtual viewpoint different from any of the viewpoints of the imaging devices 11a-11x. The virtual camera is a virtual imaging device defined to generate a virtual viewpoint image, and does not have an actual entity. Furthermore, by providing the virtual camera operation UI 15 with an operation input device such as a mouse or a joystick, the operator of the controller 16 can specify the position and orientation of the virtual camera using the operation input device. For example, the position and orientation of the virtual camera may be specified by the operator of the controller 16 specifying a virtual camera path. The virtual camera path is a string of information that indicates the position and orientation of the virtual camera for each frame.

[0019] The image computing server 13 is an information processing device configured by a computer or the like. The image computing server 13 receives image information output by each image processing device 100 as one image set, and acquires captured images (hereinafter referred to as "multiple viewpoint images") indicated by each image information in one image set. The image computing server 13 also acquires viewpoint information output by the virtual camera operation UI 15, and generates a virtual viewpoint image that looks like an image captured by a virtual camera based on the multiple viewpoint images and the viewpoint information. That is, the image computing server 13 can generate a virtual viewpoint image that looks as if it was captured by an imaging device (21 in FIG. 2(a)) located near the goal, for example. A method for generating a virtual viewpoint image based on multiple viewpoint images and viewpoint information is well known, so a detailed description will be omitted.

[0020] The image computing server 13 outputs the generated virtual viewpoint image to the end user terminal 17. The image computing server 13 may compress and encode the generated virtual viewpoint image by standard image encoding such as H.264 or HEVC (High Efficiency Video Coding), and then output the image to the end user terminal 17. For example, the image computing server 13 may output the compressed and encoded virtual viewpoint image by standard streaming distribution using MPEG-DASH (Dynamic Adaptive Streaming over HTTP) protocol or the like. Note that the streaming distribution by MPEG-DASH is an example, and the method of distributing the virtual viewpoint image is not limited to MPEG-DASH. For example, a distribution method such as HLS (HTTP Live Streaming) may be used as the method of distributing the virtual viewpoint image. The image computing server 13 may output the generated virtual viewpoint image to the end user terminal 17 without compression. The image computing server 13 switches the control mode of the image processing depending on whether the number of the image processing device 100 at the last stage that outputs image information to the image computing server 13 is one or two or more. That is, in the image computing server 13, the control mode is switched depending on whether the sensor system 10 is divided into a plurality of groups. Since the image processing devices 100 are connected to each other by a daisy chain as described above, for example, in the case of the connection form shown in FIG. 2(a), the image information output from each image processing device 100 is transmitted to the image computing server 13 as follows. The image information acquired or generated by the image processing device 100a is transmitted to the image processing device 100b via the network 19a. The image processing device 100b transmits to the image processing device 100c the image information generated by the image processing device 100b or acquired from the storage device and the image information transmitted from the image processing device 100a together. Thereafter, the same transmission is repeated up to the image processing device 100x.As a result, information combining image information generated by each of the image processing devices 100a-100w or acquired from a storage device is transmitted from the image processing device 100w to the image processing device 100x via the network 19w. In addition, the image processing device 100x outputs the information transmitted from the image processing device 100w and the image information generated by the image processing device 100x or acquired from a storage device to the image computing server 13. Specifically, the information output from the image processing device 100x, i.e., the information combining image information generated by each of the image processing devices 100a-100x or acquired from a storage device, is transmitted to the image computing server 13 via the switching hub 18. Therefore, when there is one image processing device 100 that outputs image information to the image computing server 13, the image computing server 13 can acquire all image information generated or acquired by each of the multiple image processing devices 100 at the same time. In other words, when the sensor system 10 is not divided into groups, the timing of acquiring all image information corresponding to one image set by the image computing server 13 is synchronized. However, when there are a plurality of image processing devices 100 that output image information to the image computing server 13 (when a plurality of sensor systems 10 are divided into a plurality of groups), the following may occur. For example, the delay time until the image computing server 13 acquires the image information may differ for each lane (route) of the daisy chain. Therefore, the image computing server 13 may need to wait until all image information corresponding to one image set is available before performing subsequent image processing to generate a virtual viewpoint image.

[0021] The end user terminal 17 is configured with a smartphone, a tablet terminal, a personal computer, or the like, and receives the virtual viewpoint image output by the image computing server 13 and displays the virtual viewpoint image. When the virtual viewpoint image output by the image computing server 13 is a compressed and encoded virtual viewpoint image, the end user terminal 17 receives the compressed and encoded virtual viewpoint image output by the image computing server 13 and decodes the virtual viewpoint image. The end user terminal 17 outputs the decoded virtual viewpoint image to a display provided in the terminal or connected to the terminal, and displays the virtual viewpoint image on the display. When the virtual viewpoint image output by the image computing server 13 is an uncompressed virtual viewpoint image, the end user terminal 17 may be a monitor or the like that is a terminal that can receive the uncompressed virtual viewpoint image as a display signal and display the virtual viewpoint image based on the display signal.

[0022] The time server 12 outputs a time that is a reference for the system (hereinafter referred to as "reference time"). The reference time output by the time server 12 is distributed to the image processing devices 100a-100x of the sensor systems 10a-10x via the switching hub 18. For example, the time server 12 periodically outputs the reference time at a predetermined time interval. This concludes the explanation of FIG. 1.

[0023] Next, an example of the configuration of functional blocks of the image processing device 100 will be described with reference to Fig. 3. The processing of each unit of the image processing device 100 is performed by hardware such as an ASIC (application specific integrated circuit) built into the image processing device 100. The processing may also be performed by hardware such as an FPGA (field programmable gate array). Moreover, the processing may also be performed by software using a CPU (Central Processor Unit) or a GPU (Graphic Processor Unit) and a memory.

[0024] The synchronization processing unit 313 has a function of synchronizing the time of the image processing device 100 with the time server 12. Specifically, the synchronization is performed by a method using the IEEE1588 standard. The time synchronization with the time server 12 is performed by using a time synchronization packet 359 exchanged with the time server 12 via the network processing unit 316 and a time stamp 360 output from the network processing unit 316 at the timing of transmitting and receiving the time synchronization packet. In IEEE1588, a PTP (Precision Time Protocol) packet is used for the time synchronization packet, and the synchronization processing unit 313 has a function of processing the PTP packet. Note that the method of time synchronization with the time server 12 is not limited to a method using the IEEE1588 standard, and may be a method using the EtherAVB standard or a predetermined unique protocol. Since the time difference (offset) between the time server 12 and the system time management unit 312 is calculated by the time correction process, a time correction amount is determined so as to reduce the offset, and the correction amount is set in the system time management unit 312.

[0025] The system time management unit 312 is a unit for managing time information of the image processing device 100. It has a counter that can update the system time information 353 over time. It also has a function that allows the synchronization processing unit 313 to correct the counter. There are preferably three correction methods. The first method is a method in which an arbitrary time is set (overwrite correction). Overwrite correction is mainly used in the initial process of synchronization processing. The second method is a method in which a difference, that is, an advance / return amount from the current value indicated by the system time management unit 312, is set (difference correction). Since the first method sets a specified time, a highly accurate correction result cannot always be obtained considering that fluctuations occur in the timing of setting. However, since the second method is given an offset amount, for example, even if the correction process takes a long time, the time of the correction process and the time of the system time management unit 312 are both advanced, so that the expected advance / return amount can be obtained as a result. The third method is a method in which the amount of increment (rate) of the counter is corrected (rate correction). This correction advances the counter by 1.001 seconds per second, or conversely, advances the counter by 0.999 seconds per second. Generally, the update timing of the counter is determined by the clock accuracy of the oscillator in the image processing device 100, but by configuring as described above, it is possible to perform correction in accordance with the oscillator accuracy. By having a sufficient amount of fine correction for the required accuracy, it is possible to achieve high-precision correction. In other words, if the oscillator accuracy is faster than the rate of the absolute time, the increment amount can be reduced to advance the counter so that it approaches the rate of the absolute time. This concludes the explanation of the system time management unit 312.

[0026] The reference timing generation unit 304 outputs the reference timing 352 based on the system time information 353. For example, if the imaging cycle of the imaging device 11 is 59.94 Hz, a signal of that frequency may be output as the reference timing. Note that if the imaging control unit 301 generates necessary control information, the frequency of the reference timing generation unit does not have to be 59.94 Hz, and may be output as 1 Hz Pulse Per Second (PPS).

[0027] The imaging control unit 301 controls the imaging device 11 by supplying imaging control information 351 to the imaging device 11. The imaging control information 351 includes control / setting information such as imaging parameters such as the number of pixels of an image, color depth, frame rate, or white balance, and setting information of an imaging timing signal indicating imaging timing and imaging time. The control of the imaging device 11 may be acquisition of the state of the imaging device 11 such as imaging, stopped, synchronized, or error, control of starting or stopping imaging, or adjustment of focus or zoom. Note that lens adjustment such as focus or zoom may be performed via the imaging device 11, or may be performed directly by connecting the image processing device 100 and the lens when a removable lens is attached to the imaging device 11. The imaging timing signal is generated based on a reference timing 352 and system time information 353 input by the imaging control unit 301 and is supplied to the imaging device 11. The imaging device 11 receives an imaging timing signal output from the image processing device 100 and performs GENLOCK (Generator Lock) on the imaging timing signal, thereby capturing an image at the timing of receiving a synchronization signal from the image processing device 100. That is, the imaging timing of all of the imaging devices 11 is synchronized by the imaging timing signal output from the imaging control unit 301 of each image processing device 100. The time is set by outputting information on the current time synchronized with the time server 12 to the imaging device 11 as, for example, a time code conforming to SMPTE 12M.

[0028] The imaging device 11 outputs an image signal including, as meta information, a time code corresponding to the time when the image was captured (hereinafter referred to as "capture time"), that is, a time code provided from the imaging control unit 301 to the imaging device 11, in addition to information indicating the captured image. The format of the time code is not limited to SMPTE12M, and may be another format. The meta information may include, in addition to the time code, information indicating a sequence number or data type of the captured image, or an identifier indicating an individual imaging device 11. In this way, the imaging control unit 301 included in each image processing device 100 can obtain an image signal indicating captured images captured by all imaging devices 11 in a highly accurate synchronized manner.

[0029] The image output unit 302 outputs the captured image and the time code output from the image capture device 11 to the storage device 110, and causes the storage device 110 to store the captured image and the time code.

[0030] The image acquisition unit 303 acquires image information based on the time information from the storage device 110, which stores image information and time information in association with each other. The image of which time code is to be acquired is specified by the start time code 354 output from the time code designation unit 307.

[0031] Additionally, a time code for ending acquisition may be received. The image acquisition unit 303 acquires images at the timing of playback timing 356 input from the playback control unit 306. The image acquisition unit 303 may acquire time information corresponding to the image information in addition to the image information.

[0032] The input switching unit 305 switches between the captured image data of the captured image output from the imaging device 11 and the recorded image data of the captured image output from the image acquisition unit as the data to be input to the image processing unit 314. The input switching unit 305 switches according to mode information 357 from the mode designation unit 308, and the captured image data is selected in the real-time playback mode, and the recorded data is selected in the recorded playback mode.

[0033] The image processing unit 314 executes image processing on the image data input from the input switching unit, and outputs the processed image data to the image output unit 315. Since one frame of image data is input each time the playback timing 356 is input, it is necessary to complete the image processing within the cycle of the playback timing 356. Therefore, if it is difficult to perform all the processing within the cycle, the image processing may be configured with a multi-stage pipeline. The image processing executed by the image processing unit 314 includes a foreground / background separation process that separates the foreground and background in the image, and a blur correction process that reduces the blur of the image caused by the camera vibration. Note that the foreground / background separation process is characterized by extracting foreground data from the image data, but may also include extracting background data from the image data. The details of the processing are similar to those described in Patent Document 1, so the explanation will be omitted. The recording / playback mode according to the present invention is used to determine the optimal parameters by changing and trying the parameters set in the image processing unit 314 each time the virtual viewpoint image generation process using the recording / playback mode is performed. The parameters are set from the image processing parameter designation unit 310.

[0034] By executing image processing in the image processing device 100 as in the present invention, the processing load of the image computing server 13 is reduced. For example, by executing the foreground / background separation process in the image processing unit 314, it becomes unnecessary to perform foreground / background separation on all captured images indicated by the image information acquired by the image computing server 13. As a result, the processing load of each component of the image processing system as a whole can be leveled. In addition, by executing the foreground / background separation process in the image processing device 100 and limiting the number of image processing devices 100 to which data is transferred for each foreground and background, it is possible to obtain the effect of reducing the network bandwidth used. Note that the foreground / background separation on the captured image indicated by the image information does not necessarily have to be performed by the image processing unit 314, and may be performed, for example, when the image acquisition unit 303 acquires the image information.

[0035] The playback control unit 306 controls the overall processing timing of the image processing device 100. Specifically, it supplies the processing start timing of the image processing unit 314 and the image output unit 315, which are used in both the real-time playback mode and the recording playback mode. Furthermore, it controls the timing of the image acquisition unit 303, which is used in the recording playback mode, acquiring image information from the storage device 110. The timing cycle is set to the same cycle as the image capture timing during real-time playback. During recording playback, the recorded recording data may be played back at the cycle when it was captured, but it is not limited to this. For example, playback may be performed at a longer cycle, or it may be operated at a cycle shorter than the image capture cycle if the processing time of each processing unit is sufficient. If it is executed at a short cycle, there is an advantage that the evaluation time can be reduced by the fact that the virtual viewpoint image for a certain period is generated earlier. In addition, if it is operated at a long cycle, there is an advantage that, for example, the user can check and evaluate each frame one by one. Regarding the playback time during the recording playback mode, it is preferable to include the setting of the playback cycle during the recording playback mode in the mode information 357 specified by the mode designation unit 308. The playback timing 356 is generated based on the reference timing 352. For example, when the reference timing 352 generates a signal of 59.94 Hz, which is the frame rate during recording, and when the real-time playback mode is used, the frequency of the reference timing 352 can be used as is for the playback timing 356. When operating in the recording playback mode, the reference timing 352 may be multiplied according to the playback cycle at that time. The multiplication process can be easily realized by setting the frequency of the operating clock of the playback control unit 306 to operate at a frequency sufficiently higher than the frequency of the reference timing 352. The start of output of the playback timing 356 is determined by comparing the start time 355 from the start time designation unit 309 with the system time information 353. For example, if the system time information 353 has not reached the start time 355, it is preferable to mask the playback timing 356. This concludes the explanation of the playback control unit 306.

[0036] The image output unit 315 converts the image data processed by the image processing unit 314 into packets that can be transmitted over a network and transmits them to the network processing unit 316. The destination of the transmission is set to the image computing server 13. If the size of the image data exceeds an MTU (Maximum Transmission Unit), which is the maximum datagram size that can be transferred in one packet, the image data is divided into multiple packets and transmitted to the network processing unit 316. Every time the playback timing 356 is input, one frame of image data is input to the image processing unit 315. Therefore, one frame of image data is packetized and transmitted to the network processing unit 316 within the cycle of the playback timing 356.

[0037] The network processing unit 316 has a function of allowing the image processing device 100 to communicate with external packets. If a packet arriving from the cable 18a is addressed to the device itself, the network processing unit 316 transfers the packet to the main control unit 311 or the synchronization processing unit 313 depending on the type of packet. If the packet received from the cable 18a is not addressed to the device itself, the network processing unit 316 performs forward processing to the cable 19a. The same applies to packets received from the cable 19a. The network processing unit 316 also transmits packets generated within the image processing device 100. Packets are input from either the main control unit 311, the synchronization processing unit 313, or the image output unit 315, and are output from the cable 18a or the cable 19a depending on the destination of the packet. If the network processing unit 316 has a function of providing the value of the system time information 353 to the synchronization processing unit 313 as a time stamp 360 at the timing when the network processing unit 316 transmits or receives a time synchronization packet 359, the synchronization processing unit 313 can realize highly accurate time synchronization processing. Furthermore, when forwarding a time synchronization packet, if the network processing unit 316 has a default Transparent Clock (hereinafter, TC) function, the image processing device 100 at the downstream can be time-synchronized with a higher degree of accuracy than the local device. Even if the network processing unit 316 does not have the TC function, the synchronization processing unit 313 can receive all time synchronization packets 359, regardless of whether they are addressed to the local device or another device, and the synchronization processing unit 313 can perform forward processing, thereby realizing the TC function in the synchronization processing unit 313.

[0038] The main control unit 311 is responsible for controlling the entire image processing device 100. It is configured to receive a control packet 362 from the control station 14 and issue instructions to various processing units. It also has a function of transmitting a response / status packet 361 to the control station 14. Depending on the type of the control packet 362 received from the control station 14, the following processes are implemented.

[0039] The time code specification unit 307 acquires the start time code and end time code in the recording / playback mode input by the operator of the controller 16 from the control station 14 as start / end time code information, and sets it in the image acquisition unit 303.

[0040] The mode designation unit 308 specifies the processing mode required for the image processing device 100 through a control packet 362 from the control station 14, and sets mode information 357 in the playback control unit 306 and the input switching unit 305. If the designated mode is the recording / creation mode and a playback cycle is designated by the control packet 362, the playback cycle is also set in the playback control unit 306.

[0041] The start time designation unit 309 analyzes the time when the image acquisition unit 303 starts acquiring image information from the control packet 362, and sets the start time 355 in the reproduction control unit 206. Here, the start time 355 is a time in the future from the system time information 353, and the reproduction control unit 306 of each image processing device 100 waits to output the reproduction timing 356 until the system time information 353 reaches the start time 355. When the system time information 353 reaches the start time 355, the output of the reproduction timing 356 starts. By setting the start times 355 of all image processing devices 100 to the same time, it is possible to cause the image acquisition units 303 to start acquiring image information at the same timing in all image processing devices 100. This can be realized because the system time information 353 is synchronized with the time of the time server 12 in all image processing devices 100 by the synchronization processing unit 313.

[0042] The image processing parameter designation unit 310 is responsible for extracting image processing parameters input by the operator of the controller 16 from a control packet 362 received from the control station 14 and setting the parameters in the image processing unit 314 .

[0043] The time synchronization start instruction unit 358 extracts a time synchronization start instruction input by the operator of the controller 16 or the like from a control packet 362 received from the control station 14, and issues a time synchronization start instruction 363 to the synchronization processing unit 313. An example of the configuration of functional blocks of the image processing device 100 has been described above. In the example of FIG. 3, a method for synchronizing the timing of the reproduction process has been shown, but if strict synchronization is also desired in the imaging process, information obtained by the start time designation unit 309 may be input to the reference timing generation unit 304 or the imaging control unit 301 so that the reference timing 352 or the output start timing of the imaging control information 351 may also be aligned.

[0044] In this way, the image processing device according to the present invention is provided with the mode designation unit 308, the input switching unit 306, etc., and thus it is possible to switch between a real-time playback mode and a recording playback mode. In the real-time playback mode, the imaging data acquired by the imaging device is transmitted to the next image processing device without recording (without going through a storage device). As a result, in the image processing system according to the present invention, it is possible to quickly provide a virtual viewpoint image of the imaging area to the user. In the recording playback mode, the recording data is acquired from the storage device in which the imaging data acquired by the imaging device is recorded as recording data, and the recording data is transmitted to the next image processing device. In the recording playback mode, it is considered that the influence of disturbances is reduced compared to the real-time mode, and a more accurate virtual viewpoint image can be generated by generating the virtual viewpoint image using the recording data. Since the image processing device according to the present invention is configured to be compatible with each of the above-mentioned modes, it is possible to switch to a mode that matches the user's intention, the imaging timing, or the situation of the imaging area without the need to reconfigure the system.

[0045] Next, a description will be given of the procedure for executing the image processing system according to the present invention. Fig. 4 is a flow chart showing an example of the process of the control station 14. First, the control station 14 performs a start-up process for each image processing device 100 via the switching hub 18 (401).

[0046] If the image processing device 100 supports a WOL (Wake On LAN) function, the image processing device 100 is started up in order starting from the image processing device 100a at the head of the daisy chain using a Magic Packet. After the start-up of each image processing device 100 is completed, the control station 14 checks communication with each image processing device 100 (402). For example, an ICMP (Internet Control Message Protocol) packet is sent to each image processing device 100 and a response is confirmed. Next, image processing parameters are set for each image processing device 100 (403). The image processing parameters are stored in a control packet and sent from the control station 14 to all image processing devices 100, and are set in the image processing unit 314 of each image processing device 100. Since differences occur in the imaging data depending on the position of each imaging device 11, different parameters may be set for each image processing unit 314. Next, a command to start time synchronization processing is issued (404). As a result, each image processing device 100 starts time synchronization processing with the time server 12. The system waits until the time synchronization of each image processing device 100 is completed (405). For example, an inquiry may be made about the time synchronization status of each image processing device 100, and the response may be confirmed. Completion of the time synchronization may be determined when the offset is within a predetermined range for a predetermined time, for example, one minute. Next, a mode is specified for each image processing device 100 (406). At the first shooting, real-time playback mode is selected because there is no recorded data yet. Therefore, the control station 14 specifies the real-time playback mode in a control packet specifying the mode and transmits it to each image processing device 100 (408). Next, when the real-time playback mode is specified, the playback start time is set in each image processing device 100 (410), and processing in the real-time playback mode is started at a predetermined time. When the image processing system is operated for the second or subsequent time, since the recorded data has been recorded in the storage device 110 by the real-time playback mode, it is possible to generate a virtual viewpoint image in the recording playback mode. Therefore, after the time synchronization process is completed, each image processing device 100 is set to the recording playback mode (407).This means that the recording / playback mode is specified in a control packet specifying the mode from the control station 14 and transmitted to each image processing device 100. Next, the section is specified (409) based on which period of the entire recorded period the virtual viewpoint image was generated for. Specifically, a start time code and, if necessary, an end time code are set in the control packet and transmitted to all image processing devices 100. Thereafter, the playback start time for the recording / playback mode is specified (410), and playback begins in the same way as in the real-time playback mode. This concludes the explanation of FIG. 4.

[0047] Next, the processing sequence of each functional block of the image processing device 100 will be described in each case of the real-time playback mode and the recording / playback mode with reference to Figs. 5 and 6. First, the real-time playback mode (Fig. 5) will be described. The control station 14 checks communication with the image processing device 100 (S501). The control station 14 checks communication by sending an ICMP packet to the network processing unit 316 (S502) and checking whether there is a response to the packet (S504). The image processing device 100 analyzes the control packet addressed to itself received from the control station 14 in the main control unit 311 (S503) and then performs necessary processing. After checking communication, the control station 14 sends image processing parameter setting information to the network processing unit 316 (S505). The image processing parameter setting information is sent in a control packet (S506), and the main control unit 311 sets the parameters in the image processing unit 314 (S507). Next, the control station 14 transmits an instruction to start time synchronization to the network processing unit 316 (S508). When the network processing unit 316 receives an instruction to start time synchronization from the control station 14, it issues an instruction to start time synchronization to the main control unit 311 (S507). The main control unit 311, which has received the instruction to start time synchronization from the control station 14 via the network processing unit 316, instructs the synchronization processing unit 313 to start time synchronization processing (S510). The time server 12 already starts processing as a synchronization source regardless of the state of the image processing device 100 (S511). Specifically, in this processing, the time server 12 transmits an Announce packet and a Sync packet of a PTP packet (time synchronization packet) to the network processing unit 316. Next, the synchronization processing unit 313, which has been instructed by the main control unit 311 to start synchronization processing, transmits a DelayRequest packet of the PTP packet to the time server 12 via the network processing unit 316. The time server 12, which has received the DelayRequest packet from the network processor 316, transmits a DelayResponse packet to the image processing device 100. A series of PTP packet exchanges is shown in S512.During that time, the synchronization processing unit 313 calculates an offset using the time stamp S513 and the PTP packet S512 generated by the network processing unit 316, determines the amount of time correction based on the offset, and sets the time correction information in the system time management unit 312. The system time management unit 312 always outputs system time information to the reference timing generation unit 304 (S516), but when the time correction information is set (S515), time information according to the correction is output. The reference timing generation unit 304 outputs a reference timing at a predetermined cycle in response to the input of the system time information received from the system time management unit 312 (S517). The imaging control unit 301 that has received the reference timing periodically transmits imaging control information including imaging timing to the imaging device 11 (S518). The imaging device 11 that has received the imaging control information starts imaging and outputs imaging data to the imaging output unit 302 (S519). The imaging data is recorded in the storage device 110 via the imaging output unit 302 (S520). The imaging data is also input to an input switching unit 305 via an imaging output unit 302 .

[0048] The control station 14, which has determined that the synchronization process is complete, transmits information regarding the mode designation to the network processing unit 316 as a mode designation process (S521). If the real-time playback mode is designated here, the information regarding the mode designation is included in a control packet and transmitted (S522). The main control unit 311, which has received the control packet, designates the real-time playback mode to the input switching unit 305. The input switching unit 305, which has received the designation, starts a process of transmitting the imaging data from the imaging device 11 to the image processing unit 314 (S524). The main control unit 311, which has received the information regarding the mode designation, also designates the mode to the playback control unit 306 (S523). The playback control unit 306, which has received the mode designation, operates in the real-time playback mode, and thus goes on standby to output the playback timing according to the imaging cycle (S525). After transmitting the information regarding the mode designation, the control station 14 transmits the start time of real-time playback to the image processing device 100 (S526). The network processor 316 puts the start time information into a control packet and transmits it (S527), and the main controller 311 that receives the control packet sets the start time in the playback controller 306 (S523). The playback controller 306, in which the playback start time has been set, monitors the received system time information, and starts outputting the playback timing when the system time information reaches the playback start time (S529, S530).

[0049] Thereafter, the input switching unit 305, the image processing unit 314, and the image output unit 315 perform processing in the following order by pipeline operation: (1) acquisition of imaging data, (2) image processing of the imaging data, and (3) packetization of the image-processed imaging data. The image processing unit 314, which has received the playback timing, performs image processing on the imaging data (a) received immediately before receiving the playback timing, and outputs the image-processed imaging data at the next playback timing (S531). The image output unit 315, which has received the image-processed imaging data (a), transmits the image data as an image packet to the network processing unit 316 when receiving the next playback timing (S532). The network processing unit 316 performs processing to transmit the image packet received from the image output unit 315 to the image computing server 13, and processing to transmit the image packet received from the cable 19a to the image computing server 13 (S533).

[0050] The real-time playback mode of the image processing system 1 in this embodiment has been described above with reference to FIG.

[0051] Next, an example of the operation of the image processing system 1 in the recording / playback mode will be described with reference to Fig. 6. In the recording / playback mode, since the recorded data from the storage device 110 is used, no processing is required for the imaging device 11, imaging control unit 301, or imaging output unit 302. On the other hand, processing is required for the image acquisition unit 303. The processing up to S517 in Fig. 6 is the same as the processing in Fig. 5, so the description will be omitted.

[0052] The control station 14, which has determined that the synchronization process is complete, transmits information regarding the mode designation to the network processor 316 as a mode designation process (S601). If the recording / playback mode is designated here, the information regarding the mode designation is included in a control packet and transmitted (S602). The main controller 311, which has received the control packet, designates the recording / playback mode to the input switcher 305. The input switcher 305, which has received the designation, enters a state in which it waits for input from the storage device 110 (input reception waiting state) (S604). The main controller 311, which has received the information regarding the mode designation, also designates a mode to the playback controller 306 (S603). The playback controller 306, which has received the mode designation, operates in the recording / playback mode, and therefore goes on standby to output playback timing related to the recording mode (S605). If a special playback cycle is set here, it goes into a standby state for outputting playback timing according to the setting. After transmitting the information regarding the mode designation, the control station 14 designates the period for playing back the recorded data in the recording / playback mode with a time code and transmits it to the image processing device 100 (S606). The main control unit 311 sets the start time code and, if necessary, the end time code in the image acquisition unit 303 (S608). The image acquisition unit 303 specifies the recorded data to be played back first (S608). At this time, the image acquisition unit 303 may acquire the recorded data in advance before receiving the playback timing. After designating the time code, the control station 14 transmits the start time of the recording / playback mode to the image processing device 100 (S610). The network processing unit 316 transmits the start time information in a control packet (S611), and the main control unit 311, having received the control packet, sets the start time in the playback control unit 306 (S612). The playback control unit 306, to which the playback start time has been set, monitors the received system time information, and starts outputting the playback timing when the system time information reaches the playback start time (S613, S614).

[0053] Thereafter, the image acquisition unit 303, input switching unit 305, image processing unit 314, and image output unit 315 perform processing in the following order by pipeline operation: (1) acquisition of captured image data, (2) transfer of the captured image data to the image processing unit 314, (3) image processing of the captured image data, and (4) packetization of the image-processed captured image data. Upon receiving a playback timing (S614), the image acquisition unit 303 acquires recorded image data (a) (S619). Upon receiving the next playback timing (S615), the image acquisition unit 303 outputs recorded image data (a) to the input switching unit 305 (S620). Subsequent processing is the same as in FIG. 5.

[0054] The recording / playback mode of the image processing system 1 in this embodiment has been described above with reference to FIG.

[0055] According to the above-mentioned image processing system, by operating the system in the real-time playback mode, the image computing server 13 can synchronously acquire a plurality of pieces of imaging data synchronously captured by the plurality of imaging devices 11. On the other hand, by operating the system in the recording / playback mode, the image computing server 13 can synchronously acquire a plurality of pieces of recording data stored in the plurality of recording devices 110 without preparing a subject or performing imaging processing. In other words, the image computing server 13 can generate a virtual viewpoint image without being aware of the system mode. In addition, by configuring the system so that it can operate in the real-time playback mode and the recording / playback mode without rearranging the system, the system can be operated efficiently.

[0056] <Other embodiments> In the above embodiment, the image computing server 13 generates three-dimensional shape data based on image data received from the image processing device 100, and generates a virtual viewpoint image, but this is not limited to the above. For example, the image computing server 13 may be configured to generate three-dimensional shape data for generating a virtual viewpoint image. In this case, the image processing system 1 may be configured to generate three-dimensional shape data and provide the three-dimensional shape data to a third party (for example, an external server, terminal, or recording medium such as a USB).

[0057] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for implementing one or more functions.

[0058] It should be noted that, within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0059] The disclosure of each of the above-mentioned embodiments includes the following configurations.

[0060] (Configuration 1) An image processing device that supplies image data to an image processing system that processes a plurality of image data based on a plurality of imaging data, a first acquisition means for acquiring first imaging data from an imaging means for imaging an imaging target; a second acquisition means for acquiring second imaging data captured by the imaging means from a storage means for storing the second imaging data; an image processing means for acquiring image data by performing a predetermined image processing on the first imaging data acquired by the first acquisition means or the second imaging data acquired by the second acquisition means according to a mode setting; an image data transmission means for transmitting the image data acquired by the image processing means to an external device; 13. An image processing device comprising:

[0061] (Configuration 2) In the image processing device, When the first mode is set, the image processing means acquires the image data by performing a predetermined image processing on the first imaging data acquired by the first acquisition means, When the second mode is set, the image processing means acquires the image data by performing a predetermined image processing on the second imaging data acquired by the second acquisition means. 2. The image processing device according to claim 1,

[0062] (Configuration 3) a time synchronization unit for synchronizing time information with the external device; Based on the time information synchronized by the time synchronization means, an imaging timing generating means for generating an imaging timing at which the imaging means images the imaging target; an acquisition timing generating means for generating an acquisition timing for the second acquisition means to acquire the second imaging data from the storage means; 3. The image processing device according to configuration 1 or 2, further comprising:

[0063] (Configuration 4) the imaging timing generating means and the acquisition timing generating means operate exclusively in accordance with a mode setting; In the image processing device, When the first mode is set, the imaging timing generation means generates the imaging timing, When the second mode is set, the acquisition timing generation means generates the acquisition timing. 4. The image processing device according to configuration 3.

[0064] (Configuration 5) When the first mode is set in the image processing device, the image data transmission means transmits the image data to the external device at the predetermined intervals during which the imaging means captures images. 5. The image processing device according to any one of configurations 1 to 4,

[0065] (Configuration 6) When the second mode is set in the image processing device, the image data transmission means transmits the image data to the external device at intervals different from the predetermined intervals at which the imaging means captures images. 6. The image processing device according to any one of configurations 1 to 5,

[0066] (Configuration 7) The first imaging data and the second imaging data are data in which image data obtained by imaging with the imaging means is associated with time code information indicating the imaging time. 7. The image processing device according to any one of configurations 1 to 6,

[0067] (Configuration 8) The apparatus further includes a receiving means for receiving time code information from an external device, When the second mode is set in the image processing device, the second acquisition means acquires the second imaging data from the storage means based on the time code received by the reception means. 8. The image processing device according to any one of configurations 1 to 7,

[0068] (Configuration 9) The predetermined image processing is a foreground / background separation process for extracting image data relating to the foreground in the captured image data. 9. The image processing device according to any one of configurations 1 to 8,

[0069] (Configuration 10) The imaging means and the storage means are external to the image processing device. 10. The image processing device according to any one of configurations 1 to 9,

[0070] (Configuration 11) When the first mode is set in the image processing device, the first image data captured by the imaging means is subjected to the predetermined image processing by the image processing means without being processed by the storage means. 11. The image processing device according to any one of configurations 1 to 10, [Explanation of symbols]

[0071] 1. Image Processing System 10 Sensor System 11 Imaging device 12 Time Server 13 Image Computing Server 14 Control Station 16 Controller 100 Image processing device 110 Storage device

Claims

1. An image processing device that supplies image data to an image processing system that processes a plurality of image data based on a plurality of imaging data, a first acquisition means for acquiring first imaging data from an imaging means for imaging an imaging target; a second acquisition means for acquiring second imaging data captured by the imaging means from a storage means for storing the second imaging data; an image processing means for acquiring image data by performing a predetermined image processing on the first imaging data acquired by the first acquisition means or the second imaging data acquired by the second acquisition means according to a mode setting; an image data transmission means for transmitting the image data acquired by the image processing means to an external device; 13. An image processing device comprising:

2. In the image processing device, When the first mode is set, the image processing means acquires the image data by performing a predetermined image processing on the first imaging data acquired by the first acquisition means, When the second mode is set, the image processing means acquires the image data by performing a predetermined image processing on the second imaging data acquired by the second acquisition means.

2. The image processing device according to claim 1,

3. a time synchronization unit for synchronizing time information with the external device; Based on the time information synchronized by the time synchronization means, an imaging timing generating means for generating an imaging timing at which the imaging means images the imaging target; an acquisition timing generating means for generating an acquisition timing for the second acquisition means to acquire the second imaging data from the storage means; 3. The image processing apparatus according to claim 2, further comprising:

4. the imaging timing generating means and the acquisition timing generating means operate exclusively in accordance with a mode setting; In the image processing device, When the first mode is set, the imaging timing generation means generates the imaging timing, When the second mode is set, the acquisition timing generation means generates the acquisition timing.

4. The image processing device according to claim 3.

5. When the first mode is set in the image processing device, the image data transmission means transmits the image data to the external device at the predetermined intervals during which the imaging means captures images.

5. The image processing device according to claim 4.

6. When the second mode is set in the image processing device, the image data transmission means transmits the image data to the external device at intervals different from the predetermined intervals at which the imaging means captures images.

6. The image processing device according to claim 5,

7. The first imaging data and the second imaging data are data in which image data obtained by imaging with the imaging means is associated with time code information indicating the imaging time.

7. The image processing device according to claim 6.

8. The apparatus further includes a receiving means for receiving time code information from an external device, When the second mode is set in the image processing device, the second acquisition means acquires the second imaging data from the storage means based on the time code received by the reception means.

8. The image processing device according to claim 7,

9. The predetermined image processing is a foreground / background separation process for extracting image data relating to the foreground in the captured image data.

9. The image processing device according to claim 8,

10. The imaging means and the storage means are external to the image processing device.

10. The image processing device according to claim 9.

11. When the first mode is set in the image processing device, the first image data captured by the imaging means is subjected to the predetermined image processing by the image processing means without being processed by the storage means. The image processing device according to claim 10 .

12. A method for controlling an image processing device that supplies image data to an image processing system that processes a plurality of image data based on a plurality of imaging data, comprising the steps of: a first acquisition step of acquiring first imaging data from an imaging means that images an imaging target; a second acquisition step of acquiring second imaging data captured by the imaging means from a storage means that stores the second imaging data; an image processing step of acquiring image data by performing a predetermined image processing on the first imaging data acquired in the first acquisition step or the second imaging data acquired in the second acquisition step according to a mode setting; an image data transmission step of transmitting the image data acquired in the image processing step to an external device; 13. A method for controlling an image processing apparatus comprising the steps of:

13. A program for causing a computer to function as the control method for the image processing apparatus according to claim 12.

14. An image processing system including a plurality of image processing devices, and processing a plurality of image data based on a plurality of imaging data, Each of the plurality of image processing devices a first acquisition means for acquiring first imaging data from an imaging means for imaging an imaging target; a second acquisition means for acquiring second imaging data captured by the imaging means from a storage means for storing the second imaging data; an image processing means for acquiring image data by performing a predetermined image processing on the first imaging data acquired by the first acquisition means or the second imaging data acquired by the second acquisition means according to a mode setting; an image data transmission means for transmitting the image data acquired by the image processing means to another image processing device; having The plurality of image processing devices are connected to each other in a daisy chain.

1. An image processing system comprising:

15. A plurality of imaging devices for imaging a target area from a plurality of directions; a plurality of storage devices for storing a plurality of pieces of image data captured by the plurality of image capture devices as a plurality of pieces of recorded data; acquiring the imaging data from the imaging device; a plurality of image processing devices that acquire a plurality of image data by performing a predetermined image processing on the plurality of pieces of imaging data acquired from the plurality of imaging devices or the plurality of pieces of recording data acquired from the plurality of storage devices according to a mode setting; an image generating device that generates three-dimensional shape data based on the plurality of image data acquired by the plurality of image processing devices in accordance with a mode setting; An image processing system comprising: The image processing system includes: When operating in a first generation mode, the method generates three-dimensional shape data using the plurality of image data acquired by performing a predetermined image processing on the plurality of recorded image data; When operating in the second generation mode, the plurality of pieces of image data are acquired by performing a predetermined image processing on the plurality of pieces of image data, and three-dimensional shape data is generated using the plurality of pieces of image data.

1. An image processing system comprising:

Citation Information

Patent Citations

  • Image processing system, image processor, control method, and program

    JP2019050593A

  • Image processing system

    JP2023003096A