Image synthesis apparatus and image synthesis method

Abstract

To provide an image synthesis device and image synthesis method capable of generating a composite image by stitching together multiple captured images with high precision. [Solution] The image synthesis apparatus according to the embodiment comprises an acquisition unit, a detection unit, a determination unit, and a generation unit. The acquisition unit acquires multiple captured images taken by each of a plurality of cameras arranged in a predetermined space. The detection unit detects a predetermined object from each of the plurality of captured images. The determination unit determines the synthesis method of the plurality of captured images based on the object detection result. The generation unit generates a composite image by synthesizing the plurality of captured images in the determined synthesis method.

Description

Technical Field

[0001] Embodiments of the present invention relate to an image synthesizing apparatus and an image synthesizing method.

Background Art

[0002] Conventionally, a technique for capturing images with cameras built into a plurality of lighting devices has been known (see, for example, Patent Document 1). In this type of technique, the captured images of each of the plurality of cameras may be arranged and displayed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art, even if a user such as an administrator views the captured images arranged and displayed, it is not easy to grasp the connection between the images, even if some of the captured images include a partially same imaging range. Further, as described above, it takes time and effort to manually connect the portions of the same imaging range.

[0005] An object of the problems to be solved by the present invention is to provide an image synthesizing apparatus and an image synthesizing method capable of generating a synthesized image in which a plurality of captured images are accurately connected.

Means for Solving the Problems

[0006] The image synthesis apparatus according to the embodiment comprises an acquisition unit, a detection unit, a determination unit, and a generation unit. The acquisition unit acquires a plurality of captured images taken by a plurality of cameras arranged in a predetermined space. The detection unit detects a predetermined object from each of the plurality of captured images. The determination unit determines the synthesis method of the plurality of captured images based on the detection result of the object. The generation unit generates a composite image by synthesizing the plurality of captured images in the determined synthesis method. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 shows an example of the configuration of an information processing system according to an embodiment. [Figure 2] Figure 2 is a block diagram showing an example configuration of an information processing device according to the embodiment. [Figure 3] Figure 3 is a diagram illustrating the method for combining composite images. [Figure 4] Figure 4 is a diagram illustrating the method for creating composite images. [Figure 5] Figure 5 is a timing chart showing the operating timing of each device in the information processing system according to the embodiment. [Figure 6] Figure 6 is a flowchart showing the processing procedure of the information processing device executed according to the embodiment. [Modes for carrying out the invention]

[0008] The image synthesis apparatus described below comprises an acquisition unit 121, a detection unit 122, a determination unit 123, and a generation unit 124. The acquisition unit 121 acquires multiple captured images taken by each of a plurality of cameras 11 arranged in a predetermined space SP. The detection unit 122 detects a predetermined object from each of the plurality of captured images. The determination unit 123 determines the synthesis method of the plurality of captured images based on the object detection result. The generation unit 124 generates a composite image by synthesizing the plurality of captured images in the determined synthesis method.

[0009] The determination unit 123, described below, determines a synthesis method for superimposing multiple captured images when the same predetermined object is detected from multiple captured images. The generation unit 124 generates a composite image by superimposing multiple captured images so that the same predetermined object overlaps.

[0010] The generation unit 124, described below, if the size of a predetermined object detected in each of the multiple captured images is different, enlarges or reduces the multiple captured images so that the sizes are the same, and then generates a composite image.

[0011] The determination unit 123, described below, determines a composite image configuration in which multiple captured images are placed adjacent to each other when different predetermined objects are detected from multiple captured images. The generation unit 124 generates a composite image in which the multiple captured images are placed adjacent to each other.

[0012] The acquisition unit 121, described below, acquires multiple captured images, each associated with time information provided to each of the multiple cameras 11 from the time server 300. The detection unit 122 extracts multiple captured images taken at the same time based on the time information and detects a predetermined object from the extracted multiple captured images. The generation unit 124 generates a composite image by combining multiple captured images taken at the same time.

[0013] The image synthesis apparatus described below further includes an output unit 125 that outputs the synthesized image generated by the generation unit 124 to the display device 200.

[0014] The image synthesis method described below is an image synthesis method performed by an image synthesis device, and includes an acquisition step, a detection step, a determination step, and a generation step. The acquisition step acquires multiple captured images taken by each of a plurality of cameras 11 arranged in a predetermined space SP. The detection step detects a predetermined object from each of the plurality of captured images. The determination step determines the synthesis mode of the plurality of captured images based on the object detection result. The generation step generates a composite image by synthesizing the plurality of captured images in the determined synthesis mode.

[0015] Hereinafter, an image synthesis apparatus and image synthesis method according to an embodiment will be described with reference to the drawings. In the embodiment, the same parts are denoted by the same reference numerals, and redundant descriptions are omitted.

[0016] Figure 1 is a diagram showing an example configuration of an information processing system according to an embodiment. The information processing system S shown in Figure 1 is installed in a space SP of a facility that is relatively large, such as a factory or office, and has a large flow of people and a wide variety of equipment. The information processing system S is a system that generates and displays a composite image by combining camera images captured by multiple cameras in a facility where camera images contain various information (such as people flow and equipment).

[0017] As shown in Figure 1, the information processing system S includes a plurality of lighting devices 1, an information processing device 100, and a display device 200. The information processing device 100 is an image synthesis device that performs an image synthesis method according to the embodiment. In the information processing system S shown in Figure 1, the plurality of lighting devices 1, the information processing device 100, and the display device 200 are connected by a communication network such as Ethernet (registered trademark). Although Figure 1 shows an example in which the information processing device 100 is configured as a dedicated server device, the information processing device 100 may also be incorporated into a lighting control device that performs lighting control of the plurality of lighting devices 1.

[0018] Lighting device 1 is a lighting device that illuminates a predetermined space SP in a facility or the like. Lighting device 1 is, for example, a lighting device that is directly attached to the ceiling and illuminates the floor from the ceiling. In Figure 1, it is assumed that a two-dimensional code C exists on the floor surface of space SP. The two-dimensional code C is not limited to being placed on the floor surface; it may also be placed on equipment whose position is basically fixed in space SP, such as a desk or a photocopier. The two-dimensional code C is, for example, a QR (Quick Response) code (registered trademark). In this disclosure, a two-dimensional code C is given as an example, but it is not limited to a two-dimensional code C; any object that can be uniquely identified, such as a number or a mark, can be used.

[0019] As shown in FIG. 1, the lighting device 1 includes a light source unit 10 and a camera 11. The light source unit 10 includes a light emitting element such as an LED (Light Emitting Diode), for example, and irradiates the space SP with light from the ceiling toward the floor surface.

[0020] The camera 11 is an imaging unit capable of imaging the space SP and is built in the lighting device 1. The camera 11 has an imaging angle facing the space SP from the ceiling toward the floor direction. Note that the camera 11 is, for example, a camera equipped with a fisheye lens. Note that the camera 11 is not limited to the camera built in the lighting device 1, and may be a camera installed independently in the space SP such as a security camera, for example.

[0021] The information processing device 100 is a server device that detects an object O and a two-dimensional code C existing in the space SP, and generates a composite image obtained by synthesizing a plurality of captured images captured by the plurality of cameras 11 based on the detection result. Note that the information processing device 100 may be a server device installed in a facility having the space SP, or may be a server device installed outside the facility. Further, the information processing device 100 may be realized by a cloud system. Further, the information processing device 100 may be incorporated in the lighting device 1, the above-described lighting control device, or the display device 200.

[0022] The display device 200 is a display that displays the composite image generated by the information processing device 100. The display device 200 may be a terminal device possessed by an administrator (such as a factory manager in a factory) of the space SP.

[0023] Next, an operation example of the information processing system S will be described using FIG. 1. The information processing system S in the present disclosure detects a predetermined target from each of a plurality of captured images captured by each of the plurality of cameras 11, determines a composite mode of the plurality of captured images based on the detection result of the target, and generates a composite image obtained by synthesizing the plurality of captured images in the determined composite mode.

[0024] Specifically, first, the information processing device 100 acquires multiple images captured by multiple cameras 11 (step S1). Each lighting device 1 also transmits time information provided by the time server 300 to the information processing device 100, linking it to the captured image.

[0025] Next, the information processing device 100 detects a predetermined object from each of the acquired multiple captured images (step S2). Specifically, the information processing device 100 detects a two-dimensional code C in each of the multiple captured images by image analysis such as template matching, and reads the information contained in the detected two-dimensional code C. The information contained in the two-dimensional code C is identification information uniquely determined for each two-dimensional code C, such as a number.

[0026] Next, the information processing device 100 determines the method of combining the multiple captured images based on the detection result of the two-dimensional code C (step S3). Specifically, the information processing device 100 determines, based on the number read from the two-dimensional code C, whether to combine the multiple captured images by superimposing them or by placing the multiple captured images adjacent to each other without superimposing them. Specifically, if there are two captured images among the multiple captured images in which the numbers of the two-dimensional code C are duplicated, the information processing device 100 determines the method of combining the two captured images by superimposing them. Alternatively, if there are no captured images among the multiple captured images in which the numbers of the two-dimensional code C are duplicated, the information processing device 100 determines the method of combining the multiple captured images adjacent to each other without superimposing them.

[0027] Next, the information processing device 100 generates a composite image synthesized in the determined synthesis mode (step S4). Specifically, in the case of a synthesis mode in which multiple captured images are superimposed, the information processing device 100 generates a composite image in which the two-dimensional code C of two captured images with overlapping digits overlaps so that the two-dimensional code C overlaps. In the case of a synthesis mode in which multiple captured images are placed adjacent to each other without superimposing, the information processing device 100 generates a composite image in which the multiple captured images are placed adjacent to each other.

[0028] Next, the information processing device 100 outputs the generated composite image to the display device 200, causing the display device 200 to display the composite image (step S5).

[0029] Thus, according to the information processing system S of this embodiment, a composite image can be generated by combining multiple captured images in a combination manner determined based on the detection result of the two-dimensional code C in each of the multiple captured images, thereby generating a composite image in which multiple captured images are joined together with high accuracy.

[0030] Next, an example of the configuration of the information processing device 100 will be described using Figure 2. Figure 2 is a block diagram showing an example of the configuration of the information processing device 100 according to the embodiment. As shown in Figure 2, the information processing device 100 comprises a communication unit 110, a control unit 120, and a storage unit 130. The control unit 120 comprises an acquisition unit 121, a detection unit 122, a determination unit 123, a generation unit 124, and an output unit 125. The information processing device 100 is also connected to a plurality of lighting devices 1 and a display device 200. The plurality of lighting devices 1 are each connected to a time server 300.

[0031] The time server 300 is a server device that provides time information. Multiple lighting devices 1 associate the time information provided by the time server 300 with the captured images. As a result, the information processing device 100 can easily distinguish between multiple captured images taken at the same time based on the time information.

[0032] The communication unit 110 is a network device for performing wireless communication processing and wired communication processing. For example, the communication unit 110 transmits and receives various information to and from the lighting device 1 and the display device 200 via a predetermined network.

[0033] Here, the information processing device 100 includes, for example, a computer having a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), flash memory, input / output ports, and various circuits.

[0034] The computer's CPU functions as the acquisition unit 121, detection unit 122, determination unit 123, generation unit 124, and output unit 125 of the control unit 120, for example, by reading and executing a program stored in ROM.

[0035] Furthermore, at least one or all of the functions of the acquisition unit 121, detection unit 122, determination unit 123, generation unit 124, and output unit 125 of the control unit 120 can be configured using hardware such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0036] Furthermore, the storage unit 130 can, for example, correspond to RAM or flash memory. RAM and flash memory can store information about various programs, etc. The information processing device 100 may also acquire the above-mentioned programs and various information via other computers or portable recording media connected by wired or wireless networks.

[0037] Next, the functions of the control unit 120 (acquisition unit 121, detection unit 122, determination unit 123, generation unit 124, and output unit 125) will be described.

[0038] The acquisition unit 121 acquires multiple captured images of the space SP captured by the cameras 11 of each of the multiple lighting devices 1.

[0039] The detection unit 122 detects a two-dimensional code C, which is a predetermined object present in spatial SP, based on the acquired image. Specifically, the detection unit 122 detects the two-dimensional code C in the image by image analysis such as template matching, and reads the information contained in the detected two-dimensional code C.

[0040] The determination unit 123 determines the method of combining the multiple captured images based on the detection result of the detection unit 122. Specifically, the determination unit 123 determines, based on the number read from the two-dimensional code C, whether to combine the multiple captured images by superimposing them or by placing the multiple captured images adjacent to each other without superimposing them. Specifically, if there are two captured images among the multiple captured images in which the numbers of the two-dimensional code C are duplicated, the determination unit 123 determines the method of combining the two captured images by superimposing them. Alternatively, if there are no captured images among the multiple captured images in which the numbers of the two-dimensional code C are duplicated, the determination unit 123 determines the method of combining the multiple captured images adjacent to each other without superimposing them.

[0041] The generation unit 124 generates a composite image by combining multiple captured images according to the synthesis mode determined by the determination unit 123. Specifically, in the case of a synthesis mode in which multiple captured images are superimposed, the generation unit 124 generates a composite image in which the two-dimensional code C of two captured images with overlapping digits overlaps so that the two-dimensional code C overlaps. In the case of a synthesis mode in which multiple captured images are placed adjacent to each other without superimposing, the generation unit 124 generates a composite image in which multiple captured images are placed adjacent to each other.

[0042] Next, the output unit 125 outputs the generated composite image to the display device 200, which then displays the composite image. This allows the administrator to grasp multiple captured images from a single composite image, thus saving time and effort.

[0043] Next, we will specifically explain the method of combining images using Figures 3 and 4. Figures 3 and 4 are diagrams illustrating the method of combining images.

[0044] First, using Figure 3, we will explain the synthesis method in which multiple captured images are superimposed. In Figures 3 and 4, we assume that there are two two-dimensional codes C1 with "1" embedded in it and two two-dimensional codes C2 with "2" embedded in it.

[0045] As shown in Figure 3, the two-dimensional code C1 of "1" is visible in both the captured image P1 and the captured image P2. In other words, the captured areas of the captured image P1 and the captured image P2 overlap in the range that includes the two-dimensional code C1 of "1".

[0046] Furthermore, as shown in Figure 3, the two-dimensional code C2 of "2" is visible in both the captured image P1 and the captured image P3. In other words, the captured areas of the captured image P1 and the captured image P3 overlap in the range that includes the two-dimensional code C2 of "2".

[0047] In the above case, the determination unit 123 determines a composite mode in which captured image P1 and captured image P2 are superimposed. Similarly, the determination unit 123 determines a composite mode in which captured image P1 and captured image P3 are superimposed.

[0048] Then, the generation unit 124 superimposes the captured image P1 and captured image P2 so that the two-dimensional code C1 of "1" overlaps. Specifically, the generation unit 124 superimposes the captured image P1 on top of the captured image P2. Similarly, the generation unit 124 superimposes the captured image P1 and captured image P3 so that the two-dimensional code C2 of "2" overlaps. Specifically, the generation unit 124 superimposes the captured image P1 on top of the captured image P3.

[0049] As a result, a single composite image SP is generated from the three captured images P1, P2, and P3. In other words, the generation unit 124 can generate a composite image SP in which the three captured images P1, P2, and P3 are joined together by superimposing the two-dimensional codes C1 and C2 so that they overlap.

[0050] Note that since there is some overlap between captured images P2 and P3, they are superimposed so that one of them is on top. Also, in Figure 3, captured image P1 is superimposed as the top layer, but it is also possible to superimpose them so that captured image P1 is the bottom layer or an intermediate layer.

[0051] Furthermore, for example, if the area size of the two-dimensional code C1 differs between captured image P1 and captured image P2, the generation unit 124 may enlarge or reduce either one or both of captured image P1 and captured image P2 to make the area size of the two-dimensional code C1 the same when generating the composite image SP. This reduces the sense of incongruity caused by the difference in the apparent size of objects and other elements in the images when captured image P1 and captured image P2 are combined.

[0052] Furthermore, the generation unit 124 may not only enlarge or reduce the image, but may also rotate the captured image P1 and captured image P2 to align the orientation of the two-dimensional code C1.

[0053] Next, using Figure 4, we will explain a method for combining multiple captured images in a manner where they are adjacent to each other without being superimposed.

[0054] As shown in Figure 4, the captured image P1 contains the two-dimensional code C1, which represents "1". The captured image P2 contains the two-dimensional code C2, which represents "2". Therefore, the captured areas of captured image P1 and captured image P2 do not overlap.

[0055] In the above case, the determination unit 123 determines a composite image configuration in which captured images P1 and P2 are adjacent without superimposing them. The generation unit 124 then generates a composite image SP in which captured images P1 and P2 are adjacent without superimposing them. Figure 4 shows a composite image SP in which the two-dimensional code C1 of "1" and the two-dimensional code C2 of "2" are adjacent so as close together as possible. In other words, the generation unit 124 places the captured image P2 with the two-dimensional code C2 of "2", which is the next number after the two-dimensional code C1 of "1", adjacent to the captured image P1. If the two-dimensional code C of "3" is present in another captured image, that image is placed adjacent to the captured image P2 in which the two-dimensional code C2 of "2" is present.

[0056] This allows for the generation of a single composite image SP even if the captured images P1 and P2 do not overlap in their imaging ranges. Furthermore, by arranging the captured images adjacent to each other in the order of the numbers embedded in the two-dimensional code C, users can easily determine the numbers embedded in the two-dimensional code C when viewing the composite image SP.

[0057] In Figure 4, an example is shown where the two-dimensional codes C are placed adjacent to each other in the order of the numbers embedded in them. However, if information on the relative positional relationship between the two-dimensional code C1 ("1") and the two-dimensional code C2 ("2") can be obtained, they may also be placed adjacent to each other according to their relative positional relationship. This allows for the generation of a composite image SP that matches the arrangement of the spatial SPs.

[0058] Next, the operating timing of each device in the information processing system S according to the embodiment will be explained using Figure 5. Figure 5 is a timing chart showing the operating timing of each device in the information processing system S according to the embodiment.

[0059] As shown in Figure 5, the time server 300 first provides each lighting device 1 with time information indicating the current time (step S10). Although Figure 5 illustrates that time information is provided only in step S10, in reality, the time server 300 provides time information to each lighting device 1 periodically.

[0060] Next, each lighting device 1 associates time information, which is the time of acquisition, with the image captured by the camera 11 and transmits it to the information processing device 100 (step S11).

[0061] Next, the information processing device 100 extracts images from the multiple images acquired from each lighting device 1 that are associated with the same time (step S12).

[0062] Next, the information processing device 100 detects a two-dimensional code C from each of the extracted multiple captured images (step S13). Subsequently, the information processing device 100 determines the synthesis mode of the multiple captured images based on the detection results (step S14), and generates a composite image by synthesizing the multiple captured images in the determined synthesis mode (step S15).

[0063] Next, the information processing device 100 outputs the generated composite image to the display device 200, and the display device 200 displays the composite image (step S16).

[0064] Next, the processing procedure of the information processing device 100 according to the embodiment will be described using Figure 6. Figure 6 is a flowchart showing the processing procedure of the information processing device 100 according to the embodiment.

[0065] As shown in Figure 6, the control unit 120 acquires multiple captured images from each lighting device 1 (step S101). Subsequently, the control unit 120 extracts images taken at the same time from the acquired multiple captured images (step S102).

[0066] Next, the control unit 120 detects a two-dimensional code C from each of the extracted multiple captured images (step S103).

[0067] Next, the control unit 120 identifies the information (number) indicated by the detected two-dimensional code C (step S104) and determines whether the same number exists in different captured images (step S105).

[0068] If different captured images have the same number (step S105: Yes), the control unit 120 determines a synthesis method to superimpose the different captured images and identifies the position of the two-dimensional code C with the same number for each captured image (step S106).

[0069] Next, the control unit 120 generates a composite image by superimposing multiple captured images so that the positions of the same numbers are aligned (step S107).

[0070] Next, the control unit 120 displays the generated composite image on the display device 200 (step S108), and then terminates the process.

[0071] In step S105, if there are no different captured images with the same number (step S105: No), that is, if all captured images have different numbers, the control unit 120 determines a composite method for arranging the multiple captured images next to each other, generates a composite image with the captured images next to each other in numerical order (step S109), and proceeds to step S108.

[0072] As described above, the image synthesis apparatus (information processing apparatus 100) according to the embodiment comprises an acquisition unit 121, a detection unit 122, a determination unit 123, and a generation unit 124. The acquisition unit 121 acquires multiple captured images taken by each of the multiple cameras 11 arranged in a predetermined space SP. The detection unit 122 detects a predetermined object from each of the multiple captured images. The determination unit 123 determines the synthesis mode of the multiple captured images based on the object detection result. The generation unit 124 generates a composite image by synthesizing the multiple captured images in the determined synthesis mode. This makes it possible to generate a composite image in which multiple captured images are joined together with high precision.

[0073] While embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0074] 1. Lighting device 10 Light source section 11 Cameras 100 Information Processing Devices 110 Communications Department 120 Control Unit 121 Acquisition Department 122 Detection unit 123 Decision Section 124 Generation part 125 Output section 130 Storage section 200 Display device 300 Time Servers C 2D code S Information Processing System

Claims

1. An acquisition unit that acquires multiple captured images taken by multiple cameras arranged in a predetermined space; A detection unit for detecting a predetermined target from each of the aforementioned plurality of captured images; A determination unit that determines the synthesis mode of the plurality of captured images based on the detection result of the target; A generation unit that generates a composite image by combining the plurality of captured images in the determined synthesis mode; An image synthesis device equipped with the following:

2. The aforementioned determination unit, If the same predetermined object is detected in the plurality of captured images, a synthesis method for superimposing the plurality of captured images is determined. The generating unit is The composite image is generated by superimposing the multiple captured images so that the same predetermined object overlaps. The image synthesis apparatus according to claim 1.

3. The generating unit is If the size of the predetermined object detected in each of the multiple captured images is different, the multiple captured images are enlarged or reduced so that the sizes are the same before the composite image is generated. The image synthesis apparatus according to claim 2.

4. The aforementioned determination unit, If different predetermined objects are detected from the plurality of captured images, a synthesis method is determined in which the plurality of captured images are placed adjacent to each other. The generating unit is The composite image is generated by arranging the plurality of captured images adjacent to each other. The image synthesis apparatus according to claim 1.

5. The acquisition unit is, The multiple captured images, each associated with time information provided to each of the multiple cameras from the time server, are acquired. The detection unit is Based on the aforementioned time information, the plurality of captured images taken at the same time are extracted, and the predetermined target is detected from the extracted plurality of captured images. The generating unit is The composite image is generated by combining the multiple captured images taken at the same time. The image synthesis apparatus according to claim 1.

6. The system further comprises an output unit that outputs the composite image generated by the generation unit to a display device. The image synthesis apparatus according to claim 1.

7. An image synthesis method performed by an image synthesis device, An acquisition step of acquiring multiple captured images taken by multiple cameras arranged in a predetermined space; A detection step of detecting a predetermined target from each of the aforementioned plurality of captured images; A determination step of determining the synthesis mode of the plurality of captured images based on the detection results of the target; A generation step of generating a composite image by combining the plurality of captured images in the determined synthesis mode; An image synthesis method that includes [a specific feature / method].

Images