Information management method and computer program

Abstract

To provide a moving object detection system and an information management apparatus of the same for improving efficiency of a process for detecting a plurality of moving objects from a captured image.SOLUTION: A traffic line detection system comprises: a plurality of imaging apparatuses (cameras) each for capturing an image at a predetermined period to generate image data; a plurality of image recognition apparatuses (traffic line extraction servers) for performing image recognition on the image data from each imaging apparatus to detect moving objects in the captured images represented by each image data; and an information management apparatus (crowding management server) for managing the image data supplied from each imaging apparatus to each image recognition apparatus. The information management apparatus comprises a communication unit for performing data communication with each imaging apparatus and each image recognition apparatus, and a control unit for controlling the operation of the communication unit. The control unit manages congestion information indicating the quantity of moving objects localized within the range of the captured image that is indicated by each image data for each imaging apparatus, and controls the communication unit to distribute the image data from the plurality of imaging apparatuses and supplying the same to each image recognition apparatus depending on the size of the quantity indicated by the congestion information.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to Information management method and computer program .

Background Art

[0002] Patent Document 1 discloses a monitoring system including one or more monitoring cameras, a monitoring server, and a mobile terminal possessed by a user. The monitoring server of Patent Document 1 extracts the movement line of a moving object such as a person from the video of the monitoring camera and selects the movement line included in the peripheral area of the mobile terminal or the like. Based on the selected movement line, the monitoring server identifies the video of the area including the passing route of the movement line and presents it to the user. Thus, in Patent Document 1, by presenting the video according to the analysis result of the movement line to the user, the efficiency of the video confirmation work by the user is improved.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure can efficiently process for an image recognition device imaging images using a plurality of imaging devices Information management method and Computer program .

Means for Solving the Problems

[0005] According to one aspect of the present disclosure, Information management method it is A step in which a computer acquires image data from a plurality of imaging devices, and a control unit of the computer to manage congestion information indicating the quantity of moving objects localized within the range of the imaging image indicated by each image data for each imaging device A step of generating , In the first image data obtained by the first imaging device among the plurality of imaging devices, when the magnitude of the quantity indicated by the congestion information is less than a predetermined amount, the control unit supplies the first image data to the first image recognition device among the plurality of image recognition devices; and in the first image data, when the magnitude of the quantity indicated by the congestion information is greater than or equal to the predetermined amount, the control unit distributes the first image data and supplies it to at least the first image recognition device and a second image recognition device different from the first image recognition device among the plurality of image recognition devices, including .

[0006] According to one aspect of the present disclosure, Computer program it is When executed by a control unit of a computer, causes the computer to perform a step of acquiring image data from a plurality of imaging devices, andCongestion information indicating the quantity of moving objects localized within the range of the captured image indicated by each image data for each imaging device A step of generating , In the first image data obtained by the first imaging device among the plurality of imaging devices, when the magnitude of the quantity indicated by the congestion information is less than a predetermined amount, a step of supplying the first image data to the first image recognition device among the plurality of image recognition devices; and in the first image data, when the magnitude of the quantity indicated by the congestion information is greater than or equal to the predetermined amount, a step of distributing the first image data and supplying it to at least the first image recognition device and a second image recognition device different from the first image recognition device among the plurality of image recognition devices, and executing .

Effects of the Invention

[0007] According to the present disclosure Information management method and computer program , it is possible to Using a plurality of imaging devices efficiently process the captured image for an image recognition device .

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Modes for Carrying Out the Invention

[0009] Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, a more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially the same configurations may be omitted. This is to avoid making the following description unnecessarily redundant and to facilitate the understanding of those skilled in the art. Note that the inventor(s) provide the accompanying drawings and the following description so that those skilled in the art can fully understand the present disclosure, and do not intend to limit the subject matter described in the claims thereby.

[0010] (Embodiment 1) 1. Configuration The flow line detection system according to Embodiment 1 will be described with reference to FIG. 1. FIG. 1 is a diagram showing the outline of the flow line detection system 1 according to the present embodiment.

[0011] 1-1. Outline of the System As shown in FIG. 1, this system 1 includes a plurality of cameras 2-1 to 2-3, a congestion management server 7, a plurality of flow line extraction servers 8-1 to 8-3, and a flow line combination server 9. Hereinafter, the cameras 2-1, 2-2, and 2-3 are also collectively referred to as "camera 2". Also, the flow line extraction servers 8-1, 8-2, and 8-3 are also collectively referred to as "flow line extraction server 8".

[0012] This system 1 is an example of a moving object detection system that uses a plurality of cameras 2 to image the entire environment 6 with the trajectories, i.e., flow lines, of a plurality of people 11 etc. moving in an environment 6 such as a store sales floor, a workplace, or a factory, etc. as the detection targets. This system 1 may include a display terminal 4 for presenting the flow lines of various people 11 in the environment 6 to a user 3 such as, for example, an administrator or an analyst of the environment 6.

[0013] In this system 1, corresponding to each camera 2-1, 2-2, 2-3, respective flow line extraction servers 8-1, 8-2, 8-3 are provided. The flow line extraction server 8 is a server device that executes an image recognition process for extracting flow lines from the captured images by the camera 2. The congestion management server 7 is a server device for managing the image data supplied from each of the cameras 2-1, 2-2, 2-3 to each of the flow line extraction servers 8-1, 8-2, 8-3. The flow line combination server 9 is a server device for generating information integrating the flow lines etc. extracted by each of the flow line extraction servers 8-1, 8-2, 8-3.

[0014] In the present embodiment, in the flow line detection system 1 as described above, by using the congestion management server 7, even if congestion occurs due to a large number of people 11, it is easy to obtain in real time the flow lines etc. of the entire environment 6.

[0015] Hereinafter, the specific configurations of each of the camera 2, the congestion management server 7, the flow line extraction server 8, and the flow line combination server 9 in this system 1 will be described.

[0016] 1-2. Configuration of the Camera The camera 2 is an example of an imaging device in this system 1. The camera 2 includes an imaging unit that captures an image, and a transmission unit that transmits image data indicating the image captured by the imaging unit. The imaging unit is realized by a CCD image sensor, a CMOS image sensor, or the like. The transmission unit includes an interface circuit for communicating with an external device in accordance with a predetermined communication standard (for example, IEEE802.11, USB).

[0017] The camera 2 of this embodiment is, for example, an omnidirectional camera equipped with a fish-eye lens. The camera 2 of this system 1 may be various imaging devices not particularly limited to omnidirectional cameras. Also, in FIG. 1, three cameras 2-1, 2-2, and 2-3 are illustrated, but the number of cameras 2 included in this system 1 is not particularly limited to three, and may be two or four or more.

[0018] In this system 1, from the viewpoint of accurately detecting, for example, the movement lines of the people 11 moving around in the environment 6, the imaging ranges imaged by the respective cameras 2-1, 2-2, and 2-3 in the environment 6 are set so as to partially overlap each other, for example.

[0019] 1-3. Configuration of Congestion Management Server FIG. 2 is a block diagram illustrating the configuration of the congestion management server 7 of this embodiment. The congestion management server 7 is an example of an information management device in this embodiment. The congestion management server 7 illustrated in FIG. 2 includes a control unit 70, a storage unit 74, and a communication interface 75.

[0020] The control unit 70 controls the operation of the congestion management server 7. The control unit 70 is composed of, for example, a CPU or an MPU that cooperates with software to realize a predetermined function. The control unit 70 reads out the data and programs stored in the storage unit 74 to perform various arithmetic processes and realize various functions. For example, the control unit 70 executes a program including a group of instructions for realizing various processes of the congestion management server 7. The above program is, for example, an application program and may be provided from a communication network such as the Internet or stored in a portable storage medium.

[0021] In the congestion management server 7 illustrated in FIG. 2, the control unit 70 includes, for example, a congestion determination unit 71 and an image distribution unit 72 as functional components. The congestion determination unit 71 performs a congestion determination process, which is a process of determining whether the captured images of the cameras 2-1, 2-2, and 2-3 are in a congestion state described later. The image distribution unit 72 performs an image distribution process, which is a process of determining a distribution destination for distributing the image data of the plurality of cameras 2 from the plurality of flow line extraction servers 8 based on the result of the congestion determination process, and transmitting the image data to each of the flow line extraction servers 8-1, 8-2, and 8-3.

[0022] Note that the control unit 70 may be a dedicated electronic circuit or a reconfigurable electronic circuit such as a hardware circuit designed to realize a predetermined function. The control unit 70 may be composed of semiconductor integrated circuits such as a microcomputer, a DSP, an FPGA, and an ASIC. Further, the control unit 70 may include an internal memory as a temporary storage area for holding various data and programs.

[0023] The storage unit 74 stores data and programs. The storage unit 74 is composed of, for example, a hard disk drive (HDD) or a semiconductor storage device (SSD). The storage unit 74 stores the image data acquired from the plurality of cameras 2. The storage unit 74 may include a temporary storage element composed of a RAM such as a DRAM or an SRAM, and may function as a working area of the control unit 70. For example, the storage unit 74 may temporarily store the determination result of the congestion determination unit 71 and the determination result of the distribution destination by the image distribution unit 72.

[0024] The communication interface 75 is a connection circuit that performs connection between devices and data communication, etc., in accordance with a standard such as USB, IEEE 802.11, or Bluetooth (registered trademark). The communication interface 75 connects, for example, the plurality of cameras 2 and the plurality of flow line extraction servers 8 to the congestion management server 7. The communication interface 75 may include a circuit for connecting to a communication network, and may perform data communication with the plurality of cameras 2 and the plurality of flow line extraction servers 8, etc., via the communication network.

[0025] 1-4. Configuration of the Flow Line Extraction Server FIG. 3 is a block diagram illustrating the configuration of the flow line extraction server 8 of the present embodiment. The flow line extraction server 8 is an example of an image recognition device in the present embodiment. The flow line extraction server 8 illustrated in FIG. 3 includes a control unit 80, a storage unit 84, and a communication interface 85.

[0026] The control unit 80 controls the operation of the flow line extraction server 8. The control unit 80 is composed of, for example, a CPU, MPU, GPU, GPGPU, or TPU, or a combination thereof, which cooperates with software to realize a predetermined function. The control unit 80 reads out the data and programs stored in the storage unit 84, performs various arithmetic processes, and realizes various functions.

[0027] In the flow line extraction server 8 illustrated in FIG. 3, the control unit 80 includes, for example, a flow line extraction unit 81 as a functional configuration. The flow line extraction unit 81 performs a flow line extraction process including an image recognition process of detecting an object such as a person 11 preset as an object of image recognition based on the image data received from the congestion management server 7. In the flow line extraction process, position information of the person 11 or the like is calculated based on the detection result by image recognition. The flow line extraction unit 81 generates individual flow line information Ln indicating the flow line based on the position information calculated from the image data of the camera 2-1 corresponding to the flow line extraction server 8-1, and accumulates it in the storage unit 84. For the image recognition process in the flow line extraction unit 81, for example, a learned model by various machine learning methods is applied.

[0028] Note that the control unit 80 may be a dedicated electronic circuit or a reconfigurable electronic circuit such as a hardware circuit designed to realize a predetermined function. The control unit 80 may be composed of semiconductor integrated circuits such as a microcontroller, DSP, FPGA, and ASIC. Further, the control unit 80 may include an internal memory as a temporary storage area for holding various data and programs.

[0029] The storage unit 84 stores data and programs. The storage unit 84 is constituted by, for example, a hard disk drive (HDD) or a solid state drive (SSD). The storage unit 84 stores the individual traffic line information Ln. The storage unit 84 may include a temporary storage element constituted by a RAM such as a DRAM or an SRAM, and may function as a work area of the control unit 80.

[0030] The communication interface 85 is a connection circuit that performs connection between devices and data communication, etc., in accordance with a standard such as USB, IEEE 802.11, or Bluetooth. The communication interface 85 is communicatively connected to, for example, the congestion management server 7 and the traffic line combination server 9. The communication interface 85 may be communicatively connected to another traffic line extraction server 8. The communication interface 85 may include a circuit for connecting to a communication network, and may perform data communication via the communication network between, for example, a plurality of traffic line extraction servers 8.

[0031] 1-5. Configuration of the traffic line combination server FIG. 4 is a block diagram illustrating the configuration of the traffic line combination server 9 of the present embodiment. The traffic line combination server 9 is an example of an information integration device in the present embodiment. The traffic line combination server 9 illustrated in FIG. 4 includes a control unit 90, a storage unit 94, a communication interface 95, and an output interface 96.

[0032] The control unit 90 controls the operation of the traffic line combination server 9. The control unit 90 is constituted by, for example, a CPU or an MPU that realizes a predetermined function in cooperation with software. The control unit 90 reads out the data and programs stored in the storage unit 94, performs various arithmetic processes, and realizes various functions.

[0033] In the traffic line combination server 9 illustrated in FIG. 4, the control unit 90 includes, for example, a traffic line combination unit 91 as a functional configuration. The traffic line combination unit 91 combines the respective individual traffic line information L1, etc., received from each of the traffic line extraction servers 8-1, 8-2, 8-3, generates the overall traffic line information L0 based on the image data of the plurality of cameras 2, and stores it in the storage unit 94.

[0034] Note that the control unit 90 may be a hardware circuit such as a dedicated electronic circuit or a reconfigurable electronic circuit designed to implement a predetermined function. The control unit 90 may be composed of semiconductor integrated circuits such as a microcomputer, a DSP, an FPGA, and an ASIC. Further, the control unit 90 may include an internal memory as a temporary storage area for holding various data and programs.

[0035] The storage unit 94 stores data and programs. The storage unit 94 is constituted by, for example, a hard disk drive (HDD) or a semiconductor storage device (SSD). The storage unit 94 stores the overall flow line information L0. The storage unit 94 may include a temporary storage element constituted by a RAM such as a DRAM or an SRAM, and may function as a work area for the control unit 90.

[0036] The communication interface 95 is a connection circuit that performs connection between devices and data communication, etc. in accordance with a standard such as USB, IEEE 802.11, or Bluetooth. The communication interface 95 communicatively connects, for example, a plurality of flow line extraction servers 8 and a flow line combination server 9. The communication interface 95 may include a circuit for connecting to a communication network, and may perform data communication with, for example, a plurality of flow line extraction servers 8, etc. via the communication network.

[0037] The output interface 96 is an example of an output unit that outputs information. The output interface 96 is an output interface circuit for video signals, etc. in accordance with, for example, the HDMI (registered trademark) standard. The output interface 96 outputs a signal to an external display device such as a monitor, a projector, and a head-mounted display in order to display various information. Further, the output unit of the information integration device is not limited to the output interface 96, and may be configured as a display device such as a liquid crystal display or an organic EL display.

[0038] 2. Operation 2-1. Outline of Operation The outline of the operation of the flow line detection system 1 according to this embodiment will be described with reference to FIGS. 1 and 5. FIG. 5 is a diagram for explaining an operation example of the flow line detection system 1 according to this embodiment.

[0039] In the flow line detection system 1, each of the cameras 2-1, 2-2, 2-3 captures an image of each imaging range in the environment 6 and generates image data at, for example, the frame period of a moving picture. Each of the cameras 2-1, 2-2, 2-3 sequentially transmits the respective image data to the congestion management server 7, for example, in association with the imaging times synchronized with each other.

[0040] The congestion management server 7 in this embodiment acquires, for example, the image data of the images captured by each of the cameras 2-1, 2-2, 2-3 and performs processing as the congestion determination unit 71 and the image distribution unit 72 (see S1 and S2 in FIG. 5). For example, when no particular congestion state occurs as described later, the congestion management server 7 transmits the image data from each of the cameras 2-1, 2-2, 2-3 to the corresponding flow line extraction servers 8-1, 8-2, 8-3, respectively.

[0041] Based on the received image data, each of the flow line extraction servers 8-1, 8-2, 8-3 performs image recognition processing on the captured images of the plurality of cameras 2-1, 2-2, 2-3 in parallel and detects the flow line by a person 11 or the like (see S3 in FIG. 5). Each of the flow line extraction servers 8-1, 8-2, 8-3 generates individual flow line information indicating each detection result and transmits it to the flow line combination server 9.

[0042] The flow line combination server 9 integrates information such as the flow lines detected by the plurality of flow line extraction servers 8 so that the flow lines are displayed on a map image showing the map of the environment 6 on the display terminal 4, for example (see S4 in FIG. 5).

[0043] 2-1-1. Operation during congestion The operation during congestion in the flow line detection system 1 as described above will be described with reference to FIGS. 5 and 6.

[0044] Figure 5 illustrates the operation of the present system 1 when only the imaging range of a specific camera 2-3 in the environment 6 is congested by the person 11. FIGS. 6(A), (B), and (C) illustrate the captured images I21, I22, and I23 of each camera 2-1, 2-2, and 2-3 in the operation example of FIG. 5, respectively. The state of the captured image I23 during congestion is referred to as the "congested state", and the states of the captured images I21 and I22 that are not in the congested state are referred to as the "non-congested state".

[0045] According to the captured image I23 in the congested state, since a large number of people are located within the range of the image I23, a situation is assumed in which the processing load on the flow line extraction server 8-3 that performs image recognition becomes excessive. In this case, due to the delay in the processing of a specific flow line extraction server 8-3 among the plurality of flow line extraction servers 8-1 to 8-3, the processing (S4) of the flow line combination server 9 is delayed, and as a result, it becomes difficult to realize real-time display of the flow line.

[0046] Therefore, in the flow line detection system 1 of the present embodiment, the congestion management server 7 manages the congestion state in the captured images I21 to I23.

[0047] First, the congestion management server 7 performs congestion determination processing (S1) for each of the cameras 2-1, 2-2, and 2-3 by the congestion determination unit 71. In the example of FIG. 5, it is determined that only the captured image I23 of the specific camera 2-3 is in the congested state, and the captured images I21 and I22 of the remaining cameras 2-1 and 2-2 are determined not to be in the congested state.

[0048] Next, the congestion management server 7 performs image distribution processing (S2) of the image data 21, 22, and 23 by the image distribution unit 72 according to the result of the congestion determination processing (S1). The image distribution unit 72 transmits the image data 21 and 22 of the cameras 2-1 and 2-2 determined to be in the non-congested state to the flow line extraction servers 8-1 and 8-2 corresponding to the respective cameras 2-1 and 2-2. On the other hand, the image distribution unit 72 distributes the image data 23 of the camera 2-3 determined to be in the congested state to the plurality of flow line extraction servers 8 so as to divide the captured image I23. An example of such distribution is shown in FIGS. 6(C) to (F).

[0049] FIG. 6(D), (E), and (F) illustrate divided images I23b, I23c, and I23ad obtained by dividing the captured image I23 of FIG. 6(C). In the example of FIG. 5, the congestion management server 7 distributes and supplies the image data 23ad, 23b, and 23c indicating the divided images I23ad, I23b, and I23c not only to the specific flow line extraction server 8-3 but also to the other flow line extraction servers 8-1 and 8-2.

[0050] Each of the flow line extraction servers 8-1, 8-2, and 8-3 performs a flow line extraction process (S3) of executing an image recognition process on the image data supplied from the congestion management server 7 by each flow line extraction unit 81 and extracting a flow line based on the image data of the corresponding cameras 2-1, 2-2, and 2-3. At this time, since the processing target of the image recognition of the specific flow line extraction server 8-3 is reduced from the captured image I23 in the congested state to the divided image I23ad as shown in FIGS. 6(C) and (F), the processing load is reduced. On the other hand, the remaining divided images I23b and I23c, together with the captured images I21 and I22 that are not in the congested state, respectively, become the processing targets of the other flow line extraction servers 8-1 and 8-2.

[0051] Therefore, the processing load is distributed among the plurality of flow line extraction servers 8-1 to 8-3, and the flow line extraction process (S3) can be efficiently performed. Further, in the present embodiment, the plurality of flow line extraction servers 8-1 to 8-3 transmit and receive information on the detection results by image recognition of the divided images I23ad, I23b, and I23c to and from each other, thereby generating individual flow line information L1, L2, and L3 based on the image data of the corresponding cameras 2-1 to 2-3. Each of the flow line extraction servers 8-1, 8-2, and 8-3 transmits the individual flow line information L1, L2, and L3 to the flow line combination server 9.

[0052] The flow line combination server 9 performs a flow line combination process (S4) of integrating the received individual flow line information L1, L2, and L3 to generate overall flow line information L0. The flow line combination server 9 transmits the overall flow line information L0 to the display terminal 4 in combination with, for example, map image data indicating the map image of the environment 6 stored in advance in the storage unit 94. Thereby, the display terminal 4 displays the flow line indicated by the overall flow line information L0 in a superimposed manner on the map image (S5).

[0053] The flow line detection system 1 repeats the above processes (S1 to S5), for example, periodically. As a result, for example, on the display terminal 4, the display of the flow line is updated in real time.

[0054] According to the above operation, even if an imaging image I23ad in a congested state is generated, the processing load can be dispersed among the plurality of flow line extraction servers 8-1 to 8-3, and the flow line extraction process (S3) can be efficiently performed. Therefore, delays in subsequent flow line combination processes (S4) and the like can also be suppressed, and for example, it becomes easier to perform real-time display of the flow line on the display terminal 4.

[0055] According to such a flow line detection system 1, according to the result of the congestion determination process (S1), the image data of the plurality of cameras 2 is distributed to the plurality of flow line extraction servers 8 (S2). As a result, for example, in the flow line extraction server 8-3 corresponding to the camera 2-3 determined to be congested, it is possible to suppress the concentration of the load of the image recognition process in the flow line extraction process (S3) and delay in processing. According to the movement of the person 11 and the like in the environment 6, the result of the congestion determination process (S1) also changes, and by updating the division and distribution of the images in the image distribution process (S2), it is possible to perform the flow line combination process (S4) and the display of the flow line (S5) in real time.

[0056] 2-2. Congestion determination process The congestion determination process (S1 in FIG. 5) in the congestion management server 7 will be described with reference to FIGS. 7 and 8. In the congestion determination process (S1), the congestion information of each camera 2 is used. The congestion information is various information indicating the degree of congestion by moving bodies corresponding to the number of persons 11 in the imaging images of the plurality of cameras 2.

[0057] FIG. 7 is a flowchart illustrating the congestion determination process (S1) in the congestion management server 7. FIG. 8 is a diagram for explaining the congestion determination process. FIG. 9 is a diagram illustrating a congestion information table in the congestion determination process.

[0058] Each process shown in the flowchart of FIG. 7 is executed by the control unit 70 of the congestion management server 7 that functions as the congestion determination unit 71 at, for example, the frame period of the camera 2. The flowchart of FIG. 7 is started, for example, when the congestion management server 7 receives image data from a plurality of cameras 2 via the communication interface 75. The image data is held in the storage unit 74 for a predetermined number of frames, such as two frames for each camera 2.

[0059] In the flowchart of FIG. 7, first, the congestion determination unit 71 acquires the image data of each camera 2 from the storage unit 74, for example (S10). FIGS. 8(A) and (B) illustrate the image data acquired in step S10. FIG. 8(A) illustrates the image data of the current frame F2 captured by one camera 2. FIG. 8(B) illustrates the image data of the past frame F1 one frame before that of FIG. 8(A) captured by the same camera 2. In step S10, the congestion determination unit 71 acquires the current and past image data as shown in FIGS. 8(A) and (B) for each camera 2.

[0060] For each of the plurality of cameras 2-1 to 2-3, the congestion determination unit 71 generates the motion difference data H based on the acquired image data (S11). FIG. 8(C) illustrates the motion difference data H generated based on the image data of FIGS. 8(A) and (B). The motion difference data H is data indicating the difference due to the motion from the past to the present as a heatmap, for example, over the entire area of the captured image. The congestion determination unit 71 generates the motion difference data H by comparing the image data of the past frame F1 and the image data of the current frame F2 for each pixel, such as by calculating the difference between frames at each pixel.

[0061] Next, the congestion determination unit 71 determines the congestion state in the captured images of each camera 2 based on the motion difference data H1, H2, and H3 in each of the plurality of cameras 2-1 to 2-3 (S12). For example, the congestion determination unit 71 calculates the total value over all pixels of the motion difference data H as the congestion level of the current frame F2 per one camera 2, and determines whether the congestion state or the non-congestion state based on whether the congestion level is equal to or higher than a predetermined threshold value. The predetermined threshold value is set from the correlation between the number of persons and the motion difference in consideration of, for example, the number of persons at which the processing load becomes excessive within the range of the captured image. The congestion determination unit 71 performs the above processing on the motion difference data H1, H2, and H3 of each camera 2.

[0062] Next, the congestion determination unit 71 stores, for example, a congestion information table T1 for storing the congestion information of each camera 2 such as the above determination result in, for example, the storage unit 74 of the congestion management server 7 (S13). FIG. 9 illustrates the congestion information table T1.

[0063] In the example of FIG. 9, the congestion information table T1 stores, as an example of the congestion information, the above-described "motion difference data", the above-described "congestion level", and a "congestion flag" indicating the determination result of the congestion state in association with each "camera ID" for identifying each of the plurality of cameras 2-1 to 2-3. The congestion determination unit 71 sets each congestion flag to "on" or "off" according to the determination result of the congestion state or the non-congestion state for each camera 2.

[0064] In the example of FIG. 9, congestion information such as the motion difference data H1, H2, and H3 and the congestion levels X1, X2, and X3 of each camera 2-1, 2-2, and 2-3 are stored in the congestion information table T1. Further, in the congestion information table T1 of FIG. 9, for example, as the congestion information with the camera ID being "3", based on the motion difference data H3 of the camera 2-3, a determination result that the congestion flag is "on", that is, the "congestion state" is obtained.

[0065] Returning to FIG. 7, when the congestion determination unit 71 stores the congestion information table T1 (S13), the processing according to the flowchart of FIG. 7 ends.

[0066] As described above, in the congestion management server 7 of the present embodiment, the congestion determination unit 71 acquires image data for each camera 2 in the present system 1 (S10), and generates various current congestion information (S11, S12). Thereby, the congestion management server 7 can manage various congestion information of the image data for each camera 2. Further, according to the motion difference data H (S11), it is possible to calculate the degree of congestion according to the number of people within the range of the captured image of each camera 2 without using image recognition particularly.

[0067] 2-3. Image distribution process The image distribution process (S2 in FIG. 5) in the congestion management server 7 will be described with reference to FIGS. 10 and 11. In the image distribution process (S2), based on the congestion information for each camera 2, the distribution destination for distributing the image data of each camera 2 in the plurality of traffic flow extraction servers 8-1 to 8-3 is determined.

[0068] FIG. 10 is a flowchart illustrating the image distribution process (S2) in the congestion management server 7. FIG. 11 is a diagram for explaining the image distribution process in the congestion management server 7.

[0069] Each process shown in the flowchart of FIG. 10 is executed by the control unit 70 of the congestion management server 7 that functions as the image distribution unit 72. The flowchart of FIG. 10 starts, for example, in a state where the congestion information table T1 is stored by the operation of the congestion determination unit 71 (S13 in FIG. 7).

[0070] In the flowchart of FIG. 10, first, the image distribution unit 72 of the congestion management server 7 acquires, for example, from the storage unit 74, the current image data of the plurality of cameras 2 and various congestion information in the congestion information table T1 (S20).

[0071] Next, the image distribution unit 72 selects one camera i from the plurality of cameras 2 (S21). In the example of FIG. 5, cameras 2-1, 2-2, and 2-3 are sequentially selected as camera i for each step S21.

[0072] The image distribution unit 72 determines whether the congestion flag of the selected camera i is "on" or not by referring to, for example, the congestion information table T1 in FIG. 9 (S22). The image distribution unit 72 makes this determination by referring to the congestion information corresponding to the camera ID of camera i. For example, in the example of FIG. 9, the congestion flag referred to when camera 2-1 is selected is "off".

[0073] If the congestion flag of the selected camera i is "off" (NO in S22), the corresponding captured image is determined to be in an uncongested state. In this case, the image distribution unit 72 does not particularly execute the processes of steps S23 and S24 and proceeds to step S25.

[0074] The image distribution unit 72 determines whether all of the cameras 2-1, 2-2, and 2-3 have been selected in step S21 (S25). Until all of the cameras 2-1, 2-2, and 2-3 are selected (NO in S25), the image distribution unit 72 repeats the processes after step S21.

[0075] In the example of FIG. 5, when cameras 2-1 and 2-2 are each selected, since the congestion flag is "off" in the congestion information table T1 (NO in S22), the process proceeds to step S25. Then, since not all of the cameras have been selected yet (NO in S25), camera 2-3 is selected (S21). At this time, the congestion flag referred to in the congestion information table T1 in FIG. 9 when camera 2-3 is selected is "on".

[0076] If the congestion flag of the selected camera i is "on" (YES in S22), the corresponding captured image is determined to be in a congested state. In this case, the image distribution unit 72 performs a process for dividing the captured image indicated by the image data of camera i (S23).

[0077] An example of the process in step S23 is shown in FIG. 11. The image distribution unit 72 sets regions R1 to R4 for dividing the captured image I23 indicated by the image data into a predetermined number of divisions (for example, 4 or 9). FIGS. 11(A) and 11(B) show an example of the number of divisions being "4". In the example of FIG. 11(B), the divided images I23a to I23d corresponding to the respective regions R1 to R4 are set so as to overlap in adjacent portions.

[0078] In step S23, the image distribution unit 72 calculates the congestion level of each of the divided images I23a to I23d in the same manner as in step S12 of FIG. 7, for example, by referring to the congestion information table T1 of FIG. 9. For example, when the image distribution unit 72 calculates a congestion level less than a predetermined threshold, it adopts the number of divisions. On the other hand, when the image distribution unit 72 calculates a congestion level equal to or greater than a predetermined threshold, it repeats the above calculation using a larger number of divisions. The predetermined threshold is a value small enough to enable real-time traffic flow extraction processing in each of the plurality of traffic flow extraction servers 8, and is set in the same manner as the threshold in step S12, for example.

[0079] The image data indicating the divided images I23a to I23d is held in the internal memory of the control unit 70 of the congestion management server 7, for example, in association with the respective congestion levels. Based on such congestion levels, the image distribution unit 72 may, for example, integrate two or more of the divided images I23a and I23d within the range less than the above-mentioned threshold to form a divided image I23ad (see FIG. 6(F)).

[0080] In the example of FIG. 5, when camera 2-3 is selected as camera i in step S21, since the congestion flags of the respective congestion information in the congestion information table T1 are "on" (YES in S22), the image I23 of camera 2-3 is divided as shown in FIGS. 11(A) and 11(B).

[0081] The image distribution unit 72 determines the flow line extraction server 8 that is the destination for distributing the divided image of the selected camera i based on the congestion information of the plurality of cameras 2 (S24). The flow line extraction server 8 corresponding to each camera 2 is set in advance as the supply destination of the image data of that camera 2. For example, the image distribution unit 72 refers to the congestion information table T1 and determines an additional destination from the flow line extraction servers 8 corresponding to the cameras 2 with the congestion flag being "off". Then, the image distribution unit 72 determines the distribution of the divided image to the flow line extraction server 8 that is set in advance as the supply destination corresponding to the camera 2 with the congestion flag being "on". In the example of FIG. 5, the distribution destinations of the divided images I23b and I23c are determined to be the flow line extraction servers 8-1 and 8-2 respectively. And the distribution of the divided image I23ad, which is the integration of the divided images I23a and I23d, to the flow line extraction server 8-3 is determined.

[0082] In step S24, the image distribution unit 72 may determine the distribution destination so that the congestion levels of the respective flow line extraction servers 8 are within a predetermined range where they can be regarded as being of the same degree, based on, for example, the congestion levels of each camera 2 in the congestion information table T1 and the congestion levels calculated for each divided image. Also, the congestion level for each camera 2 in the congestion information table T1 may be updated according to the determination of the distribution destination, and subsequent processing may be executed based on the updated congestion information table T1.

[0083] When the control unit 70 as the image distribution unit 72 determines that, for example, all the cameras 2-1 to 2-3 have been selected (YES in S25), it controls the communication interface 75 to transmit each image data to the plurality of flow line extraction servers 8 (S26). The image data of the captured image or the divided image of each camera 2 is transmitted to the flow line extraction server 8 corresponding to each camera 2. Further, the image data of the divided image is transmitted to the flow line extraction server 8 that is additionally determined as the distribution destination. At this time, each image data is transmitted, for example, in association with the camera ID of each camera 2.

[0084] When the image distribution unit 72 transmits the image data to the plurality of flow line extraction servers 8 (S26), it ends the processing shown in this flowchart.

[0085] As described above, based on the congestion information of the plurality of cameras 2, the image distribution unit 72 divides (S22, S23) the captured images of the cameras 2 determined to be in a "congested state" based on the congestion information among the captured images of the respective cameras 2, and distributes them to the plurality of flow line extraction servers 8 (S24). Thereby, the processing load in each of the plurality of flow line extraction servers 8 can be dispersed.

[0086] Note that in step S23, the threshold value of the congestion level is not limited to the same value as in step S12 of FIG. 7, and different values may be set as appropriate. For example, the image distribution unit 72 may calculate, as a threshold value, the average value of the congestion levels of the plurality of cameras 2 regarded as the vicinity of the selected camera i based on the congestion information table T1. In this case, for example, a combination of cameras regarded as the vicinity of each camera is preset based on the positional relationship among the plurality of cameras 2.

[0087] Also, in step S23, the region division of the image may be performed such that, for example, as shown in FIG. 11(B), each divided image partially overlaps with an adjacent divided image. By dividing the image in this overlapping manner, it is possible to suppress detection omission when detecting a moving object such as a person 11 at the boundary portion of the divided images in the subsequent flow line extraction process by the plurality of flow line extraction servers 8.

[0088] 2-4. Flow Line Extraction Process The operation of the flow line extraction server 8 will be described with reference to FIGS. 12 and 13. Each of the flow line extraction servers 8-1 to 8-3 performs a flow line extraction process (S3 in FIG. 5) of generating individual flow line information based on the image data of the camera 2 corresponding to the flow line extraction server 8 among the plurality of cameras 2-1 to 2-3.

[0089] FIG. 12 is a flowchart illustrating the flow line extraction process (S3) in the flow line extraction server 8. FIG. 13 is a diagram for explaining the flow line extraction process. Each process shown in the flowchart of FIG. 12 is executed by a control unit 80 that functions as a flow line extraction unit 81 in each flow line extraction server 8. The flowchart of FIG. 12 is started, for example, at the frame period of a moving image.

[0090] In the flowchart of FIG. 12, first, the flow line extraction unit 81 receives image data from the congestion management server 7 via the communication interface 85 (S30). For example, the flow line extraction server 8-1 in the example of FIG. 5 receives the image data 21 of the captured image I21 by the corresponding camera 2-1 (i.e., the responsible camera) and the image data 23b of the divided image I23b.

[0091] In the received image data, the flow line extraction unit 81 performs image recognition processing for object detection to detect an image position indicating the position on the image of a person 11 or the like that is the recognition target (S31). An example of the processing in step S31 is shown in FIGS. 13(A) and (B).

[0092] FIG. 13(A) illustrates the image position P21 detected by the corresponding flow line extraction server 8-1 in the captured image I21 in a non-congested state. FIG. 13(B) illustrates the image position P23b detected in the divided image I23b by the same flow line extraction server 8-1 as in FIG. 13(A).

[0093] In the image recognition processing for object detection in step S31, as illustrated in FIG. 13(A), for example, a rectangular detection region 40 is recognized as the detection result of the object to be recognized in the captured image I21. The flow line extraction unit 81 calculates, for example, the coordinates on the image of the center point in the detection region 40 as the image position P21. The image position P21 is held, for example, in the internal memory of the control unit 80.

[0094] In the examples of FIGS. 13(A) and (B), the flow line extraction unit 81 of the flow line extraction server 8-1 detects the image position P21 in the image data 21 of the responsible camera 2-1 as shown in FIG. 13(A), and detects the image position P23b in the image data 23b of the non-responsible camera 2-3 from the divided image I23b.

[0095] Next, the flow line extraction unit 81 determines whether there is image data of the camera 2 outside the scope of responsibility that does not correspond to the own vehicle in the image data in which the image position was detected in step S31 (S32). For example, the flow line extraction unit 81 refers to the camera ID associated with each image data to determine whether it is the image data of the camera 2 outside the scope of responsibility.

[0096] If the flow line extraction unit 81 determines that there is image data of the camera 2 outside the scope of responsibility (YES in S32), it transmits information indicating the image position detected from the above image data to the flow line extraction server 8 corresponding to the camera 2 among the plurality of flow line extraction servers 8-1 to 8-3 (S33). In the present embodiment, the plurality of flow line extraction servers 8 transmit and receive information on the image position to each other via their respective communication interfaces 85.

[0097] For example, since the flow line extraction unit 81 of the flow line extraction server 8-1 determines that the image data 23b of the divided image I23b (FIG. 13(B)) is the image data of the camera 2-3 outside the scope of responsibility (NO in S32), it transmits the information on the image position P32 in the divided image I23b to the flow line extraction server 8-3 corresponding to the camera 2-3 (S33).

[0098] On the other hand, if the flow line extraction unit 81 determines that there is no image data of the camera 2 outside the scope of responsibility (NO in S32), it does not particularly execute step S33 and proceeds to step S34.

[0099] The flow line extraction unit 81 determines whether the communication interface 85 of the own vehicle has received information on the image position from other flow line extraction servers 8 among the plurality of flow line extraction servers 8-1 to 8-3, for example, within a predetermined period (S34).

[0100] When information on the image position of the assigned camera 2 is received from another trajectory extraction server 8 (YES in S34), the trajectory extraction unit 81 updates the detection result of the image position of the assigned camera 2 based on the received information on the image position (S35). FIG. 13(C) illustrates the image position P23 of the detection result in the captured image I23 in a congested state. For example, the trajectory extraction unit 81 of the trajectory extraction server 8-3 receives information on the image position P23b of the corresponding camera 2-3 from the trajectory extraction server 8-1 (YES in S32) and adds it to the image position P23 of the detection result (S35).

[0101] On the other hand, when information on the image position is not received from another trajectory extraction server 8 (NO in S34), the trajectory extraction unit 81 does not specifically execute step S35 and proceeds to step S36.

[0102] Next, the trajectory extraction unit 81 calculates a map position indicating the position of a recognition target such as a person 11 on the map of the environment 6, for example, by performing an operation to coordinate-transform the image position of the detection result (S36). In each of the trajectory extraction servers 8, the trajectory extraction unit 81 performs coordinate transformation based on parameters that define the transformation from the image position of the assigned camera 2 to the map position, which are stored in advance in the storage unit 84 of its own device, for example.

[0103] The trajectory extraction unit 81 acquires past trajectory information within a predetermined range, such as from several frames before the current time when the map position was calculated (S37). The trajectory extraction unit 81 acquires past trajectory information such as the map position and speed from the individual trajectory information stored in the storage unit 84 of its own device, for example.

[0104] Next, based on the calculated current map position and the acquired past movement path information, the movement path extraction unit 81 generates current individual movement path information (S38). For example, the movement path extraction unit 81 calculates the map position in the current frame, i.e., the predicted position, predicted from the map position and speed in the past movement path information of each movement path, and associates the current map position with the movement path for which the distance between the predicted position and the calculated map position is closer than a predetermined value, thereby generating individual movement path information. The predetermined value of the distance is, for example, a value small enough to be regarded as the same point on the map. Also, since the map positions that could not be associated are considered to indicate the positions of people coming from outside the imaging range, etc., such positions are included in the individual movement path information. The generated individual movement path information is stored, for example, in the storage unit 84.

[0105] The movement path extraction unit 81 transmits the generated individual movement path information to the movement path combination server 9 via, for example, the communication interface 85 (S39), and ends the processing shown in this flowchart. Through the above processing in each of the movement path extraction servers 8-1, 8-2, 8-3, the individual movement path information L1, L2, L3 is transmitted from the movement path extraction servers 8-1, 8-2, 8-3 to the movement path combination server 9 respectively.

[0106] As described above, in each of the plurality of movement path extraction servers 8, the movement path extraction unit 81 acquires image data from the image distribution unit 72 of the congestion management server 7 (S30) and generates individual movement path information based on the image data of the corresponding camera (S38). Thereby, the processing load of image recognition is dispersed among the plurality of movement path extraction servers 8, and it is possible to suppress the prolongation of the processing time in some of the movement path extraction servers.

[0107] Also, the movement path extraction unit 81 detects the image position of the moving object that is the recognition target by image recognition processing from the image data (S31), and transmits and receives the image position among the plurality of movement path extraction servers 8 (S33, S34), thereby using the image position by the assigned camera 2 for each movement path extraction server 8. Thereby, it is possible to obtain individual movement path information based on the image data of the assigned camera 2 in each movement path extraction server 8.

[0108] 2-5. Movement Path Combination Processing The operation of the flow line combination server 9 will be described with reference to FIG. 14. The flow line combination server 9 performs a flow line combination process (S4 in FIG. 5) of combining the individual flow line information L1, L2, and L3 generated in each of the plurality of flow line extraction servers 8-1 to 8-3 into overall flow line information L0 indicating all the flow lines within the imaging ranges of the plurality of cameras 2-1 to 2-3.

[0109] FIG. 14 is a flowchart illustrating the flow line combination process (S4) by the flow line combination server 9. Each process shown in the flowchart of FIG. 14 is executed by a control unit 90 that functions as a flow line combination unit 91 in the flow line combination server 9. The flowchart of FIG. 14 starts, for example, at the frame period of a video.

[0110] First, the flow line combination unit 91 receives the respective individual flow line information L1, L2, and L3 from each of the plurality of flow line extraction servers 8-1 to 8-3 via the communication interface 95 (S40).

[0111] Next, the flow line combination unit 91 generates overall flow line information L0 based on the received plurality of individual flow line information L1 to L3 (S41). The flow line combination unit 91, for example, compares the map positions of the respective flow lines in the separate individual flow line information L1 to L3 with each other and combines the flow lines considered to be those of the same person. The combination of the flow lines is performed, for example, in consideration of a distance range that can be regarded as the same on the map, similar to step S38.

[0112] The flow line combination unit 91 outputs, for example, the overall flow line information L0 and map data indicating the map of the environment 6 to the display terminal 4 via the output interface 96 (S42). The map data is, for example, stored in advance in the storage unit 84. Thereby, the flow line combination unit 91 causes the overall flow line information to be superimposed on the map data and displayed on the display terminal 4.

[0113] The flow line combination unit 91 ends the process shown in this flowchart by outputting the overall flow line information L0 (S42).

[0114] As described above, the traffic line connection unit 91 combines the individual traffic line information L1 to L3 obtained from the plurality of traffic line extraction servers 8 to generate the overall traffic line information L0 (S41, S42), and outputs it in combination with, for example, map data (S42). As a result, the traffic lines based on the plurality of cameras 2 can be combined and displayed, making it easier for the user 3 to analyze the traffic lines across the imaging ranges of the respective plurality of cameras 2.

[0115] 3. Effects, etc. As described above, the traffic line detection system 1 according to the present embodiment is an example of a moving object detection system that detects a plurality of moving objects such as a person 11 based on a captured image. The traffic line detection system 1 includes a plurality of cameras 2 (a plurality of imaging devices) that capture images at a predetermined cycle to generate image data respectively, a plurality of traffic line extraction servers 8 (a plurality of image recognition devices) that perform image recognition on the image data captured by each camera to detect moving objects in the captured images shown by the respective image data, and a congestion management server 7 (information management device) that manages the image data supplied from each camera to each traffic line extraction server. The congestion management server 7 includes a communication interface 75 (communication unit) that performs data communication with each of the cameras 2-1, 2-2, 2-3 and the traffic line extraction servers 8-1, 8-2, 8-3, and a control unit 70 that controls the operation of the communication interface 75. The control unit 70 manages congestion information indicating the quantity of moving objects localized within the range of the captured image shown by each image data for each of the cameras 2-1, 2-2, 2-3, for example, in a congestion information table T1 (S1), and controls the communication interface 75 to distribute the image data captured by the plurality of cameras 2 and supply it to each of the traffic line extraction servers 8-1, 8-2, 8-3 according to the magnitude of the quantity such as the congestion level indicated by the congestion information (S2).

[0116] According to the above traffic line detection system 1, the image data captured by the plurality of cameras 2 is distributed and supplied to the plurality of traffic line extraction servers 8 according to the magnitude of the quantity indicated by the congestion information. As a result, the process of detecting moving objects in the plurality of traffic line extraction servers 8 can be performed efficiently.

[0117] In this embodiment, the control unit 70 functions as an image distribution unit 72 and sets the plurality of flow line extraction servers at the distribution destinations for the image data in which the congestion information indicates a quantity equal to or greater than a predetermined quantity among the image data obtained by the plurality of cameras 2 (S2). Thereby, it is possible to suppress an increase in the load of the process of detecting a moving body by performing image recognition in a part of the plurality of flow line extraction servers 8, and it is possible to efficiently perform the processes in the plurality of flow line extraction servers 8.

[0118] In this embodiment, the control unit 70 functions as an image distribution unit 72 and controls the communication interface 75 so as to separately supply the plurality of divided data obtained by dividing the captured image indicated by one image data into a plurality of regions to the plurality of flow line extraction servers at the distribution destinations (S2). Thereby, it is possible to easily disperse the loads of the processes in the respective flow line extraction servers 8-1, 8-2, 8-3 to the same degree.

[0119] In this embodiment, each of the flow line extraction servers 8-1, 8-2, 8-3 repeatedly detects a moving body and generates individual flow line information L1, L2, L3 as an example of the flow line information indicating the flow line by the moving body (S3). Each of the flow line extraction servers 8-1, 8-2, 8-3 may perform, for example, only image recognition and transmit the detection result of the moving body via the communication interface 85, respectively. In this case, the generation of the flow line information may be performed by, for example, the flow line combination server 9 that has received the detection result or an information processing device outside the flow line detection system 1.

[0120] The flow line detection system 1 of this embodiment further includes a flow line combination server 9 (information integration device) that generates overall flow line information L0 as an example of the information indicating the flow line of the moving body in the entire range imaged by the plurality of cameras 2 based on the individual flow line information L1, L2, L3 generated by each of the flow line extraction servers 8-1, 8-2, 8-3 (S4). Thereby, for example, the user 3 can analyze the flow line in the entire imaging range by the plurality of cameras 2.

[0121] In this embodiment, the control unit 70 calculates a motion difference as an example of the amount of movement of the moving object in the captured image based on the image data of a plurality of frames sequentially captured by each of the cameras 2-1, 2-2, and 2-3, and generates congestion information based on the calculated motion difference (S1). Thereby, by using the motion difference that can be calculated with a small amount of calculation, it is possible to generate congestion information including information based on the latest frame while suppressing the processing load on the congestion management server 7.

[0122] The congestion management server 7 of this embodiment is an example of an information management device in a traffic flow detection system 1 including a plurality of cameras 2 and a plurality of traffic flow extraction servers 8 that respectively recognize a plurality of moving objects in the captured images indicated by the image data based on the image data from each of the cameras 2-1, 2-2, and 2-3. The congestion management server 7 includes a communication interface 75 that performs data communication with each of the cameras 2-1, 2-2, and 2-3 and each of the traffic flow extraction servers 8-1, 8-2, and 8-3, and a control unit 70 that controls the operation of the communication interface 75 so as to manage the image data supplied from each of the cameras 2-1, 2-2, and 2-3 to each of the traffic flow extraction servers 8-1, 8-2, and 8-3. The control unit 70 manages congestion information in which a moving object is localized within the range of the captured image indicated by each image data for each of the cameras 2-1, 2-2, and 2-3 (S1), and controls the communication interface 75 to distribute the image data from the plurality of cameras 2 and supply it to each of the traffic flow extraction servers 8-1, 8-2, and 8-3 according to the magnitude of the quantity indicated by the congestion information (S2).

[0123] According to the above congestion management server 7, in the traffic flow detection system 1, the plurality of traffic flow extraction servers 8 can efficiently perform the process of recognizing the moving object.

[0124] (Embodiment 2) Hereinafter, Embodiment 2 of the present disclosure will be described with reference to FIG. 15. In Embodiment 2, the operation of the congestion management server 7 that manages the distribution of congestion information and image data is described using the positional relationship between the plurality of cameras 2.

[0125] Hereinafter, the description of the configuration and operation similar to those of the flow line detection system 1 and the congestion management server 7 according to Embodiment 1 will be omitted as appropriate, and the flow line detection system 1 and the congestion management server 7 according to the present embodiment will be described.

[0126] FIG. 15 is a diagram for explaining the propagation of the congestion state among a plurality of cameras in the present embodiment. FIG. 15(A) illustrates the adjacent information of a plurality of cameras 2. The adjacent information is an example of arrangement information indicating the positional relationship between the cameras in a state where the plurality of cameras 2 are arranged in the environment 6.

[0127] In FIG. 15(A), the relationship between adjacent cameras among the plurality of cameras 2 is illustrated by solid lines. For example, in the illustration of FIG. 15(A), the camera 2-1 is adjacent to the cameras 2-2 and 2-3. The adjacent information is stored, for example, in advance in the storage unit 74 of the congestion management server 7. FIG. 15(B) is a diagram for explaining the congestion determination process of the present embodiment based on the adjacent information of FIG. 15(A).

[0128] In the present embodiment, the congestion determination unit 71 of the congestion management server 7 first generates congestion information from the image data of one of the plurality of cameras 2, for example, in the same manner as in Embodiment 1. The congestion information in the present embodiment does not particularly include motion difference data, and may be information in which a camera ID and a congestion flag are associated. FIG. 15(B) shows an example in which the congestion information of the camera 2-1 is generated and the congestion flag in the congestion information is "ON".

[0129] Next, the congestion determination unit 71 updates the congestion information about the cameras adjacent to the camera whose congestion flag in the congestion information is "ON" based on the adjacent information of the plurality of cameras 2. Cameras adjacent to a camera with a "ON" congestion flag are considered likely to become congested in the future. Therefore, the congestion determination unit 71 of the present embodiment generates congestion information indicating that the adjacent cameras are "scheduled to be congested" by means of a flag different from the congestion flag. In the example of FIG. 15(B), the cameras 2-2 and 2-3 adjacent to the camera 2-1 are set to "scheduled to be congested".

[0130] The congestion determination unit 71 changes the congestion flag to "ON" for cameras that are "scheduled to be congested", for example, after a predetermined number of frames from the current frame, and sets them as the processing targets of the image distribution unit 72. The predetermined number of frames indicates a period during which congestion in the imaging ranges of adjacent cameras is assumed to spread. The number of frames may be set according to the moving objects that are the recognition targets of the traffic flow extraction process in the plurality of traffic flow extraction servers 8.

[0131] Based on the congestion information in the plurality of cameras 2, the congestion determination unit 71 updates the congestion flag of each camera every time the congestion determination process executed at the frame period of the plurality of cameras 2 is performed. For cameras with the congestion flag "ON", for example, if they are not adjacent to other cameras with the congestion flag "ON" after a predetermined number of frames from the current frame, the congestion flag may be updated to "OFF".

[0132] According to the above processing, based on the adjacency information of each camera in the plurality of cameras 2, the congestion flag of the congestion information is updated, and a congestion determination process can be executed to reflect the propagation of the congestion state caused by moving objects such as the person 11. Thereby, in the image distribution process, the images of the cameras adjacent to the cameras with the congestion flag "ON" in the plurality of cameras 2 can be pre-divided and distributed to the plurality of traffic flow extraction servers 8, making it easier to perform the real-time traffic flow extraction process.

[0133] As described above, the control unit 70 of the congestion management server 7 according to the present embodiment functions as the congestion determination unit 71 and updates the congestion information based on the adjacency information as an example of the information regarding the arrangement of the plurality of cameras 2 (see FIGS. 15(A) and (B)). Thereby, for example, the division of the image can be started before the moving object actually moves in the imaging range of each camera.

[0134] (Embodiment 3) Hereinafter, Embodiment 3 of the present disclosure will be described with reference to FIG. 16. In Embodiment 1, an example in which the image distribution unit 72 operates based on the congestion flag of the determination result by the congestion determination unit 71 in the congestion management server 7 was described. In Embodiment 3, the image distribution unit 72 will be described as a congestion management server 7 that first divides the images of all cameras in a plurality of cameras 2 and distributes them according to congestion information, without particularly using the congestion flag.

[0135] The operation of the image distribution unit 72 in the congestion management server 7 of the present embodiment will be described with reference to FIG. 16. FIG. 16 is a flowchart illustrating the image distribution process by the image distribution unit 72 of Embodiment 3. Hereinafter, the description of the same processes as the image distribution process (FIG. 10) of Embodiment 1 will be omitted as appropriate.

[0136] First, a control unit 70 as the image distribution unit 72 in the congestion management server 7 of the present embodiment acquires image data of a plurality of cameras 2 (S20A).

[0137] Based on the acquired image data of each camera, the control unit 70 calculates the degree of congestion in each captured image (S50). The control unit 70 calculates the degree of congestion in the same manner as, for example, the congestion determination process (FIG. 7) of Embodiment 1. The control unit 70 calculates an average congestion degree indicating the average degree of congestion for all cameras as the average of the congestion degrees of each camera (S51).

[0138] The control unit 70 selects one camera i from the plurality of cameras 2 (S21), and divides the image indicated by the image data of camera i into a predetermined number of divisions (S23A). The predetermined number of divisions is set in the same manner as, for example, step S23 in FIG. 10. The control unit 70 calculates the degree of congestion for each divided image in the same manner as in step S50, associates it with the image data of each divided image, and stores it in, for example, an internal memory. The predetermined number of divisions may be appropriately set according to the average congestion degree in step S51 so that the degree of congestion in the divided image data is smaller than the average congestion degree.

[0139] Next, based on the congestion level and average congestion level of the divided images of camera i, the control unit 70 determines the distribution destination of the divided images from the plurality of flow line extraction servers 8 (S24A). Specifically, for each of the plurality of flow line extraction servers 8, the control unit 70 determines the distribution destination such that the total congestion level of the divided images to be distributed is approximately the same as the average congestion level. For example, the control unit 70 performs the distribution so that the total congestion level of the divided images is smaller than the average congestion level.

[0140] The control unit 70 repeats steps S21 to S24A until all cameras in the plurality of cameras 2 are selected (NO in S25). When all cameras are selected (YES in S25), the control unit 70 transmits the image data indicating the divided images of each camera of the plurality of cameras 2 from the plurality of flow line extraction servers 8 to the flow line extraction server determined as the distribution destination in step S24A (S26A).

[0141] The image distribution unit 72 transmits the image data to the plurality of flow line extraction servers 8 (S26A) and ends the processing shown in this flowchart.

[0142] According to the above processing, the image distribution unit 72 of the present embodiment calculates the average congestion level in the plurality of cameras 2 (S51), divides the images for all cameras (S23A), and distributes them to the plurality of flow line extraction servers 8 based on congestion information such as the congestion level and average congestion level of the divided images (S24A). Thereby, the images of the plurality of cameras 2 can be distributed so that the load in the flow line extraction process in each of the plurality of flow line extraction servers 8 approaches evenly, and it becomes easier to execute the flow line extraction process in real time.

[0143] Also, in step S50 of the present embodiment, the congestion level is calculated in the image data indicating the current frame acquired from the congestion determination unit 71. However, the present invention is not limited to this. For example, the congestion level in the image data of past frames may be calculated, and in step S23A, the image of the current frame may be divided based on the congestion level in the past frames.

[0144] As described above, the control unit 70 of the congestion management server 7 according to the present embodiment functions as an image distribution unit 72, and calculates a congestion degree indicating congestion information in each area for each area that divides the image indicated by each image data among the image data from the plurality of cameras 2 (S23A). Based on the calculated congestion degree, the control unit 70 distributes and transmits each image data from the plurality of cameras 2 to the plurality of traffic line extraction servers 8 (S24A, S26A). Note that in the present embodiment, the control unit 70 may divide and distribute the image of the current frame based on the congestion degree calculated from the image data of the past frames of the moving images captured by the plurality of cameras 2.

[0145] (Embodiment 4) Hereinafter, Embodiment 4 of the present disclosure will be described with reference to FIG. 17. In Embodiment 1, an operation example using the congestion degree and congestion information based on the motion difference in the image data in the congestion management server 7 was described. In Embodiment 4, an operation example of the congestion management server 7 using the congestion information based on the detection results of the plurality of traffic line extraction servers 8 will be described.

[0146] FIG. 17 is a flowchart illustrating the congestion determination process in Embodiment 4. Hereinafter, the same processes as the congestion determination process (FIG. 7) in Embodiment 1 will be omitted from the description as appropriate.

[0147] The congestion determination unit 71 of the present embodiment acquires the detection results of the recognition targets in the image data of each of the plurality of cameras 2 from the plurality of traffic line extraction servers 8, for example, via the communication interface 75 (S60). Specifically, the congestion determination unit 71 acquires the number of recognition targets detected in the image data of the past frames for each of the plurality of cameras 2. The past frame is, for example, the frame transmitted to the plurality of traffic line extraction servers 8 one frame period before the congestion determination process is executed.

[0148] The congestion determination unit 71 generates congestion information for the plurality of cameras 2 based on the number of recognition targets acquired as a detection result (S12A). For example, the congestion determination unit 71 of the present embodiment generates congestion information in which the camera ID of each camera, a congestion flag indicating whether the number of recognition targets is equal to or greater than a predetermined number, and the number of recognition targets are associated with each other. The predetermined number is a sufficiently small number from the same perspective as the predetermined threshold value in step S12 of Embodiment 1.

[0149] According to the above processing, the congestion determination unit 71 of the present embodiment generates congestion information for each of the plurality of cameras 2 based on the number of recognition targets in the image data of past frames. As a result, the image distribution unit 72 performs image distribution processing on the image data of the current frame based on the congestion information.

[0150] As described above, the control unit 70 of the congestion management server 7 according to the present embodiment functions as the congestion determination unit 71, acquires the number of individuals of moving objects such as the person 11 detected by each of the plurality of traffic flow extraction servers 8 (S60), and generates congestion information based on the acquired number of individuals of the moving objects (S12A). As a result, by using the detection results by each traffic flow extraction server, the processing performed by the congestion management server 7 can be reduced, and the congestion determination processing can be efficiently executed.

[0151] (Other Embodiments) As described above, as examples of the technology disclosed in the present application, Embodiments 1 to 4 have been described. However, the technology in the present disclosure is not limited thereto, and is also applicable to embodiments in which changes, substitutions, additions, omissions, etc. are appropriately made. In addition, it is also possible to combine the components described in each of the above embodiments to form a new embodiment. Therefore, other embodiments will be exemplified below.

[0152] In the above-described embodiment, the flow line extraction unit 81 acquired the image data from the congestion management server 7 (S30). The flow line extraction unit 81 of the present embodiment may acquire congestion information in addition to the image data. Further, in the present embodiment, the flow line extraction server 8 may be configured to be communicable with the camera 2 via, for example, the communication interface 85. Thereby, the flow line extraction unit 81 may directly acquire image data other than the divided image from the camera 2 without going through the congestion management server 7, for example, according to the acquired congestion information.

[0153] In Embodiment 1, in the image distribution process (S2), the flow line extraction server 8 corresponding to each camera 2 was set in advance as the supply destination of the image data of the camera 2 (S24). In the image distribution process of the present embodiment, the image data of a plurality of cameras 2 may be supplied to one flow line extraction server 8. For example, the image distribution unit 72 of the present embodiment may transmit the image data 21 and 22 of the cameras 2-1 and 2-2 determined to be in a non-congested state to the flow line extraction server 8-1. In this case, the image distribution unit 72 may distribute the image data 23 of the camera 2-3 determined to be in a congested state to the flow line extraction servers 8-2 and 8-3 so as to divide the captured image I23.

[0154] According to the above-described image distribution process, even when the congestion degree in the captured image I23 is particularly higher than the congestion degrees of the captured images I21 and I22, for example, it is possible to easily disperse the processing load among the plurality of flow line extraction servers 8-1 to 8-3.

[0155] In Embodiment 1, in the flow line detection system 1, the flow line extraction servers 8-1, 8-2, and 8-3 were provided corresponding to the cameras 2-1, 2-2, and 2-3, respectively. In the flow line detection system 1 of the present embodiment, the number of the plurality of cameras 2 and the number of the plurality of flow line extraction servers 8 do not have to be the same. For example, a larger number of flow line extraction servers 8 than the number of the plurality of cameras 2 may be provided.

[0156] In Embodiment 1, the congestion determination unit 71 generated congestion information using the motion difference based on the image data (S11, S12). Instead of the motion difference, the congestion determination unit 71 of this embodiment may acquire the overall flow line information L0 stored in the storage unit 94 of the flow line connection server 9 and use the past flow line information in the overall flow line information L0. Thereby, the congestion determination unit 71 may generate a congestion flag indicating congestion or non-congestion as congestion information for each of the plurality of cameras 2 based on, for example, the position and speed of the recognition target in the past flow line information and the imaging range of each camera.

[0157] In Embodiment 1, the image distribution unit 72 divided (S22, S23) and distributed (S24) the image data based on the congestion flag of the congestion information by the motion difference. Instead of the congestion flag, the image distribution unit 72 of this embodiment may acquire the overall flow line information L0 stored in the storage unit 94 of the flow line connection server 9 and use the past flow line information in the overall flow line information L0. For the images of each of the plurality of cameras 2, the image distribution unit 72 may divide the images so that the number of people in each divided image is approximately the same based on the position of the recognition target such as the person 11 in the past flow line information, and determine the distribution destination so that the total number of people in the divided images at each distribution destination is approximately the same.

[0158] In Embodiment 2, the congestion flag of each camera was set based on the adjacency information of the plurality of cameras 2. The congestion management server 7 of this embodiment is not particularly limited to the above, and may update the congestion information using various information for predicting congestion and distributing the image data. For example, the congestion determination unit 71 of this embodiment may set the congestion flag based on the prior information in the imaging range of each of the plurality of cameras 2 instead of the adjacency information. As the prior information, for example, in the environment 6 such as a store sales floor, various kinds of knowledge of the site of the environment 6 to which the present system 1 is applied, such as information indicating a schedule for performing specific sales by dividing time zones, can be used.

[0159] The above-mentioned prior information is defined, for example, such that when the congestion flag is "on" in the imaging range of a specific camera among a plurality of cameras 2, the congestion flag in the imaging range of another camera becomes "on" after a predetermined time. Also, the prior information may be defined to indicate that the congestion flag is "on" in a specific time zone within the imaging range of a specific camera. As a result, even for cameras that are not adjacent, the congestion flag can be set and used for the division and distribution of images by the image distribution unit 72. The above prior information is an example of information regarding the tendency of a moving object such as the person 11 to move.

[0160] Also, in the present embodiment, a learned model that performs congestion prediction by machine learning of prior information such as past trajectory information may be applied to manage the congestion state in the present system 1. In such machine learning, for example, additional information indicating various events related to the time, day of the week, season, weather, and environment 6 when using the present system 1 may be used.

[0161] In the above embodiment, the trajectory detection system 1 has been described as an example of a moving object detection system. In the present embodiment, the moving object detection system is not particularly limited to the trajectory detection system 1, and may be applied, for example, to a monitoring system for monitoring the movement of a moving object such as a person by a plurality of cameras, or a tracking system for tracking. Also, in the present embodiment, the detection target of the moving object detection system is not particularly limited to a person, and may be various moving objects. For example, the moving object to be detected may be a cart in an environment 6 such as a store, an automatic transporter in an environment 6 such as a factory, or various vehicles.

[0162] In the above embodiment, the trajectory extraction server 8 has been described as an example of an image recognition device. In the present embodiment, the image recognition device is not particularly limited to the trajectory extraction server 8, and may be configured, for example, as a GPU or a TPU. Also, when applying a learned model by deep learning to the processing of image recognition, the image recognition device may be configured as dedicated hardware specialized for the processing of deep learning.

[0163] In the above embodiment, the congestion management server 7 has been described as an example of the information management device. In this embodiment, the information management device may be, for example, a PC. Similarly, the information integration device and the image recognition device may each be, for example, a PC.

[0164] As described above, the embodiments have been described as examples of the technology in the present disclosure, and the accompanying drawings and detailed description have been provided for this purpose.

[0165] Therefore, among the components described in the attached drawings and the detailed description, not only are there components essential for solving the problem, but there may also be components that are not essential for solving the problem in order to illustrate the above technology. Therefore, the fact that such non-essential components are described in the attached drawings or the detailed description should not be interpreted as immediately indicating that such non-essential components are essential.

[0166] Furthermore, since the above-described embodiments are intended to illustrate the technology in the present disclosure, various modifications, substitutions, additions, omissions, and the like can be made within the scope of the claims or their equivalents. [Industrial Applicability]

[0167] The present disclosure relates to various imaging systems that process information based on multiple imaging devices in real time. Technology For example, it can be applied to motion detection systems, surveillance systems, and tracking systems. such as a moving body detection system is applicable to.

Claims

1. An information management method, comprising: a step in which a computer acquires image data from a plurality of imaging devices; a step in which a control unit of the computer generates congestion information indicating the quantity of moving objects localized within the range of the captured image for each of the imaging devices; a step in which, in first image data obtained by a first imaging device among the plurality of imaging devices, when the magnitude of the quantity indicated by the congestion information is less than a predetermined quantity, the control unit supplies the first image data to a first image recognition device among the plurality of image recognition devices; a step in which, in the first image data, when the magnitude of the quantity indicated by the congestion information is greater than or equal to the predetermined quantity, the control unit distributes the first image data and supplies it to at least the first image recognition device and a second image recognition device different from the first image recognition device among the plurality of image recognition devices. An information management method.

2. Each of the image recognition devices repeatedly detects the moving object and generates trajectory information indicating the trajectory of the moving object. The information management method according to Claim 1.

3. The information integration device further includes a step of generating information indicating the trajectory of the moving object in the entire range captured by the plurality of imaging devices based on the trajectory information generated by each of the image recognition devices. The information management method according to Claim 2.

4. The control unit: calculates the amount of movement of the moving object in the captured image based on image data of a plurality of frames sequentially captured by each of the imaging devices; generates the congestion information based on the calculated amount of movement. The information management method according to any one of Claims 1 to 3.

5. The control unit: acquires the number of individuals of the moving object detected by each of the image recognition devices; generates the congestion information based on the acquired number of individuals of the moving object. The information management method according to any one of Claims 1 to 4.

6. The control unit determines in advance to distribute the first image data before the magnitude of the quantity indicated by the congestion information in the first image data obtained by the first imaging device becomes greater than or equal to the predetermined quantity in accordance with at least one of the congestion information regarding the imaging device adjacent to the first imaging device among the plurality of imaging devices and information indicating the tendency of the congestion information to change due to the movement of the moving object in the environment where the plurality of imaging devices are arranged, and updates the congestion information. The information management method according to any one of claims 1 to 5.

7. The control unit In each piece of image data obtained by the plurality of imaging devices, for each region that divides the image indicated by each piece of image data, calculates a congestion degree indicating congestion information in the region, Based on the calculated congestion degree, distributes and transmits each piece of image data obtained by the plurality of imaging devices to the plurality of image recognition devices The information management method according to any one of claims 1 to 6.

8. The plurality of imaging devices includes a second imaging device corresponding to the second image recognition device, In the second image data obtained by the second imaging device, the magnitude of the quantity indicated by the congestion information is less than the predetermined amount The information management method according to any one of claims 1 to 7.

9. The control unit Supplies the second image data to the second image recognition device, In the first image data, when the magnitude of the quantity indicated by the congestion information is greater than or equal to the predetermined amount, supplies a plurality of divided data obtained by dividing the captured image indicated by the first image data into a plurality of regions to the first and second image recognition devices as distribution destinations separately The information management method according to claim 8.

10. A computer program which, when executed by a control unit of a computer, causes the computer to acquire image data from a plurality of imaging devices; generate congestion information indicating the quantity of moving objects in the captured image indicated by each piece of image data for each imaging device; in the first image data obtained by the first imaging device among the plurality of imaging devices, when the magnitude of the quantity indicated by the congestion information is less than a predetermined amount, supply the first image data to the first image recognition device among the plurality of image recognition devices; in the first image data, when the magnitude of the quantity indicated by the congestion information is greater than or equal to the predetermined amount, distribute the first image data and supply it to at least the first image recognition device and a second image recognition device different from the first image recognition device among the plurality of image recognition devices Computer program.

Images