Information management method and computer program
The system efficiently manages image data from multiple cameras by using a congestion management server to distribute data and reduce processing delays, ensuring real-time flow line detection and display.
Patent Information
- Application Number
- JP2024139502
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2020-04-13
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-11-26
AI Technical Summary
Existing systems struggle to efficiently manage and process image data from multiple imaging devices, particularly when congestion occurs, leading to delays in real-time flow line detection and display.
A system that includes a congestion management server to determine congestion levels and distribute image data to multiple flow line extraction servers, dividing images if necessary, to distribute processing load and maintain real-time flow line detection.
The system efficiently processes image data from multiple cameras, reducing processing delays and enabling real-time display of flow lines by distributing the load across multiple servers.
Smart Images

Figure 2024161559000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to Information management method and computer program Regarding. [Background technology]
[0002] Patent Document 1 discloses a monitoring system including one or more monitoring cameras, a monitoring server, and a mobile terminal carried by a user. The monitoring server of Patent Document 1 extracts the movement line of a moving object such as a person from the video of the monitoring camera, and selects the movement line included in the surrounding area of the mobile terminal. Based on the selected movement line, the monitoring server identifies an image of an area including the passage path of the movement line and presents it to the user. In this way, Patent Document 1 aims to improve the efficiency of the user's work of checking the image by presenting the user with an image according to the analysis result of the movement line. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent No. 6532234 Summary of the Invention [Problem to be solved by the invention]
[0004] The present disclosure relates to an image capturing method using a plurality of image capturing devices. Image recognition device for Processing can be made more efficient Information management method and Computer Programs to provide. [Means for solving the problem]
[0005] According to one aspect of the present disclosure Information management method teeth, A step of a computer acquiring image data from a plurality of image capture devices; Manages congestion information indicating the number of moving objects present within the range of the captured image indicated by each image data for each imaging device A step of generating , The method includes a step of, when a magnitude of a quantity indicated by congestion information in first image data by a first imaging device of the multiple imaging devices is less than a predetermined amount, the control unit supplies the first image data to a first image recognition device of the multiple image recognition devices, and, when a magnitude of a quantity indicated by congestion information in the first image data is equal to or greater than the predetermined amount, the control unit distributes the first image data and supplies it to at least the first image recognition device and a second image recognition device different from the first image recognition device of the multiple image recognition devices. .
[0006] According to one aspect of the present disclosure Computer Programs teeth, When executed by a control unit of a computer, the method causes the computer to acquire image data from a plurality of image capture devices;Congestion information indicating the number of moving objects present within the range of the captured image indicated by each image data for each imaging device A step of generating , The method executes a step of supplying the first image data to a first image recognition device of the plurality of image recognition devices when the magnitude of the quantity indicated by the congestion information in first image data by a first imaging device of the plurality of imaging devices is less than a predetermined amount, and a step of distributing the first image data to at least the first image recognition device and a second image recognition device different from the first image recognition device of the plurality of image recognition devices when the magnitude of the quantity indicated by the congestion information in the first image data is equal to or greater than the predetermined amount. . Effect of the Invention
[0007] The present disclosure Information management method and computer program According to Using multiple imaging devices Captured image Image recognition device for The processing can be made more efficient. [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 is a diagram showing an overview of a traffic line detection system according to a first embodiment of the present disclosure. [Diagram 2] FIG. 1 is a block diagram illustrating a configuration of a congestion management server according to a first embodiment. [Diagram 3] FIG. 1 is a block diagram illustrating a configuration of a trajectory extraction server according to a first embodiment; [Figure 4] FIG. 1 is a block diagram illustrating a configuration of a trajectory connection server according to a first embodiment; [Diagram 5] FIG. 1 is a diagram for explaining an example of the operation of the traffic line detection system 1. [Figure 6] FIG. 2 is a diagram for explaining the operation of the traffic line detection system 1 during congestion. [Figure 7] A flowchart illustrating a congestion determination process in the congestion management server of the first embodiment. [Figure 8] FIG. 1 is a diagram for explaining congestion determination processing according to the first embodiment; [Figure 9] FIG. 13 is a diagram illustrating a congestion information table in the congestion determination process of the first embodiment. [Figure 10] 1 is a flowchart illustrating an example of an image distribution process in a congestion management server according to a first embodiment; [Figure 11] FIG. 1 is a diagram for explaining image distribution processing according to the first embodiment; [Figure 12] 1 is a flowchart illustrating a flow line extraction process in a flow line extraction server according to the first embodiment; [Figure 13] FIG. 1 is a diagram for explaining a flow line extraction process according to the first embodiment; [Figure 14]1 is a flowchart illustrating a flow line connection process in a flow line connection server according to the first embodiment; [Figure 15] FIG. 11 is a diagram for explaining the propagation of a congestion state among a plurality of cameras in the second embodiment. [Figure 16] 11 is a flowchart illustrating an image distribution process according to the third embodiment. [Figure 17] 11 is a flowchart illustrating a congestion determination process according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Hereinafter, the embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of already well-known matters or duplicate explanation of substantially the same configuration may be omitted. This is to avoid the following explanation becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. Note that the inventor(s) provide the attached drawings and the following explanation so that those skilled in the art can fully understand the present disclosure, and do not intend for them to limit the subject matter described in the claims.
[0010] (Embodiment 1) 1. Configuration A traffic line detection system according to a first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an overview of a traffic line detection system 1 according to the present embodiment.
[0011] 1-1. System Overview 1, the system 1 includes multiple cameras 2-1 to 2-3, a congestion management server 7, multiple flow line extraction servers 8-1 to 8-3, and a flow line connection server 9. Hereinafter, the cameras 2-1, 2-2, and 2-3 are collectively referred to as "cameras 2." In addition, the flow line extraction servers 8-1, 8-2, and 8-3 are collectively referred to as "flow line extraction servers 8."
[0012] The present system 1 is an example of a moving object detection system that uses multiple cameras 2 to capture the entire environment 6, such as a sales floor of a store, a workshop, or a factory, by detecting the trajectories, i.e., movement lines, of multiple people 11, etc. moving in the environment 6. The present system 1 may include a display terminal 4 for presenting the movement lines of various people 11 in the environment 6 to a user 3, such as a manager or analyst of the environment 6.
[0013] In this system 1, traffic line extraction servers 8-1, 8-2, and 8-3 are provided corresponding to the cameras 2-1, 2-2, and 2-3. The traffic line extraction server 8 is a server device that executes image recognition processing to extract traffic lines from images captured by the camera 2. The congestion management server 7 is a server device for managing image data supplied from the cameras 2-1, 2-2, and 2-3 to the traffic line extraction servers 8-1, 8-2, and 8-3. The traffic line integration server 9 is a server device for generating information that integrates the traffic lines extracted by the traffic line extraction servers 8-1, 8-2, and 8-3.
[0014] In this embodiment, in the flow line detection system 1 as described above, by using the congestion management server 7, it becomes easier to obtain the flow lines and the like of the entire environment 6 in real time even if congestion occurs due to a large number of people 11.
[0015] The specific configurations of the camera 2, the congestion management server 7, the flow line extraction server 8, and the flow line connection server 9 in this system 1 will be described below.
[0016] 1-2.Camera configuration Camera 2 is an example of an imaging device in system 1. Camera 2 includes an imaging unit that captures an image, and a transmission unit that transmits image data representing the image captured by the imaging unit. The imaging unit is realized by a CCD image sensor, a CMOS image sensor, or the like. The transmission unit includes an interface circuit for communicating with an external device in compliance with a predetermined communication standard (e.g., IEEE802.11, USB).
[0017] The camera 2 in this embodiment is, for example, an omnidirectional camera equipped with a fisheye lens. The camera 2 in the present system 1 is not limited to an omnidirectional camera and may be various imaging devices. In addition, although three cameras 2-1, 2-2, and 2-3 are illustrated in FIG. 1, the number of cameras 2 included in the present system 1 is not limited to three, and may be two or four or more.
[0018] In the present system 1, from the viewpoint of accurately detecting the movement line of a person 11 moving around in the environment 6, the imaging ranges captured by the cameras 2-1, 2-2, and 2-3 in the environment 6 are set so as to partially overlap with each other, for example.
[0019] 1-3. Configuration of the congestion management server 2 is a block diagram illustrating a configuration of the congestion management server 7 of this embodiment. The congestion management server 7 is an example of an information management device in this embodiment. The congestion management server 7 illustrated in FIG. 2 includes a control unit 70, a storage unit 74, and a communication interface 75.
[0020] The control unit 70 controls the operation of the congestion management server 7. The control unit 70 is composed of, for example, a CPU or MPU that cooperates with software to realize predetermined functions. The control unit 70 reads data and programs stored in the storage unit 74 and performs various arithmetic processing to realize various functions. For example, the control unit 70 executes a program including a group of instructions for realizing various processes of the congestion management server 7. The above program is, for example, an application program, and may be provided from a communication network such as the Internet, or may be stored in a portable storage medium.
[0021] 2, the control unit 70 includes, for example, as functional components, a congestion determination unit 71 and an image distribution unit 72. The congestion determination unit 71 performs congestion determination processing, which is processing for determining whether or not the images captured by the cameras 2-1, 2-2, and 2-3 indicate a congestion state, which will be described later. The image distribution unit 72 performs image distribution processing, which is processing for determining destinations to distribute image data of the multiple cameras 2 from the multiple traffic line extraction servers 8 based on the results of the congestion determination processing, and for transmitting image data to the multiple traffic line extraction servers 8-1, 8-2, and 8-3.
[0022] The control unit 70 may be a dedicated electronic circuit designed to realize a predetermined function or a hardware circuit such as a reconfigurable electronic circuit. The control unit 70 may be configured with a semiconductor integrated circuit such as a microcomputer, a DSP, an FPGA, or an ASIC. The control unit 70 may also include an internal memory as a temporary storage area for storing various data and programs.
[0023] The storage unit 74 stores data and programs. The storage unit 74 is configured, for example, by a hard disk drive (HDD) or a semiconductor storage device (SSD). The storage unit 74 stores image data acquired from the multiple cameras 2. The storage unit 74 may include a temporary storage element configured, for example, by a RAM such as a DRAM or an SRAM, and may function as a working area for the control unit 70. For example, the storage unit 74 may temporarily store the determination result of the congestion determination unit 71 and the determination result of the distribution destination by the image distribution unit 72.
[0024] The communication interface 75 is a connection circuit that connects devices and communicates data in accordance with standards such as USB, IEEE 802.11, or Bluetooth (registered trademark). The communication interface 75 connects, for example, the multiple cameras 2 and the multiple flow line extraction servers 8 to the congestion management server 7. The communication interface 75 may include a circuit for connecting to a communication network, and may communicate data with, for example, the multiple cameras 2 and the multiple flow line extraction servers 8 via the communication network.
[0025] 1-4. Configuration of the Traffic Line Extraction Server 3 is a block diagram illustrating a configuration of the traffic line extraction server 8 of this embodiment. The traffic line extraction server 8 is an example of an image recognition device in this embodiment. The traffic line extraction server 8 illustrated in FIG. 3 includes a control unit 80, a storage unit 84, and a communication interface 85.
[0026] The control unit 80 controls the operation of the traffic line extraction server 8. The control unit 80 is composed of, for example, a CPU, MPU, GPU, GPGPU, or TPU, or a combination of these, which cooperates with software to realize a predetermined function. The control unit 80 reads out data and programs stored in the storage unit 84, performs various calculation processes, and realizes various functions.
[0027] In the traffic line extraction server 8 illustrated in FIG. 3, the control unit 80 includes a traffic line extraction unit 81 as a functional configuration, for example. The traffic line extraction unit 81 performs a traffic line extraction process including an image recognition process for detecting an object such as a person 11 that is set in advance as a target for image recognition, based on image data received from the congestion management server 7. In the traffic line extraction process, position information of the person 11, etc. is calculated based on the detection result by the image recognition. The traffic line extraction unit 81 generates individual traffic line information Ln indicating a traffic line based on the position information calculated from the image data of the camera 2-1 corresponding to the traffic line extraction server 8-1, and accumulates it in the storage unit 84. For example, various types of machine learning-based trained models are applied to the image recognition process in the traffic line extraction unit 81.
[0028] The control unit 80 may be a hardware circuit such as a dedicated electronic circuit or a reconfigurable electronic circuit designed to realize a predetermined function. The control unit 80 may be configured with a semiconductor integrated circuit such as a microcomputer, a DSP, an FPGA, or an ASIC. The control unit 80 may also include an internal memory as a temporary storage area for storing various data and programs.
[0029] The storage unit 84 stores data and programs. The storage unit 84 is configured, for example, by a hard disk drive (HDD) or a semiconductor storage device (SSD). The storage unit 84 stores the individual trajectory information Ln. The storage unit 84 may include a temporary storage element configured, for example, by a RAM such as a DRAM or an SRAM, and may function as a working area for the control unit 80.
[0030] The communication interface 85 is a connection circuit that connects devices and communicates data in accordance with standards such as USB, IEEE 802.11, or Bluetooth. The communication interface 85 is communicatively connected to, for example, the congestion management server 7 and the flow line connection server 9. The communication interface 85 may be communicatively connected to other flow line extraction servers 8. The communication interface 85 may include a circuit for connecting to a communication network, and may communicate data between, for example, a plurality of flow line extraction servers 8 via the communication network.
[0031] 1-5. Configuration of flow line connection server 4 is a block diagram illustrating a configuration of the trajectory connection server 9 of this embodiment. The trajectory connection server 9 is an example of an information integration device in this embodiment. The trajectory connection server 9 illustrated in FIG. 4 includes a control unit 90, a storage unit 94, a communication interface 95, and an output interface 96.
[0032] The control unit 90 controls the operation of the flow line connection server 9. The control unit 90 is composed of, for example, a CPU or MPU that cooperates with software to realize predetermined functions. The control unit 90 reads data and programs stored in the storage unit 94, performs various calculation processes, and realizes various functions.
[0033] 4, the control unit 90 includes, as a functional configuration, a flow line combination unit 91. The flow line combination unit 91 combines the individual flow line information L1 and the like received from each of the flow line extraction servers 8-1, 8-2, and 8-3 to generate overall flow line information L0 based on image data of the multiple cameras 2, and stores the overall flow line information L0 in a storage unit 94.
[0034] The control unit 90 may be a dedicated electronic circuit designed to realize a predetermined function or a hardware circuit such as a reconfigurable electronic circuit. The control unit 90 may be configured with a semiconductor integrated circuit such as a microcomputer, a DSP, an FPGA, or an ASIC. The control unit 90 may also include an internal memory as a temporary storage area for storing various data and programs.
[0035] The storage unit 94 stores data and programs. The storage unit 94 is configured, for example, by a hard disk drive (HDD) or a semiconductor storage device (SSD). The storage unit 94 stores the overall flow line information L0. The storage unit 94 may include a temporary storage element configured, for example, by a RAM such as a DRAM or an SRAM, and may function as a working area for the control unit 90.
[0036] The communication interface 95 is a connection circuit that connects devices and communicates data in accordance with standards such as USB, IEEE 802.11, or Bluetooth. The communication interface 95 communicatively connects, for example, a plurality of trajectory extraction servers 8 and a trajectory connection server 9. The communication interface 95 may include a circuit for connecting to a communication network, and may communicate data with, for example, a plurality of trajectory extraction servers 8 via the communication network.
[0037] The output interface 96 is an example of an output unit that outputs information. The output interface 96 is an output interface circuit for video signals conforming to, for example, the HDMI (registered trademark) standard. The output interface 96 outputs signals to an external display device such as a monitor, a projector, or a head-mounted display in order to display various information. The output unit of the information integration device is not limited to the output interface 96, and may be configured as a display device such as a liquid crystal display or an organic EL display.
[0038] 2. Operation 2-1. Overview of operation An overview of the operation of the traffic line detection system 1 according to this embodiment will be described with reference to Fig. 1 and Fig. 5. Fig. 5 is a diagram for explaining an example of the operation of the traffic line detection system 1 according to this embodiment.
[0039] In the traffic line detection system 1, each of the cameras 2-1, 2-2, and 2-3 captures an image of its respective imaging range in the environment 6 and generates image data, for example, at a frame rate of a video. Each of the cameras 2-1, 2-2, and 2-3 sequentially transmits each of the image data to the congestion management server 7 in association with, for example, the imaging times that are synchronized with each other.
[0040] The congestion management server 7 in this embodiment, for example, acquires image data of images captured by the cameras 2-1, 2-2, and 2-3, and performs processing as the congestion determination unit 71 and the image distribution unit 72 (see S1 and S2 in FIG. 5). For example, when a particular congestion state, which will be described later, does not occur, the congestion management server 7 transmits the image data from the cameras 2-1, 2-2, and 2-3 to the corresponding traffic line extraction servers 8-1, 8-2, and 8-3, respectively.
[0041] Based on the image data received, each of the flow line extraction servers 8-1, 8-2, and 8-3 performs image recognition processing in parallel on the images captured by the multiple cameras 2-1, 2-2, and 2-3 to detect the flow lines of the person 11, etc. (see S3 in FIG. 5). Each of the flow line extraction servers 8-1, 8-2, and 8-3 generates individual flow line information indicating each detection result and transmits it to the flow line connection server 9.
[0042] The flow line integration server 9 integrates information such as flow lines detected by a plurality of flow line extraction servers 8 so that the flow lines are displayed on a map image showing a map of the environment 6 on the display terminal 4 (see S4 in FIG. 5).
[0043] 2-1-1. Behavior during peak hours The operation of the above-described flow line detection system 1 during crowded times will be described with reference to FIGS.
[0044] Fig. 5 illustrates an example of the operation of the system 1 in a case where only the imaging range of a specific camera 2-3 in an environment 6 is crowded with people 11. Figs. 6(A), (B), and (C) illustrate images I21, I22, and I23 captured by the cameras 2-1, 2-2, and 2-3, respectively, in the example of operation in Fig. 5. The state of the captured image I23 when there is a crowd is referred to as a "crowded state," and the states of the captured images I21 and I22 when there is no crowd are referred to as a "non-crowded state."
[0045] According to the captured image I23 of a crowded state, many people are localized within the range of the image I23, so it is expected that the processing load of the trajectory extraction server 8-3 that performs image recognition will become excessive. In this case, a processing delay of a specific trajectory extraction server 8-3 among the multiple trajectory extraction servers 8-1 to 8-3 will delay the processing (S4) of the trajectory connection server 9, which will make it difficult to display trajectories in real time.
[0046] Therefore, in the traffic line detection system 1 of this embodiment, the congestion management server 7 manages the congestion state in the captured images I21 to I23.
[0047] First, the congestion management server 7 performs congestion determination processing (S1) for each of the cameras 2-1, 2-2, and 2-3 by the congestion determination unit 71. In the example of Fig. 5, only the captured image I23 of the specific camera 2-3 is determined to be in a congested state, and the captured images I21 and I22 of the remaining cameras 2-1 and 2-2 are determined to be not in a congested state.
[0048] Next, the congestion management server 7 performs image distribution processing (S2) of the image data 21, 22, 23 by the image distribution unit 72 according to the result of the congestion determination processing (S1). The image distribution unit 72 transmits the image data 21, 22 of the cameras 2-1, 2-2 determined to be in a non-congested state to the flow line extraction servers 8-1, 8-2 corresponding to the cameras 2-1, 2-2, respectively. On the other hand, the image distribution unit 72 distributes the image data 23 of the camera 2-3 determined to be in a congested state to the multiple flow line extraction servers 8 so as to divide the captured image I23. An example of such distribution is shown in Figures 6 (C) to (F).
[0049] 6(D), (E), and (F) show examples of divided images I23b, I23c, and I23ad obtained by dividing the captured image I23 in Fig. 6(C). In the example of Fig. 5, the congestion management server 7 distributes and supplies image data 23ad, 23b, and 23c representing the divided images I23ad, I23b, and I23c not only to a specific traffic line extraction server 8-3 but also to other traffic line extraction servers 8-1 and 8-2.
[0050] Each of the flow line extraction servers 8-1, 8-2, and 8-3 performs an image recognition process on the image data provided from the congestion management server 7 by using the flow line extraction unit 81 to perform a flow line extraction process (S3) for extracting a flow line based on the image data of the corresponding cameras 2-1, 2-2, and 2-3. At this time, the processing load of the specific flow line extraction server 8-3 is reduced because the processing target of the image recognition is reduced from the captured image I23 in the crowded state to the divided image I23ad as shown in Figs. 6(C) and (F). On the other hand, the remaining divided images I23b and I23c, together with the captured images I21 and I22 in the non-crowded state, become the processing target of the other flow line extraction servers 8-1 and 8-2.
[0051] Therefore, the processing load is distributed among the multiple flow line extraction servers 8-1 to 8-3, and the flow line extraction process (S3) can be performed efficiently. In this embodiment, the multiple flow line extraction servers 8-1 to 8-3 transmit and receive information on the detection results by image recognition of the divided images I23ad, I23b, and I23c to each other, and generate individual flow line information L1, L2, and L3 based on the image data of the corresponding cameras 2-1 to 2-3. Each of the flow line extraction servers 8-1, 8-2, and 8-3 transmits the individual flow line information L1, L2, and L3 to the flow line connection server 9.
[0052] The flow line combination server 9 performs a flow line combination process (S4) to generate overall flow line information L0 by integrating the received individual flow line information L1, L2, and L3. The flow line combination server 9 transmits the overall flow line information L0 to the display terminal 4 together with map image data showing a map image of the environment 6 stored in advance in, for example, the storage unit 94. As a result, the display terminal 4 displays the flow lines indicated by the overall flow line information L0 by superimposing them on the map image (S5).
[0053] The flow line detection system 1 repeats the above processes (S1 to S5) periodically, for example. As a result, the display of the flow line is updated in real time, for example, on the display terminal 4.
[0054] According to the above operation, even if a captured image I23ad of a crowded state occurs, the processing load can be distributed among the multiple flow line extraction servers 8-1 to 8-3, and the flow line extraction process (S3) can be performed efficiently. Therefore, delays in the subsequent flow line combination process (S4) can be suppressed, and it is possible to easily display the flow lines in real time on the display terminal 4, for example.
[0055] According to such a traffic line detection system 1, image data from multiple cameras 2 are distributed (S2) to multiple traffic line extraction servers 8 in accordance with the result of the congestion determination process (S1). This makes it possible to prevent a delay in processing due to a concentration of the load of image recognition processing in the traffic line extraction process (S3) in the traffic line extraction server 8-3 corresponding to the camera 2-3 determined to be congested, for example. The result of the congestion determination process (S1) also changes in accordance with the movement of people 11 and the like in the environment 6, and the division and distribution of images in the image distribution process (S2) are updated, making it possible to perform the traffic line combination process (S4) and the display of traffic lines (S5) in real time.
[0056] 2-2. Congestion determination process The congestion determination process (S1 in FIG. 5) in the congestion management server 7 will be described with reference to FIG. 7 and FIG. 8. The congestion determination process (S1) uses congestion information from each camera 2. The congestion information is various information indicating the degree of congestion caused by moving objects corresponding to the number of people 11 in each captured image of the multiple cameras 2.
[0057] Fig. 7 is a flowchart illustrating the congestion determination process (S1) in the congestion management server 7. Fig. 8 is a diagram for explaining the congestion determination process. Fig. 9 is a diagram illustrating a congestion information table in the congestion determination process.
[0058] Each process shown in the flowchart in Fig. 7 is executed, for example, at a frame period of the camera 2 by the control unit 70 of the congestion management server 7 functioning as the congestion determination unit 71. The flowchart in Fig. 7 is started, for example, when the congestion management server 7 receives image data from the multiple cameras 2 via the communication interface 75. The image data is held in the storage unit 74 for a predetermined number of frames, for example, two frames for each camera 2.
[0059] In the flowchart of Fig. 7, first, the congestion determination unit 71 acquires image data of each camera 2, for example, from the storage unit 74 (S10). Figs. 8(A) and (B) show examples of image data acquired in step S10. Fig. 8(A) shows an example of image data of a current frame F2 captured by one camera 2. Fig. 8(B) shows an example of image data of a previous frame F1, which is one frame earlier, captured by the same camera 2 as in Fig. 8(A). In step S10, the congestion determination unit 71 acquires current and past image data such as those shown in Figs. 8(A) and (B) for each camera 2.
[0060] The congestion determination unit 71 generates motion difference data H for each of the multiple cameras 2-1 to 2-3 based on the acquired image data (S11). Fig. 8(C) illustrates motion difference data H generated based on the image data of Figs. 8(A) and (B). The motion difference data H is data indicating the difference due to motion from the past to the present as, for example, a heat map in the entire area of the captured image. The congestion determination unit 71 generates the motion difference data H by comparing the image data of the past frame F1 and the image data of the current frame F2 for each pixel, for example by calculating the difference between frames at each pixel.
[0061] Next, the congestion determination unit 71 determines the congestion state in the captured image of each camera 2 based on the motion difference data H1, H2, and H3 of each of the multiple cameras 2-1 to 2-3 (S12). The congestion determination unit 71 calculates, for example, the total value of all pixels of the motion difference data H as the congestion degree of the current frame F2 per camera 2, and determines whether the congestion state is congested or not depending on whether the congestion degree is equal to or greater than a predetermined threshold. The predetermined threshold is set, for example, in consideration of the number of people within the captured image that will cause an excessive processing load, based on the correlation between the number of people and the motion difference. The congestion determination unit 71 performs the above processing on the motion difference data H1, H2, and H3 of each camera 2.
[0062] Next, the congestion determination unit 71 stores a congestion information table T1 for storing congestion information of each camera 2, such as the above-mentioned determination results, in, for example, the storage unit 74 of the congestion management server 7 (S13). An example of the congestion information table T1 is shown in FIG.
[0063] 9, the congestion information table T1 stores the above-mentioned "motion difference data", the above-mentioned "congestion level", and a "congestion flag" indicating the determination result of the congestion state as examples of congestion information, in association with a "camera ID" that identifies each of the multiple cameras 2-1 to 2-3. The congestion determination unit 71 sets each congestion flag to "on" or "off" depending on the determination result of the congestion state or non-congestion state for each camera 2.
[0064] In the example of Fig. 9, congestion information such as the motion difference data H1, H2, H3 of each camera 2-1, 2-2, 2-3 and the congestion degrees X1, X2, X3 are stored in the congestion information table T1. Furthermore, in the congestion information table T1 of Fig. 9, for example, for the congestion information of the camera ID "3", the congestion flag is "ON", that is, the judgment result of "crowded state" is obtained based on the motion difference data H3 of the camera 2-3.
[0065] Returning to FIG. 7, once the congestion determination unit 71 stores the congestion information table T1 (S13), the process according to the flowchart in FIG. 7 ends.
[0066] As described above, in the congestion management server 7 of this embodiment, the congestion determination unit 71 acquires image data for each camera 2 in the system 1 (S10) and generates various types of current congestion information (S11, S12). This allows the congestion management server 7 to manage various types of congestion information in the image data for each camera 2. Furthermore, according to the motion difference data H (S11), it is possible to calculate the degree of congestion according to the number of people within the range of the captured image of each camera 2, without using image recognition in particular.
[0067] 2-3. Image distribution processing The image distribution process (S2 in FIG. 5) in the congestion management server 7 will be described with reference to FIG. 10 and FIG. 11. In the image distribution process (S2), a distribution destination to which the image data of each camera 2 is distributed is determined in the multiple flow line extraction servers 8-1 to 8-3 based on the congestion information for each camera 2.
[0068] Fig. 10 is a flowchart illustrating the image distribution process (S2) in the congestion management server 7. Fig. 11 is a diagram for explaining the image distribution process in the congestion management server 7.
[0069] Each process shown in the flowchart of Fig. 10 is executed by the control unit 70 of the congestion management server 7 functioning as the image distribution unit 72. The flowchart of Fig. 10 starts in a state in which the congestion information table T1 has been stored by the operation of the congestion determination unit 71 (S13 in Fig. 7), for example.
[0070] In the flowchart of FIG. 10, first, the image distributor 72 of the congestion management server 7 acquires current image data from the multiple cameras 2 and various congestion information from the congestion information table T1, for example, from the storage unit 74 (S20).
[0071] Next, the image distribution unit 72 selects one camera i from the multiple cameras 2 (S21). In the example of Fig. 5, the cameras 2-1, 2-2, and 2-3 are selected as the camera i in turn at each step S21.
[0072] The image distribution unit 72 judges whether the congestion flag of the selected camera i is "ON" (S22), for example, by referring to the congestion information table T1 in Fig. 9. The image distribution unit 72 makes this judgment by referring to the congestion information corresponding to the camera ID of the camera i. For example, in the example of Fig. 9, the congestion flag referred to when selecting camera 2-1 is "OFF".
[0073] If the congestion flag of the selected camera i is “OFF” (NO in S22), the corresponding captured image is determined to be in a non-crowded state. In this case, the image distributor 72 does not particularly execute the processes of steps S23 and S24, and proceeds to step S25.
[0074] The image distribution unit 72 judges whether or not all of the cameras 2-1, 2-2, and 2-3 have been selected in step S21 (S25). The image distribution unit 72 repeats the processes from step S21 onwards until all of the cameras 2-1, 2-2, and 2-3 have been selected (NO in S25).
[0075] In the example of Fig. 5, when cameras 2-1 and 2-2 are selected, the congestion flag is "off" in the congestion information table T1 (NO in S22), so the process proceeds to step S25. Then, since not all cameras have been selected yet (NO in S25), camera 2-3 is selected (S21). At this time, the congestion flag referenced in the congestion information table T1 of Fig. 9 when selecting camera 2-3 is "on."
[0076] If the congestion flag of the selected camera i is “ON” (YES in S22), the corresponding captured image is determined to be in a crowded state. In this case, the image distributor 72 performs processing to divide the captured image indicated by the image data of the camera i (S23).
[0077] An example of the process of step S23 is shown in Fig. 11. The image distribution unit 72 sets regions R1 to R4 that divide the captured image I23 indicated by the image data into a predetermined number of divisions (for example, 4 or 9). Figs. 11(A) and (B) show an example of the number of divisions "4". In the example of Fig. 11(B), divided images I23a to I23d corresponding to the regions R1 to R4 are set to overlap each other in adjacent portions.
[0078] In step S23, the image distribution unit 72 refers to the congestion information table T1 in Fig. 9, for example, to calculate the congestion degree of each of the divided images I23a-I23d in the same manner as in step S12 in Fig. 7. For example, when the image distribution unit 72 calculates a congestion degree less than a predetermined threshold, the image distribution unit 72 adopts the division number. On the other hand, when the image distribution unit 72 calculates a congestion degree equal to or greater than the predetermined threshold, the image distribution unit 72 performs the above calculations again using a larger division number. The predetermined threshold is a value small enough that the traffic line extraction process can be executed in real time in each of the multiple traffic line extraction servers 8, and is set, for example, in the same manner as the threshold in step S12.
[0079] Image data representing the divided images I23a to I23d are associated with the respective congestion degrees and are stored, for example, in an internal memory of the control unit 70 of the congestion management server 7. Based on such congestion degrees, the image distribution unit 72 may combine two or more divided images I23a, I23d within a range less than the above threshold to generate a divided image I23ad (see FIG. 6(F)).
[0080] In the example of Figure 5, when camera 2-3 is selected as camera i in step S21, the congestion flag for each congestion information in the congestion information table T1 is "on" (YES in S22), so image I23 of camera 2-3 is divided as shown in Figures 11(A) and (B).
[0081] The image distribution unit 72 determines the flow line extraction server 8 to which the segmented image of the selected camera i is to be distributed based on the congestion information of the multiple cameras 2 (S24). The flow line extraction server 8 corresponding to each camera 2 is set in advance as a supply destination of the image data of the camera 2. For example, the image distribution unit 72 refers to the congestion information table T1 and determines an additional distribution destination from the flow line extraction server 8 corresponding to the camera 2 whose congestion flag is "off". Then, the image distribution unit 72 determines the distribution of the segmented image to the flow line extraction server 8 that is set in advance as a supply destination corresponding to the camera 2 whose congestion flag is "on". In the example of FIG. 5, the distribution destinations of the segmented images I23b and I23c are determined to be the flow line extraction servers 8-1 and 8-2, respectively. Then, it is determined that the distribution of the segmented image I23ad obtained by integrating the segmented images I23a and I23d to the flow line extraction server 8-3 is to be performed.
[0082] In step S24, the image distribution unit 72 may determine the distribution destination based on the congestion degree of each camera 2 in the congestion information table T1 and the congestion degree calculated for each divided image, for example, so that the congestion degrees of each traffic line extraction server 8 are within a predetermined range in which they can be considered to be approximately the same. Furthermore, the congestion degree for each camera 2 in the congestion information table T1 may be updated in response to the determination of the distribution destination, and the subsequent processing may be performed based on the updated congestion information table T1.
[0083] When the control unit 70 as the image distribution unit 72 determines that, for example, all of the cameras 2-1 to 2-3 have been selected (YES in S25), it controls the communication interface 75 to transmit each image data to a plurality of traffic line extraction servers 8 (S26). The image data of the captured image or divided image of each camera 2 is transmitted to the traffic line extraction server 8 corresponding to each camera 2. Furthermore, the image data of the divided image is transmitted to the traffic line extraction server 8 that has been additionally determined as the distribution destination. At this time, each image data is transmitted in association with, for example, the camera ID of each camera 2.
[0084] When the image distributor 72 transmits the image data to the plurality of flow line extraction servers 8 (S26), the process shown in this flowchart ends.
[0085] As described above, the image distribution unit 72 divides the images captured by the cameras 2 that are determined to be in a "crowded state" based on the congestion information of the multiple cameras 2 (S22, S23) and distributes the divided images to the multiple flow line extraction servers 8 (S24). This makes it possible to distribute the processing load on each of the multiple flow line extraction servers 8.
[0086] In step S23, the congestion degree threshold is not limited to the same value as in step S12 in Fig. 7, and may be set to a different value as appropriate. For example, the image distribution unit 72 may calculate the average value of the congestion degrees of multiple cameras 2 regarded as being in the vicinity of the selected camera i as the threshold based on the congestion information table T1. In this case, a combination of cameras regarded as being in the vicinity of each camera is set in advance based on the positional relationship between the multiple cameras 2, for example.
[0087] In step S23, the image may be divided into regions such that each divided image partially overlaps with an adjacent divided image, as shown in Fig. 11(B) for example. By dividing the image by overlapping them in this manner, it is possible to reduce oversights when detecting a moving object such as a person 11 at the boundary between divided images in a flow line extraction process to be described later by multiple flow line extraction servers 8.
[0088] 2-4. Flow line extraction processing The operation of the flow line extraction server 8 will be described with reference to Fig. 12 and Fig. 13. Each of the flow line extraction servers 8-1 to 8-3 performs a flow line extraction process (S3 in Fig. 5) such as generating individual flow line information based on image data of the camera 2 corresponding to the flow line extraction server 8 among the multiple cameras 2-1 to 2-3.
[0089] Fig. 12 is a flowchart illustrating the flow line extraction process (S3) in the flow line extraction server 8. Fig. 13 is a diagram for explaining the flow line extraction process. Each process shown in the flowchart in Fig. 12 is executed by a control unit 80 functioning as a flow line extraction unit 81 in each flow line extraction server 8. The flowchart in Fig. 12 is started, for example, at a frame period of a video.
[0090] In the flowchart of Fig. 12, first, the flow line extraction unit 81 receives image data from the congestion management server 7 via the communication interface 85 (S30). For example, the flow line extraction server 8-1 in the example of Fig. 5 receives image data 21 of the image I21 captured by the corresponding camera 2-1 (i.e., the camera in charge) and image data 23b of the divided image I23b.
[0091] The flow line extraction unit 81 performs image recognition processing for object detection in the received image data to detect an image position indicating the position on the image of the person 11, etc., which is the recognition target (S31). An example of the processing of step S31 is shown in Figures 13(A) and (B).
[0092] Fig. 13(A) illustrates an example of an image position P21 detected by a corresponding flow line extraction server 8-1 in a captured image I21 in a non-crowded state. Fig. 13(B) illustrates an example of an image position P23b detected in a divided image I23b by the same flow line extraction server 8-1 as in Fig. 13(A).
[0093] In the image recognition process for object detection in step S31, for example, a rectangular detection area 40 is recognized as a detection result of an object to be recognized in a captured image I21, as shown in Fig. 13(A). The movement line extraction unit 81 calculates, for example, the coordinates on the image of the center point of the detection area 40 as an image position P21. The image position P21 is held in, for example, an internal memory of the control unit 80.
[0094] In the examples of Figures 13(A) and (B), the traffic flow extraction unit 81 of the traffic flow extraction server 8-1 detects image position P21 in the image data 21 of the responsible camera 2-1, as shown in Figure 13(A), and detects image position P23b in the image data 23b of the non-responsible camera 2-3 from the split image I23b.
[0095] Next, the flow line extraction unit 81 determines whether or not the image data whose image position has been detected in step S31 includes image data from a camera 2 that is not assigned to the device itself (S32). The flow line extraction unit 81 determines whether or not the image data is from a camera 2 that is not assigned to the device itself, for example, by referring to a camera ID associated with each image data.
[0096] When the flow line extraction unit 81 determines that there is image data from a camera 2 that is not its responsibility (YES in S32), it transmits information indicating the image position detected from the image data to the flow line extraction server 8 corresponding to that camera 2 among the plurality of flow line extraction servers 8-1 to 8-3 (S33). In this embodiment, the plurality of flow line extraction servers 8 transmit and receive image position information to each other via their respective communication interfaces 85.
[0097] For example, since the image data 23b of the split image I23b (Figure 13(B)) is image data from a camera 2-3 that is not in charge of the flow line extraction server 8-1 (NO in S32), the flow line extraction unit 81 of the flow line extraction server 8-1 transmits information on the image position P32 in the split image I23b to the flow line extraction server 8-3 corresponding to the camera 2-3 (S33).
[0098] On the other hand, if the flow line extraction unit 81 determines that there is no image data from a camera 2 that is not in charge of the flow line extraction unit 81 (NO in S32), the flow line extraction unit 81 does not particularly execute step S33, and proceeds to step S34.
[0099] The flow line extraction unit 81 judges whether its own communication interface 85 has received image position information from another of the plurality of flow line extraction servers 8-1 to 8-3 within a predetermined period of time, for example (S34).
[0100] When image position information about the responsible camera 2 is received from another flow line extraction server 8 (YES in S34), the flow line extraction unit 81 updates the detection result of the image position of the responsible camera 2 based on the received image position information (S35). Fig. 13(C) illustrates an image position P23 of the detection result in a captured image I23 of a crowded state. For example, the flow line extraction unit 81 of the flow line extraction server 8-3 receives information about the image position P23b of the corresponding camera 2-3 from the flow line extraction server 8-1 (YES in S32) and adds it to the image position P23 of the detection result (S35).
[0101] On the other hand, if the information on the image position has not been received from another flow line extraction server 8 (NO in S34), the flow line extraction unit 81 does not particularly execute step S35, and proceeds to step S36.
[0102] Next, the flow line extraction unit 81 performs a calculation to convert the image position of the detection result into coordinates, thereby calculating a map position indicating the position of the recognition target, such as the person 11 on the map of the environment 6 (S36). In each of the flow line extraction servers 8, the flow line extraction unit 81 performs the coordinate conversion based on parameters that prescribe the conversion from the image position of the camera 2 in charge to a map position, and that are stored in advance in the memory unit 84 of the server itself.
[0103] The flow line extraction unit 81 acquires past flow line information of a predetermined range, such as from the present when the map position is calculated to several frames ago (S37). The flow line extraction unit 81 acquires past flow line information such as map position and speed from individual flow line information stored in the memory unit 84 of the own device, for example.
[0104] Next, the flow line extraction unit 81 generates current individual flow line information based on the calculated current map position and the acquired past flow line information (S38). For example, the flow line extraction unit 81 calculates a map position in the current frame, i.e., a predicted position, predicted from the map position and speed in the past flow line information of each flow line, and generates individual flow line information by linking the current map position to a flow line whose predicted position and calculated map position are closer than a predetermined value. The predetermined distance value is, for example, a small value that can be considered as the same point on the map. Furthermore, since a map position that cannot be linked is considered to indicate the position of a person who came from outside the imaging range, the position is included in the individual flow line information. The generated individual flow line information is stored in, for example, the storage unit 84.
[0105] The flow line extraction unit 81 transmits the generated individual flow line information to the flow line connection server 9 via, for example, the communication interface 85 (S39), and ends the process shown in this flowchart. Through the above process in each of the flow line extraction servers 8-1, 8-2, and 8-3, the individual flow line information L1, L2, and L3 are transmitted from the flow line extraction servers 8-1, 8-2, and 8-3 to the flow line connection server 9, respectively.
[0106] As described above, in each of the multiple flow line extraction servers 8, the flow line extraction unit 81 acquires image data from the image distribution unit 72 of the congestion management server 7 (S30) and generates individual flow line information based on the image data of the corresponding camera (S38). This distributes the processing load of image recognition among the multiple flow line extraction servers 8, making it possible to prevent the processing time in some of the flow line extraction servers from becoming longer.
[0107] Furthermore, the flow line extraction unit 81 detects the image position of the moving object to be recognized from the image data by image recognition processing (S31), and transmits and receives the image positions between the multiple flow line extraction servers 8 (S33, S34), thereby utilizing the image positions captured by the camera 2 in charge of each flow line extraction server 8. This allows each flow line extraction server 8 to obtain individual flow line information based on the image data of the camera 2 in charge.
[0108] 2-5. Flow line connection processing The operation of the trajectory connection server 9 will be described with reference to Fig. 14. The trajectory connection server 9 performs a trajectory connection process (S4 in Fig. 5) that combines the individual trajectory information L1, L2, and L3 generated in each of the multiple trajectory extraction servers 8-1 to 8-3 into overall trajectory information L0 that indicates all trajectories within the imaging ranges of the multiple cameras 2-1 to 2-3.
[0109] Fig. 14 is a flowchart illustrating the flow line combination process (S4) by the flow line combination server 9. Each process shown in the flowchart in Fig. 14 is executed by the control unit 90 functioning as the flow line combination unit 91 in the flow line combination server 9. The flowchart in Fig. 14 is started, for example, at a frame cycle of a video.
[0110] First, the flow line combination unit 91 receives the individual flow line information L1, L2, and L3 from each of the plurality of flow line extraction servers 8-1 to 8-3 via the communication interface 95 (S40).
[0111] Next, the flow line combination unit 91 generates the overall flow line information L0 based on the received multiple individual flow line information L1 to L3 (S41). For example, the flow line combination unit 91 compares the map positions of the flow lines in the separate individual flow line information L1 to L3 with each other, and combines the flow lines that are considered to be the flow lines of the same person. The flow lines are combined, for example, in the same manner as in step S38, taking into account the distance range that can be considered to be the same on the map.
[0112] The flow line combination unit 91 outputs, for example, the overall flow line information L0 and map data showing a map of the environment 6 to the display terminal 4 via the output interface 96 (S42). The map data is, for example, stored in advance in the storage unit 84. As a result, the flow line combination unit 91 causes the display terminal 4 to display the overall flow line information superimposed on the map data.
[0113] The flow line combination unit 91 ends the process shown in this flowchart by outputting the overall flow line information L0 (S42).
[0114] As described above, the flow line combination unit 91 combines the individual flow line information L1-L3 acquired from the multiple flow line extraction servers 8 to generate overall flow line information L0 (S41, S42), and outputs it together with, for example, map data (S42). This makes it possible to combine and display the flow lines based on the multiple cameras 2, making it easier for the user 3 to analyze the flow lines across the imaging ranges of each of the multiple cameras 2.
[0115] 3. Effects etc. As described above, the traffic line detection system 1 according to this embodiment is an example of a moving object detection system that detects multiple moving objects such as a person 11 based on captured images. The traffic line detection system 1 includes multiple cameras 2 (multiple imaging devices) that capture images at a predetermined period and generate image data, multiple traffic line extraction servers 8 (multiple image recognition devices) that perform image recognition on the image data from each camera and detect moving objects in the captured images represented by each image data, and a congestion management server 7 (information management device) that manages the image data supplied from each camera to each traffic line extraction server. The congestion management server 7 includes a communication interface 75 (communication unit) that performs data communication with each camera 2-1, 2-2, 2-3 and traffic line extraction servers 8-1, 8-2, 8-3, and a control unit 70 that controls the operation of the communication interface 75. The control unit 70 manages congestion information indicating the number of moving objects localized within the range of the captured image shown by each image data of each camera 2-1, 2-2, 2-3, for example as a congestion information table T1 (S1), and controls the communication interface 75 to distribute image data from the multiple cameras 2 and supply it to each traffic flow extraction server 8-1, 8-2, 8-3 according to the magnitude of the quantity such as the congestion degree indicated by the congestion information (S2).
[0116] According to the above-described traffic line detection system 1, image data captured by multiple cameras 2 is distributed according to the quantity indicated by the congestion information and supplied to multiple traffic line extraction servers 8. This allows the multiple traffic line extraction servers 8 to efficiently perform the process of detecting moving objects.
[0117] In this embodiment, the control unit 70 functions as an image distribution unit 72, and sets multiple traffic line extraction servers as distribution destinations for image data captured by multiple cameras 2 in which congestion information indicates a predetermined amount or more (S2). This prevents the processing load of performing image recognition to detect moving objects on some of the multiple traffic line extraction servers 8 from becoming high, and allows the processing on the multiple traffic line extraction servers 8 to be performed efficiently.
[0118] In this embodiment, the control unit 70 functions as an image distribution unit 72 and controls the communication interface 75 to separately supply a plurality of divided data, which divides a captured image represented by one image data into a plurality of regions, to a plurality of destination trajectory extraction servers (S2). This makes it easier to distribute the processing load in each of the trajectory extraction servers 8-1, 8-2, and 8-3 to the same extent.
[0119] In this embodiment, each of the trajectory extraction servers 8-1, 8-2, and 8-3 repeatedly detects a moving object and generates individual trajectory information L1, L2, and L3 as an example of trajectory information indicating the trajectory of the moving object (S3). Each of the trajectory extraction servers 8-1, 8-2, and 8-3 may, for example, only perform image recognition and transmit the detection result of the moving object via the communication interface 85. In this case, the generation of the trajectory information may be performed, for example, by the trajectory connection server 9 that receives the detection result or an information processing device outside the trajectory detection system 1.
[0120] The movement line detection system 1 of this embodiment further includes a movement line combination server 9 (information integration device) that generates (S4) overall movement line information L0 as an example of information showing the movement line of a moving object in the entire range captured by the multiple cameras 2 based on the individual movement line information L1, L2, and L3 generated by each of the movement line extraction servers 8-1, 8-2, and 8-3. This allows, for example, a user 3 to analyze the movement line in the entire range captured by the multiple cameras 2.
[0121] In this embodiment, the control unit 70 calculates a motion difference as an example of the amount of motion of a moving object in a captured image based on multiple frames of image data captured sequentially by the cameras 2-1, 2-2, and 2-3, and generates congestion information based on the calculated motion difference (S1). By using the motion difference that can be calculated with a small amount of calculation, congestion information can be generated including information based on the latest frame while suppressing the processing load of the congestion management server 7.
[0122] The congestion management server 7 of this embodiment is an example of an information management device in a traffic line detection system 1 including a plurality of cameras 2 and a plurality of traffic line extraction servers 8 that recognize a plurality of moving objects in captured images represented by the image data of each camera 2-1, 2-2, 2-3 based on the image data from each camera. The congestion management server 7 includes a communication interface 75 that performs data communication with each camera 2-1, 2-2, 2-3 and each traffic line extraction server 8-1, 8-2, 8-3, and a control unit 70 that controls the operation of the communication interface 75 so as to manage the image data supplied from each camera 2-1, 2-2, 2-3 to each traffic line extraction server 8-1, 8-2, 8-3. The control unit 70 manages congestion information in which moving objects are localized within the range of the captured images represented by each image data for each camera 2-1, 2-2, 2-3 (S1), and controls the communication interface 75 to distribute the image data from the plurality of cameras 2 and supply them to each traffic line extraction server 8-1, 8-2, 8-3 according to the magnitude of the quantity indicated by the congestion information (S2).
[0123] According to the above-described congestion management server 7, in the flow line detection system 1, the multiple flow line extraction servers 8 can efficiently perform the process of recognizing moving objects.
[0124] (Embodiment 2) A second embodiment of the present disclosure will be described below with reference to Fig. 15. In the second embodiment, the operation of the congestion management server 7 that manages distribution of congestion information and image data by utilizing the positional relationship between multiple cameras 2 will be described.
[0125] Hereinafter, the description of the configuration and operation similar to those of the flow line detection system 1 and the congestion management server 7 according to the first embodiment will be omitted as appropriate, and the flow line detection system 1 and the congestion management server 7 according to this embodiment will be described.
[0126] Fig. 15 is a diagram for explaining the propagation of a crowded state between multiple cameras in this embodiment. Fig. 15(A) illustrates an example of neighbor information of multiple cameras 2. The neighbor information is an example of arrangement information that indicates the positional relationship between the cameras when the multiple cameras 2 are arranged in an environment 6.
[0127] In Fig. 15(A), the relationship between adjacent cameras among the multiple cameras 2 is illustrated by solid lines. For example, in the illustration of Fig. 15(A), camera 2-1 is adjacent to cameras 2-2 and 2-3. The adjacent information is stored in advance, for example, in the storage unit 74 of the congestion management server 7. Fig. 15(B) is a diagram illustrating the congestion determination process of this embodiment based on the adjacent information of Fig. 15(A).
[0128] In this embodiment, the congestion determination unit 71 of the congestion management server 7 first generates congestion information for one of the multiple cameras 2 from the image data of that camera, for example, in the same manner as in embodiment 1. Note that the congestion information in this embodiment does not particularly include motion difference data, and may simply be information that associates a camera ID with a congestion flag. Fig. 15(B) shows an example in which congestion information for camera 2-1 is generated and the congestion flag in the congestion information is "on".
[0129] Next, the congestion determination unit 71 updates the congestion information for cameras adjacent to a camera whose congestion flag in the congestion information is "on" based on the adjacent information of multiple cameras 2. It is considered that a camera adjacent to a camera whose congestion flag is "on" is likely to become congested in the future. Therefore, the congestion determination unit 71 of this embodiment generates congestion information indicating that the adjacent camera is "scheduled to be congested" by using a flag other than the congestion flag. In the example of FIG. 15(B), cameras 2-2 and 2-3 adjacent to camera 2-1 are set to "scheduled to be congested".
[0130] The congestion determination unit 71 changes the congestion flag of a camera that is "scheduled to be congested" to "ON" after, for example, a predetermined number of frames from the current frame, and sets the camera as a processing target of the image distribution unit 72. The predetermined number of frames indicates a period during which congestion in the imaging range of an adjacent camera is expected to propagate. The number of frames may be set according to a moving object that is a recognition target for the flow line extraction process in the multiple flow line extraction servers 8.
[0131] The congestion determination unit 71 updates the congestion flag of each camera every time the congestion determination process is executed in a frame cycle by the multiple cameras 2, based on the congestion information of the multiple cameras 2. Note that for a camera with a congestion flag set to "on", if the camera is not adjacent to another camera with a congestion flag set to "on" after, for example, a predetermined number of frames from the current frame, the congestion flag may be updated to "off".
[0132] According to the above process, the congestion flag of the congestion information is updated based on the adjacent information of each camera in the multiple cameras 2, and congestion determination processing can be performed that reflects the propagation of the congestion state caused by moving objects such as people 11. As a result, in the image distribution processing, images of cameras adjacent to a camera with an "ON" congestion flag in the multiple cameras 2 can be divided in advance and distributed to multiple flow line extraction servers 8, making it easier to perform flow line extraction processing in real time.
[0133] As described above, the control unit 70 of the congestion management server 7 according to this embodiment functions as the congestion determination unit 71 and updates the congestion information based on the adjacent information as an example of information related to the arrangement of the multiple cameras 2 (see FIGS. 15(A) and (B)). This makes it possible to start dividing the image before, for example, a moving object actually moves into the imaging range of each camera.
[0134] (Embodiment 3) Hereinafter, a third embodiment of the present disclosure will be described with reference to Fig. 16. In the first embodiment, an example has been described in which the image distribution unit 72 in the congestion management server 7 operates based on the congestion flag of the determination result by the congestion determination unit 71. In the third embodiment, a congestion management server 7 will be described in which the image distribution unit 72 does not particularly use a congestion flag, but first divides images from all of the cameras in the multiple cameras 2, and distributes them according to congestion information.
[0135] The operation of the image distribution unit 72 in the congestion management server 7 of this embodiment will be described with reference to Fig. 16. Fig. 16 is a flowchart illustrating an example of image distribution processing by the image distribution unit 72 of embodiment 3. In the following, descriptions of processing similar to the image distribution processing of embodiment 1 (Fig. 10) will be omitted as appropriate.
[0136] First, the control unit 70 serving as the image distribution unit 72 in the congestion management server 7 of this embodiment acquires image data from the multiple cameras 2 (S20A).
[0137] The control unit 70 calculates the congestion degree for each captured image based on the acquired image data of each camera (S50). The control unit 70 calculates the congestion degree in the same manner as in the congestion determination process (FIG. 7) of the first embodiment, for example. The control unit 70 calculates an average congestion degree indicating the average congestion degree for all cameras as the average of the congestion degrees of each camera (S51).
[0138] The control unit 70 selects one camera i from the multiple cameras 2 (S21) and divides the image indicated by the image data of the camera i into a predetermined number of divisions (S23A). The predetermined number of divisions is set, for example, in the same manner as in step S23 of Fig. 10. The control unit 70 calculates the congestion degree for each divided image in the same manner as in step S50, associates it with the image data of each divided image, and stores it in, for example, an internal memory. The predetermined number of divisions may be set as appropriate according to the average congestion degree in step S51 so that the congestion degree in the divided image data is smaller than the average congestion degree.
[0139] Next, the control unit 70 determines distribution destinations of the segmented images from the multiple flow line extraction servers 8 based on the congestion degree and average congestion degree for the segmented images of the camera i (S24A). Specifically, the control unit 70 determines the distribution destination for each of the multiple flow line extraction servers 8 such that the total value of the congestion degrees of the distributed segmented images is approximately equal to the average congestion degree. For example, the control unit 70 performs distribution such that the total value of the congestion degrees of the segmented images is smaller than the average congestion degree.
[0140] The control unit 70 repeats steps S21 to S24A until all of the cameras 2 have been selected (NO in S25). When all of the cameras have been selected (YES in S25), the control unit 70 transmits image data showing the divided images of each of the cameras 2 from the multiple flow line extraction servers 8 to the flow line extraction server determined as the distribution destination in step S24A (S26A).
[0141] The image distribution unit 72 transmits the image data to a plurality of flow line extraction servers 8 (S26A), and ends the process shown in this flowchart.
[0142] According to the above process, the image distribution unit 72 of this embodiment calculates the average congestion degree for the multiple cameras 2 (S51), divides the images for all cameras (S23A), and distributes the images to the multiple traffic line extraction servers 8 based on congestion information such as the congestion degree and average congestion degree of the divided images (S24A). This allows the images from the multiple cameras 2 to be distributed so that the load of the traffic line extraction process on each of the multiple traffic line extraction servers 8 is nearly equal, making it easier to execute the traffic line extraction process in real time.
[0143] Furthermore, in step S50 of this embodiment, the congestion degree is calculated in image data showing the current frame obtained from the congestion determination unit 71, but this is not limited thereto, and for example, the congestion degree may be calculated in image data of a past frame, and in step S23A, the image of the current frame may be divided based on the congestion degree in the past frame.
[0144] As described above, the control unit 70 of the congestion management server 7 according to this embodiment functions as the image distribution unit 72 and calculates a congestion degree indicating congestion information for each area into which an image shown by each image data of each of the multiple cameras 2 is divided (S23A). Based on the calculated congestion degree, the control unit 70 distributes and transmits each image data of the multiple cameras 2 to multiple traffic line extraction servers 8 (S24A, S26A). Note that in this embodiment, the control unit 70 may divide and distribute an image of a current frame based on a congestion degree calculated from image data of a past frame of a video captured by the multiple cameras 2.
[0145] (Embodiment 4) Hereinafter, a fourth embodiment of the present disclosure will be described with reference to Fig. 17. In the first embodiment, an operation example in which the congestion management server 7 uses the congestion degree and congestion information based on the motion difference in image data has been described. In the fourth embodiment, an operation example in which the congestion management server 7 uses congestion information based on the detection results of a plurality of traffic line extraction servers 8 will be described.
[0146] 17 is a flowchart illustrating the congestion determination process in embodiment 4. In the following description, the same processes as those in the congestion determination process (FIG. 7) in embodiment 1 will be omitted as appropriate.
[0147] The congestion determination unit 71 of this embodiment acquires the detection results of the recognition objects in the image data of each of the multiple cameras 2 from the multiple flow line extraction servers 8, for example, via the communication interface 75 (S60). Specifically, the congestion determination unit 71 acquires the number of recognition objects detected in the image data of a past frame for each of the multiple cameras 2. The past frame is, for example, a frame transmitted to the multiple flow line extraction servers 8 a frame period before the congestion determination process is executed.
[0148] The congestion determination unit 71 generates congestion information for the multiple cameras 2 based on the number of recognition targets acquired as a detection result (S12A). For example, the congestion determination unit 71 of this embodiment generates congestion information in which the camera ID of each camera, a congestion flag that determines whether the recognition targets are congested or not depending on whether the number of recognition targets is equal to or greater than a predetermined number, and the number of recognition targets are associated with each other. The predetermined number is a sufficiently small number from the same perspective as the predetermined threshold value in step S12 of the first embodiment.
[0149] According to the above process, the congestion determination unit 71 of this embodiment generates congestion information based on the number of recognition targets in the image data of the past frame for each of the multiple cameras 2. As a result, the image distribution unit 72 performs image distribution processing on the image data of the current frame based on the congestion information.
[0150] As described above, the control unit 70 of the congestion management server 7 according to this embodiment functions as the congestion determination unit 71, acquires the number of moving objects such as people 11 detected by each of the multiple flow line extraction servers 8 (S60), and generates congestion information based on the acquired number of moving objects (S12A). In this way, by using the detection results by each flow line extraction server, the processing performed by the congestion management server 7 can be reduced, and congestion determination processing can be efficiently executed.
[0151] (Other embodiments) As described above, the first to fourth embodiments have been described as examples of the technology disclosed in the present application. However, the technology in the present disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are appropriately performed. In addition, it is also possible to combine the components described in each of the above embodiments to form a new embodiment. Therefore, other embodiments will be exemplified below.
[0152] In the above embodiment, the flow line extraction unit 81 acquires image data from the congestion management server 7 (S30). The flow line extraction unit 81 in this embodiment may acquire congestion information in addition to image data. Furthermore, in this embodiment, the flow line extraction server 8 may be configured to be able to communicate with the camera 2 via, for example, the communication interface 85. Thereby, the flow line extraction unit 81 may acquire image data other than the divided images directly from the camera 2 without going through the congestion management server 7, for example, in accordance with the acquired congestion information.
[0153] In the first embodiment, in the image distribution process (S2), the traffic line extraction servers 8 corresponding to each camera 2 are set in advance as the supply destination of the image data of the camera 2 (S24). In the image distribution process of this embodiment, the image data of a plurality of cameras 2 may be supplied to one traffic line extraction server 8. For example, the image distribution unit 72 of this embodiment may transmit both the image data 21 and 22 of the cameras 2-1 and 2-2, which are determined to be in a non-crowded state, to the traffic line extraction server 8-1. In this case, the image distribution unit 72 may distribute the image data 23 of the camera 2-3, which is determined to be in a congested state, to the traffic line extraction servers 8-2 and 8-3 so as to divide the captured image I23.
[0154] According to the above image distribution process, even if the congestion degree in the captured image I23 is particularly higher than the congestion degrees in the captured images I21 and I22, the processing load can be easily distributed among the multiple flow line extraction servers 8-1 to 8-3.
[0155] In the first embodiment, the traffic line detection system 1 is provided with traffic line extraction servers 8-1, 8-2, and 8-3 corresponding to the cameras 2-1, 2-2, and 2-3. In the traffic line detection system 1 of the present embodiment, the number of the cameras 2 and the number of the traffic line extraction servers 8 do not have to be the same, and for example, the number of the traffic line extraction servers 8 may be greater than the number of the cameras 2.
[0156] In the first embodiment, the congestion determination unit 71 generates congestion information by using a motion difference based on image data (S11, S12). The congestion determination unit 71 of the present embodiment may acquire the overall flow line information L0 stored in the storage unit 94 of the flow line combination server 9 and use past flow line information in the overall flow line information L0 instead of the motion difference. In this way, the congestion determination unit 71 may generate a congestion flag indicating congestion or non-congestion as congestion information for each camera in the multiple cameras 2, for example, based on the position and speed of a recognition target in the past flow line information and the imaging range of each camera.
[0157] In the first embodiment, the image distribution unit 72 divides (S22, S23) and distributes (S24) the image data based on the congestion flag of the congestion information based on the motion difference. The image distribution unit 72 in this embodiment may acquire the overall flow line information L0 stored in the storage unit 94 of the flow line connection server 9 and use the past flow line information in the overall flow line information L0 instead of the congestion flag. The image distribution unit 72 may divide the images of each camera of the multiple cameras 2 based on the position of a recognition target such as the person 11 in the past flow line information so that the number of people in each divided image is approximately the same, and may determine the distribution destination so that the total number of people in the divided images in each distribution destination is approximately the same.
[0158] In the second embodiment, the congestion flag for each camera is set based on the adjacent information of the multiple cameras 2. The congestion management server 7 of the present embodiment is not limited to the above, and may update the congestion information using various information for predicting congestion and distributing image data. For example, the congestion determination unit 71 of the present embodiment may set the congestion flag based on advance information in the imaging range of each of the multiple cameras 2 instead of the adjacent information. As the advance information, for example, various knowledge of the environment 6 to which the present system 1 is applied, such as information indicating a schedule for conducting specific sales by dividing time periods in an environment 6 such as a sales floor of a store, may be used.
[0159] The above-mentioned advance information is defined, for example, to indicate that when the congestion flag is "on" in the imaging range of a specific camera among the multiple cameras 2, the congestion flag of the imaging range of another camera will become "on" after a predetermined time. The advance information may also be defined to indicate that the congestion flag is "on" in the imaging range of a specific camera during a specific time period. In this way, even cameras that are not adjacent can set a congestion flag and be used for dividing and distributing images by the image distribution unit 72. The above-mentioned advance information is an example of information related to the tendency of a moving object such as a person 11 to move.
[0160] In this embodiment, a trained model that predicts congestion by machine learning of prior information such as past traffic line information may be applied to manage the congestion state in the system 1. In such machine learning, additional information indicating, for example, the time, day of the week, season, weather, and various events related to the environment 6 when the system 1 is used may be used.
[0161] In the above embodiment, the traffic line detection system 1 has been described as an example of a moving object detection system. In this embodiment, the moving object detection system is not particularly limited to the traffic line detection system 1, and may be applied to, for example, a monitoring system for monitoring the movement of a moving object such as a person using a plurality of cameras, or a tracking system for tracking. In addition, in this embodiment, the detection target of the moving object detection system is not particularly limited to a person, and may be various moving objects. For example, the detection target moving object may be a cart in an environment 6 such as a store, an automatic conveyor in an environment 6 such as a factory, or various vehicles.
[0162] In the above embodiment, the trajectory extraction server 8 has been described as an example of an image recognition device. In this embodiment, the image recognition device is not particularly limited to the trajectory extraction server 8, and may be configured as, for example, a GPU or a TPU. In addition, when a trained model by deep learning is applied to image recognition processing, the image recognition device may be configured as dedicated hardware specialized for deep learning processing.
[0163] In the above embodiment, the congestion management server 7 has been described as an example of the information management device. In this embodiment, the information management device may be, for example, a PC. Similarly, the information integration device and the image recognition device may each be, for example, a PC.
[0164] As described above, the embodiments have been described as examples of the technology in the present disclosure, and the accompanying drawings and detailed description have been provided for this purpose.
[0165] Therefore, among the components described in the attached drawings and the detailed description, not only are there components essential for solving the problem, but there may also be components that are not essential for solving the problem in order to illustrate the above technology. Therefore, the fact that such non-essential components are described in the attached drawings or the detailed description should not be interpreted as immediately indicating that such non-essential components are essential.
[0166] Furthermore, since the above-described embodiments are intended to illustrate the technology in the present disclosure, various modifications, substitutions, additions, omissions, and the like can be made within the scope of the claims or their equivalents. [Industrial Applicability]
[0167] The present disclosure relates to various imaging systems that process information based on multiple imaging devices in real time. technology For example, it can be applied to motion detection systems, surveillance systems, and tracking systems. Mobile detection systems such as is applicable to.
Claims
1. A moving object detection system that detects a plurality of moving objects based on captured images, A plurality of imaging devices each capturing an image and generating image data; a plurality of image recognition devices provided corresponding to the plurality of imaging devices, each of which performs image recognition on image data captured by the corresponding imaging device to detect a moving object in a captured image represented by each image data; an information management device that manages image data supplied from each of the imaging devices to each of the image recognition devices, The information management device includes: a communication unit that performs data communication with each of the imaging devices and each of the image recognition devices; A control unit that controls an operation of the communication unit, the plurality of imaging devices includes a first imaging device, the plurality of image recognition devices include a first image recognition device corresponding to the first imaging device and a second image recognition device not corresponding to the first imaging device, The control unit is managing congestion information indicating the number of moving objects present within a range of a captured image indicated by each image data for each of the imaging devices; When the magnitude of the quantity indicated by the congestion information in the first image data by the first imaging device is less than a predetermined amount, the first image data is supplied to the first image recognition device; When the size of the quantity indicated by the congestion information in the first image data is equal to or larger than the predetermined amount, the communication unit is controlled so as to distribute the first image data and supply it to at least the first image recognition device and the second image recognition device. Mobile detection system.
2. Each of the image recognition devices repeatedly detects the moving object and generates movement line information indicating a movement line of the moving object. The moving object detection system according to claim 1 .
3. The information integration device further includes: an information integration device that generates information indicating a line of movement of the moving object in the entire area captured by the plurality of imaging devices based on the line of movement information generated by each of the image recognition devices. The moving object detection system according to claim 2 .
4. The control unit is Calculating an amount of movement of a moving object in the captured images based on a plurality of frames of image data captured sequentially by each of the imaging devices; The congestion information is generated based on the calculated amount of movement. The moving object detection system according to claim 1 .
5. The control unit is Obtaining the number of moving objects detected by each of the image recognition devices; The congestion information is generated based on the acquired number of moving objects. A moving object detection system according to any one of claims 1 to 4.
6. The control unit determines in advance to distribute the first image data before a quantity indicated by the congestion information in the first image data by the first imaging device becomes equal to or larger than the predetermined quantity according to at least one of the congestion information regarding an imaging device adjacent to the first imaging device among the plurality of imaging devices and information indicating a tendency of the congestion information to change due to movement of the moving object in an environment in which the plurality of imaging devices are arranged, and updates the congestion information. The moving object detection system according to any one of claims 1 to 5.
7. The control unit is Calculating a congestion degree indicating congestion information for each region dividing an image represented by each image data of each of the plurality of image capturing devices; Based on the calculated congestion degree, the image data captured by the plurality of imaging devices is distributed and transmitted to the plurality of image recognition devices. The moving object detection system according to any one of claims 1 to 6.
8. the plurality of imaging devices includes a second imaging device corresponding to the second image recognition device, In the second image data by the second imaging device, the magnitude of the quantity indicated by the congestion information is less than the predetermined amount. The moving object detection system according to any one of claims 1 to 7.
9. The control unit is Providing the second image data to the second image recognition device; When the size of the quantity indicated by the congestion information in the first image data is equal to or larger than the predetermined amount, the communication unit is controlled so as to separately supply a plurality of pieces of divided data for dividing the captured image indicated by the first image data into a plurality of areas to the first and second image recognition devices as distribution destinations. The moving object detection system according to claim 8.
10. An information management device in a moving object detection system including a plurality of imaging devices and a plurality of image recognition devices provided corresponding to the plurality of imaging devices, each of which recognizes a moving object in a captured image represented by each image data based on image data by each of the corresponding imaging devices, The information management device includes: a communication unit that performs data communication with each of the imaging devices and each of the image recognition devices; a control unit that controls an operation of the communication unit so as to manage image data supplied from each of the imaging devices to each of the image recognition devices; the plurality of imaging devices includes a first imaging device, the plurality of image recognition devices include a first image recognition device corresponding to the first imaging device and a second image recognition device not corresponding to the first imaging device, The control unit is managing congestion information indicating the number of moving objects in the captured images represented by the image data for each of the imaging devices; When the magnitude of the quantity indicated by the congestion information in the first image data by the first imaging device is less than a predetermined amount, the first image data is supplied to the first image recognition device; When the size of the quantity indicated by the congestion information in the first image data is equal to or larger than the predetermined amount, the communication unit is controlled so as to distribute the first image data and supply it to at least the first image recognition device and the second image recognition device. Information management device.
Citation Information
Patent Citations
Information processing apparatus, information processing method, and information processing system
JP2015103104A
Image analyzer, image analysis method, image analysis program and image analysis system
JP2017045438A
Image processing apparatus, image processing method, and program
JP2019200715A
Flow line detection process data distribution system, flow line detection process data distribution method, and program
WO2012098853A1
Congestion-state-monitoring system
WO2017122258A1
