Information processing system, method, and program

Abstract

To specify an object projected in an image by utilizing a plurality of detection results corresponding to different detection means used for detecting the object.SOLUTION: The monitoring system includes an acquisition unit configured to acquire information relating to a detection result of a detection unit configured to detect an object appearing in an image generated by image capturing of an image capturing apparatus, and a specification unit configured to specify the object appearing in the image in accordance with a relationship between whether or not the detection result satisfies a condition predetermined for each detection unit and the detection result, for each detection unit from which the information is acquired by the acquisition unit.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing system, a method, and a program. [Background technology]

[0002] Conventionally, there have been technologies for detecting objects from images for purposes such as monitoring traffic conditions on roads. There have also been technologies for detecting objects from images using multiple detection means for purposes such as improving object detection accuracy. Patent Document 1 discloses performing object detection by a first detection means and object detection by a second detection means in parallel, and performing object detection by the first detection means when an object cannot be detected by the second detection means. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-175103 Summary of the Invention [Problem to be solved by the invention]

[0004] However, as in the technology described in Patent Document 1, when the detection results of multiple detection means are used directly as the result of identifying an object, or when the detection result of only one of the multiple detection means is used, the multiple detection results are not fully utilized in accordance with the detection means to identify an object. The present invention aims to identify an object captured in an image by utilizing a plurality of detection results obtained by different detection means used to detect the object in accordance with the detection means. [Means for solving the problem]

[0005] In order to solve the above problem, the monitoring system of the present invention comprises: an acquisition means for acquiring information about the detection results of a detection means for detecting an object captured in an image generated by an imaging device; and an identification means for identifying an object captured in the image, for each of the detection means from which the information is acquired by the acquisition means, in accordance with a relationship between the detection result and whether or not the detection result satisfies a predetermined condition for each of the detection means. It is an information processing system equipped with the above. [Effects of the Invention]

[0006] According to the present invention, it is possible to identify an object captured in an image by utilizing a plurality of detection results obtained by different detection means used to detect the object in accordance with the detection means. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 2 is a diagram illustrating an example of the functional configuration of the monitoring system according to the first embodiment. [Figure 2] FIG. 2 illustrates a hardware configuration of a monitoring server. [Figure 3] FIG. 10 is a flowchart showing the flow of an output process. [Figure 4] FIG. 10 is a flowchart showing the flow of a specification process. [Figure 5] FIG. 1 is a diagram showing the relationship between an image generated by an imaging device and a partitioned area in a surveillance system. [Figure 6] FIG. 1A is a diagram showing the results of detection by the first detection unit, and FIG. 1B is a diagram showing the results of detection by the second detection unit. [Figure 7] FIG. 10(A) is a diagram showing first result information, (B) is a diagram showing second result information, and (C) is a diagram showing area relation information. [Figure 8] FIG. 10 is a diagram illustrating an example of the functional configuration of a monitoring system according to a second embodiment. [Figure 9] FIG. 10 is a diagram showing the relationship between an image generated by image capture with an imaging device and a partitioned area in a monitoring system in the second embodiment. [Figure 10] (A) is a diagram showing the results of detection by the first detection unit, (B) is a diagram showing the results of detection by the second detection unit, and (C) is a diagram showing the results of detection by the third detection unit. [Figure 11] (A) is a diagram showing the first result information, (B) is a diagram showing the second result information, (C) is a diagram showing the third result information, (D) is a diagram showing the area relationship information of the second embodiment, and (E) is a diagram showing the specific information. [Figure 12] FIG. 10 is a flowchart showing the flow of a type identification process. [Figure 13] 10 is a diagram showing area relation information when two imaging devices 0 are installed in the monitoring system 100. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing an example of the functional configuration of a monitoring system 100 according to a first embodiment. The monitoring system 100, which is an example of an information processing system, is a system that monitors objects. Examples of objects include moving objects such as cars and people such as pedestrians. The monitoring system 100 generates an image by capturing an image of an area defined as a monitoring area, detects objects from the generated image, and identifies the object shown in the image from the detection results. Furthermore, the monitoring system 100 according to this embodiment detects objects using multiple detection means and identifies the object using the respective detection results.

[0009] The monitoring area of ​​the monitoring system 100 may be an area used by mobile objects to move, such as a roadway. When the monitoring area is a roadway, the monitoring system 100 identifies objects such as mobile objects moving on the roadway and pedestrians moving on a crosswalk from captured images, and notifies the autonomously moving mobile objects of information indicating the identification results to use in driving, thereby facilitating smooth traffic. In the following, it is assumed that the monitoring area of ​​the monitoring system 100 is a roadway. The surveillance system 100 includes a surveillance server 110, an imaging device 120, and a storage device .

[0010] The imaging device 120 generates images by capturing images and transmits the generated images to the monitoring server 110. The images generated by the imaging device 120 and transmitted to the monitoring server 110 may be multiple still images or may be videos. When the images generated by the imaging device 120 and transmitted to the monitoring server 110 are still images, the imaging device 120 may generate images by capturing images at predetermined intervals, such as every second, and transmit the generated images to the monitoring server 110 each time an image is generated. Examples of the imaging device 120 include image sensor cameras such as a color camera, a monochrome camera, an infrared camera, a stereo camera, a ToF (Time of Flight) camera, and a SPAD (Single Photon Avalanche Diode) camera, a digital autofocus camera, etc. The imaging device 120 may also be a ranging device such as a LiDAR (Laser Imaging Detection And Ranging).

[0011] The monitoring server 110, which is an example of an information processing device, is a server device that monitors objects. The monitoring server 110 includes a detection unit 111, an identification unit 112, an acquisition unit 113, a communication unit 114, a management unit 115, and an output unit . The communication unit 114 communicates with devices external to the monitoring system 100 . The acquisition unit 113, which is an example of an acquisition means, acquires information from the imaging device 120, an external device connected via the communication unit 114, and the storage device 130. The information acquired by the acquisition unit 113 from the imaging device 120 includes an image generated by the imaging device 120.

[0012] The detection unit 111, which is an example of a detection means, detects an object from an image acquired by the acquisition unit 113. Examples of the detection unit 111 include a detection means that detects an object through machine learning using a learning model, a detection means that detects an object from the difference between multiple images, and a detection means that detects an object by comparing predetermined object features with an object captured in an image. Examples of learning models used in the detection means that detects an object through machine learning include HOG (Histgram of Oriented Gradient), CNN (Convolutional Neural Network), and ViT (Vision Transformer). Examples of methods for detecting an object from the difference between multiple images include background subtraction and optical flow. Examples of methods for detecting an object by comparing object features with an object captured in an image include pattern matching. The detection unit 111 may also be a detection means that detects an object from distance information. Examples of methods for detecting an object from distance information include clustering based on the distance between point clouds generated for each feature in an image and a method of detecting an object using a depth sensor based on the distance from the imaging device 120 to the object. Note that any detector 111 may be used as long as it can detect an object from information acquired from the image capture device 120.

[0013] When the detection unit 111 detects an object from an image, it identifies the area of ​​the image where the object is detected, the type of the detected object, and an index indicating the detection accuracy according to the detection performed, and outputs the identified information as a detection result. Note that information indicating the detection result generated by the detection unit 111 may be referred to as result information hereinafter. Also, information indicating the area of ​​the image where the object is detected, which is indicated in the result information, may be referred to as area information hereinafter. Also, information indicating the type of object, which is indicated in the result information, may be referred to as type information hereinafter. Also, information on the index indicating the detection accuracy, which is indicated in the result information, may be referred to as accuracy information hereinafter. Area information, type information, and accuracy information targeting the same object among the objects detected by the detection unit 111 are each associated and indicated in the result information.

[0014] The region detected by the detection unit 111 may be a rectangular or rectangular parallelepiped region surrounding an object in an image, or may be a result of labeling an object region on a pixel-by-pixel basis, such as in semantic segmentation or instance segmentation. Moreover, the result information generated by the first detection unit 111a may be referred to as first result information hereinafter. Moreover, the result information generated by the second detection unit 111b may be referred to as second result information hereinafter. Moreover, the area information included in the first result information may be referred to as first area information hereinafter. Moreover, the area information included in the second result information may be referred to as second area information hereinafter.

[0015] The monitoring server 110 is also provided with a plurality of detection units 111. The monitoring server 110 of this embodiment is provided with a first detection unit 111a and a second detection unit 111b as the detection units 111. In the following description, the first detection unit 111a detects objects through machine learning using CNN as a learning model, and the second detection unit 111b detects objects through machine learning using ViT as a learning model. Here, the second detection unit 111b, which detects objects using ViT as a learning model, has higher detection accuracy for large objects than the first detection unit 111a, which detects objects using CNN as a learning model. On the other hand, the second detection unit 111b has difficulty detecting small objects compared to the first detection unit 111a. In other words, the second detection unit 111b has higher detection accuracy for objects present in a monitoring area close to the image capture device 120 compared to the first detection unit 111a, but has difficulty detecting objects present in a monitoring area far from the image capture device 120. In other words, the second detection unit 111b has a higher detection accuracy and detection rate for objects present in a monitoring area closer to the image capture device 120 than the first detection unit 111a. Also, the first detection unit 111a has a higher detection accuracy and detection rate for objects present in a monitoring area farther from the image capture device 120 than the second detection unit 111b. In this way, the monitoring server 110 is provided with detection units 111 each having different characteristics. When the first detection unit 111a and the second detection unit 111b are described without any particular distinction, they are simply referred to as detection units 111. The number of detection units 111 provided in the monitoring server 110 is not limited to the example shown in the figure. Three or more detection units 111 may be provided in the monitoring server 110.

[0016] The management unit 115 manages information indicating the areas divided into sections of the monitoring area. As described above, the object detection rate by the detection unit 111 may differ depending on the area of ​​the monitoring area. Therefore, in this embodiment, the monitoring area is divided into areas where the object detection rate by the detection unit 111 differs, and information indicating the area for each section is managed by the management unit 115.

[0017] The identification unit 112, which is an example of an identification means, identifies an object appearing in an image. More specifically, the identification unit 112 uses the result information for each detection unit 111 to identify whether an object appears in the image, the area in the image where the object appears, the type of object appearing in the image, and the like. The identification unit 112 also identifies the object based on which category managed by the management unit 115 the area in the image identified from the result information, in which the object was detected by the detection unit 111, belongs to. The identification unit 112 generates information indicating the result of the object identification. Note that, hereinafter, the information indicating the result of the object identification by the identification unit 112 may be referred to as identification information.

[0018] The output unit 116 outputs the generated identification information to the identification unit 112. Examples of output of the identification information by the output unit 116 include transmitting the identification information to an external device and displaying the identification information on a display means (not shown) in the monitoring server 110. Examples of the external device include a mobile object located in a predetermined range such as the monitored area or near the monitored area, and storage means for managing the identification information. The acquisition of information from the image capture device 120 by the acquisition unit 113 and the transmission of information to an external device by the output unit 116 may be performed via the communication unit 114.

[0019] The storage device 130 stores images captured by the imaging device 120, result information, specific information, etc. Examples of the storage device 130 include a hard disk drive (HDD), a solid state drive (SSD), and a flash memory. In the illustrated example, the storage device 130 is provided outside the monitoring server 110, but the storage device 130 may be built into the monitoring server 110. Alternatively, the storage device 130 may be provided outside the monitoring system 100 and connected to the monitoring server 110 via a network.

[0020] The monitoring server 110 and the storage device 130 are realized, for example, by a computer. The monitoring server 110 and the storage device 130 may each be configured by a single computer, or may be realized by distributed processing using multiple computers. The monitoring server 110 and the storage device 130 may also be configured by a single computer. That is, the monitoring server 110 and the storage device 130 may be the same device. The monitoring server 110 and the storage device 130 may also be realized on virtual hardware provided by cloud computing.

[0021] The monitoring server 110 and the imaging device 120, and the monitoring server 110 and the storage device 130 are connected to each other via a network. The type of these networks is not particularly limited as long as they are capable of transmitting and receiving data, and may be, for example, the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), etc. Furthermore, the communication line used for transmitting and receiving data may be either wired or wireless.

[0022] FIG. 2 is a diagram showing the hardware configuration of the monitoring server 110. The monitoring server 110 includes a CPU 201, a storage device 202, a memory 203, an operation I / F 204, and a communication I / F 205. The CPU 201 controls the monitoring server 110. The CPU 201 loads programs from the storage device 202 into the memory 203 and executes them to perform various controls. The storage device 202 stores an operating system (OS), programs, management data, data collected from external systems and devices, etc. The memory 203 functions as a work area for the CPU 201, etc. The operation I / F 204 outputs various data and execution results of programs to a connected output device such as a display, and receives input from a connected input device. The communication I / F 205 is a network interface for communicating with external systems and devices. The acquisition unit 113, communication unit 114, and output unit 116 of the monitoring server 110 are realized by the communication I / F unit 205. Furthermore, the processing by the detection unit 111 and identification unit 112 of the monitoring server 110 is realized by the CPU 201 loading programs stored in the storage device 202 into the memory 203 and executing them. Furthermore, the management unit 115 is realized by the storage device 202 or the memory 203.

[0023] 3 is a flowchart showing the flow of the output process. The output process is a process in which the monitoring system 100 outputs the results of identifying an object shown in an image generated by capturing an image as identification information. In this embodiment, the output process is started every time the imaging device 120 transmits an image generated by capturing an image to the monitoring server 110. Note that each step in the output process is realized by the CPU 201 of the monitoring server 110 loading a program stored in the storage device 202 into the memory 203 and executing it. The acquisition unit 113 of the monitoring server 110 acquires an image generated by the imaging device 120 through imaging by the imaging device 120 from the imaging device 120 (step (hereinafter referred to as "S") 200).

[0024] The first detection unit 111a and the second detection unit 111b each detect an object from the image acquired by the acquisition unit 113, and generate result information indicating the detection result (S201). Note that the processing by the first detection unit 111a and the processing by the second detection unit 111b in step 201 may be performed sequentially or in parallel. The identification unit 112 performs an identification process using the result information generated by the first detection unit 111a and the second detection unit 111b (S202). As will be described in detail later, this identification process identifies an object appearing in the image and generates identification information indicating the identification result. The output unit 116 outputs the identification information generated as a result of the object identification by the identification unit 112 (S203).

[0025] 4 is a flowchart showing the flow of the identification process, in which the monitoring server 110 identifies an object captured in an image generated by the imaging device 120. The identifying unit 112 of the monitoring server 110 acquires the result information generated by each detecting unit 111 in step 201 of the output process (see FIG. 3) (S301). The identification unit 112 identifies, from the region information included in the acquired result information, to which of the regions managed by the management unit 115 the region in which the object was detected in the image belongs (S302). When one piece of result information includes multiple pieces of region information, in other words, when multiple objects are detected in one image, the identification unit 112 identifies, for each detected object, to which region the region in which the object was detected belongs. Furthermore, the identification unit 112 identifies, for each piece of result information, to which region the region in which the object was detected belongs. Furthermore, the identification unit 112 associates information indicating the identified region with the result information. Note that, hereinafter, information indicating the region identified by the identification unit 112 may be referred to as region information.

[0026] The identification unit 112 associates information identifying the object with the result information (S303). More specifically, for each object detected by the detection unit 111, the identification unit 112 newly associates information identifying the object with the region information, type information, and accuracy information associated with each other in the result information. Hereinafter, the information identifying the object may be referred to as identification information. When first region information and second region information satisfying an identity condition exist, the identification unit 112 determines that the objects detected in the regions identified from the first region information and the second region information are the same object, and associates the same identification information with the first region information and the second region information. The identity condition is a condition used by the identification unit 112 to determine whether or not objects detected by different detection units 111 are identified as the same object. In this embodiment, the identity condition is defined as the overlap rate between the region identified from the first region information and the region identified from the second region information being equal to or greater than a predetermined value. The identification unit 112 associates different identification information with each piece of region information that does not satisfy the identity condition. The overlap rate between the first region information and the second region information is calculated using a known method such as IoU (Intersection over Union).The same condition may be satisfied if the Euclidean distance between the center of gravity of the region identified from the first region information and the center of gravity of the region identified from the second region information is equal to or less than a predetermined distance.

[0027] The identification unit 112 determines, among the objects detected by each detection unit 111, one object for which a determination, which will be described later, has not been made in the identification process, as a target for determination (S304). Note that, hereinafter, the object determined as a target for determination may be referred to as a target object. The identification unit 112 calculates an index related to the detection accuracy of the target object (S305). Note that the index related to the detection accuracy calculated by the identification unit 112 may be referred to as an accuracy index hereinafter. The identification unit 112 identifies, as the accuracy index, a value calculated using an index indicated in the accuracy information for the target object and an index identified from the classification information for the target object. Furthermore, if the target object is an object identified from first region information and second region information that satisfy the same condition, that is, if the target object is an object identified from the same identification information associated with both the first result information and the second result information, the identification unit 112 identifies, as the accuracy index, the cumulative value of the value calculated using the index indicated in the accuracy information included in the first result information and the value calculated using the index indicated in the accuracy information included in the second result information.

[0028] The identification unit 112 determines whether the calculated accuracy index satisfies a normal condition (S306). The normal condition is a condition used by the identification unit 112 to determine whether the object detection by the detection unit 111 is normal. In this embodiment, the normal condition is defined as the accuracy index calculated by the identification unit 112 for the target object being equal to or greater than a predetermined threshold. The threshold may be any value, but is, for example, 0.4. If the accuracy index satisfies the normality condition (YES in S306), the identification unit 112 identifies that the detection of the target object by the detection unit 111 is normal. In this case, the identification unit 112 identifies the type of object and the area in which the object is located from the result information, and generates identification information indicating the identification result (S307).

[0029] On the other hand, if the accuracy index does not satisfy the normal condition (NO in S306), the identification unit 112 identifies that the detection of the target object by the detection unit 111 is an erroneous detection. In this case, the identification information is not generated. If a negative result is obtained in step 306, or after step 307, the identification unit 112 determines whether there are any objects detected by each detection unit 111 that have not yet been determined to satisfy the normal conditions (S308).

[0030] If there is an object for which the determination as to whether it satisfies the normal condition has not been performed (NO in S308), the identification unit 112 repeats the process from step 304. As a result, the determination as to whether it satisfies the normal condition is performed for all objects detected by each detection unit 111, and identification information is generated for the objects that are the detection targets and that are determined to be normal. If there is no object for which it has not been determined whether or not it satisfies the normal condition (YES in S308), the identification process ends.

[0031] FIG. 5 is a diagram showing the relationship between an image 400 generated by image capture by the imaging device 120 and the areas partitioned in the monitoring system 100. As shown in FIG. As shown in Fig. 5, image 400 shows multiple vehicles and multiple people. Here, in surveillance system 100, the area shown in image 400 is divided into two areas with dashed line L1 as the boundary, depending on the distance from image capture device 120. More specifically, of the area shown in image 400, the area below dashed line L1 in the figure is divided as divided area 401, and the area above dashed line L1 in the figure is divided as divided area 402. Here, divided area 402 is closer to image capture device 120 than divided area 401. In other words, divided area 401 is perceived as a monitoring area closer to image capture device 120, and divided area 402 is perceived as a monitoring area farther from image capture device 120. In the illustrated example, three vehicles and one person appear in a divided area 401 of the image 400. Also, two vehicles and one person appear in a divided area 402 of the image 400.

[0032] Fig. 6 is a diagram showing the results of detection by the detection unit 111 for the image 400 shown in Fig. 5. More specifically, Fig. 6(A) is a diagram showing the results of detection by the first detection unit 111a, and Fig. 6(B) is a diagram showing the results of detection by the second detection unit 111b. 6(A), the first detection unit 111a detects eight objects from the image 400. More specifically, a vehicle C1, a vehicle C2, a vehicle C3, a vehicle C4, a vehicle C5, a person P1, a person P2, and a person P3 are detected from the image 400. In addition, the first detection unit 111a erroneously detects the person P3 from an area of ​​the image 400 where no person is captured. The symbols shown for each of the vehicles C1 to C5 and the people P1 to P3 are type information indicating the type of object detected by the first detection unit 111a.

[0033] 6(B), the second detection unit 111b detects six objects from the image 400. More specifically, cars Ca, Cb, Cc, Cd, Ce, and a person Pa are detected from the image 400. In addition, the second detection unit 111b does not detect the object detected as person P2 by the first detection unit 111a. The symbols shown for each of the cars Ca to Ce and the person Pa are type information indicating the type of object detected by the second detection unit 111b.

[0034] 7(A) is a diagram showing first result information 503, and FIG. 7(B) is a diagram showing second result information 504. The first result information 503 shown in FIG. 7(A) is information showing the result of detection by the first detection unit 111a shown in FIG. 6(A). The second result information 504 shown in FIG. 7(B) is information showing the result of detection by the second detection unit 111b shown in FIG. 6(B). When the first result information 503 and the second result information 504 are described without any particular distinction, they may be simply referred to as result information. As shown in Figures 7(A) and 7(B), the first result information 503 and the second result information 504 respectively show, in association with each detected object, the "object type," "minimum coordinates," "maximum coordinates," and "accuracy score."

[0035] "Object type" indicates type information. The "minimum coordinate" indicates the smallest coordinate among the coordinates in the image where the object detected by the detection unit 111 is located. In addition, in the surveillance system 100, the area in the image (see FIG. 5, etc.) where the object is detected is defined in a two-dimensional Cartesian coordinate system consisting of the x-axis and the y-axis, and the "minimum coordinate" indicates the smallest coordinate among the coordinates corresponding to the area detected by the detection unit 111. The "maximum coordinate" indicates the maximum coordinate among the coordinates corresponding to the area detected by the detection unit 111. The "minimum coordinate" and "maximum coordinate" indicated in the first result information 503 and the second result information 504 are examples of area information. The "accuracy score" indicates an index of detection accuracy identified from the accuracy information.

[0036] Furthermore, the first result information 503 and the second result information 504 are associated with a "divided area" and an "object ID." In the "divided area" field, either divided area 401 or divided area 402 is displayed as division information indicating the divided area to which the area in which the object is detected by detection unit 111 belongs.

[0037] The "object ID" indicates identification information. Here, the type information "C1" of the first result information 503 and the type information "Ca" of the second result information 504 are associated with the same identification information "OBJ001." This means that the first region information "minimum coordinate" and "maximum coordinate" associated with the type information "C1" of the first result information 503 and the second region information "minimum coordinate" and "maximum coordinate" associated with the type information "Ca" of the second result information 504 satisfy the same condition. In this way, the first region information and the second region information that satisfy the same condition are associated with the same identification information. In addition, the type information "C2" of the first result information 503 and the type information "Cb" of the second result information 504 are associated with the same identification information "OBJ002." Furthermore, the type information "C3" of the first result information 503 and the type information "Cc" of the second result information 504 are associated with the same identification information "OBJ003." Furthermore, the type information "C4" of the first result information 503 and the type information "Cd" of the second result information 504 are associated with the same identification information "OBJ004." Furthermore, the type information "C5" of the first result information 503 and the type information "Ce" of the second result information 504 are associated with the same identification information "OBJ005." Furthermore, the type information "P1" of the first result information 503 and the type information "Pa" of the second result information 504 are associated with the same identification information "OBJ006." Furthermore, the identification information "OBJ007" is associated with the type information "P2" of the first result information 503, but is not associated with the second result information 504. This means that the first area information "minimum coordinate" and "maximum coordinate" associated with the type information "P2" of the first result information 503 do not satisfy the same condition. Furthermore, the identification information "OBJ008" is associated with the type information "P3" of the first result information 503, but is not associated with the second result information 504. This means that the first area information "minimum coordinate" and "maximum coordinate" associated with the type information "P3" of the first result information 503 do not satisfy the same condition.

[0038] 7C is a diagram showing area relation information 505. The area relation information is information indicating the relationship between a divided area and an index related to detection accuracy. The area relation information 505 is managed by the management unit 115. The area relation information 505 indicates the "divided area" and the "coefficient." In the "Divided Area" field, "Divided Area 401" and "Divided Area 402" are shown as the divided areas to which the areas in which the object is detected by the detection unit 111 belong.

[0039] The "coefficient" indicates a coefficient used by the identification unit 112 to calculate an accuracy index for the detection of an object by the detection unit 111. In this embodiment, the "coefficient" indicates a value depending on whether the "delimited area" is the "delimited area 401" or the "delimited area 402." The "coefficient" also indicates a value depending on whether the detection unit 111 that detected the object is the "first detection unit" or the "second detection unit." In the illustrated example, for the "first detection unit," the "coefficient" associated with the "delimited area 401" is "0.3," and the "coefficient" associated with the "delimited area 402" is "0.7." The "coefficient" associated with the "second detection unit" is "0.7," and the "coefficient" associated with the "delimited area 402" is "0.3."

[0040] The "divided area" and "coefficient" may be set by a user of the monitoring system 100, or may be set based on the results of machine learning from the history of detection results of the detection unit 111. The "divided area" and "coefficient" may also be changed over time while the monitoring system 100 is in operation, such as by changing them according to the season, or by changing them to different contents between day and night.

[0041] In step 305 of the identification process (see Figure 4), the identification unit 112 calculates an accuracy index for the target object determined as the object to be determined from the "accuracy score" indicated in the result information (see Figures 7(A) and 7(B)) and the "coefficient" indicated in the area relationship information 505. A specific method for calculating the accuracy index by the identification unit 112 will be described. The identification unit 112 extracts an "accuracy score" associated with the "object ID" of the target object in the result information. The identification unit 112 also extracts a "coefficient" associated with the "divided area" in the area relation information 505 that matches the "divided area" associated with the "object ID" of the target object in the result information, the "coefficient" being indicated for the detection unit 111 that detected the target object. The identification unit 112 then multiplies the extracted "accuracy score" by the "coefficient" to obtain a product value as the accuracy index. Furthermore, there are cases where the target object has been detected by both the first detection unit 111a and the second detection unit 111b, i.e., where the same "object ID" is associated with the first result information 503 and the second result information 504. In this case, the identification unit 112 determines the accuracy index for the target object as the cumulative value of the multiplication value obtained by multiplying the "accuracy score" extracted for the first detection unit 111a by the "coefficient" and the multiplication value obtained by multiplying the "accuracy score" extracted for the second detection unit 111b by the "coefficient." The identification unit 112 also calculates the accuracy index for each object identified from the "object ID" associated with the result information.

[0042] An example of calculation of the detection accuracy by the identification unit 112 will be described. If the object whose "object ID" is "OBJ001" is the target object, the identification unit 112 extracts "0.91" as the "accuracy score" associated with "OBJ001" in the first result information 503. Furthermore, the identification unit 112 extracts "0.3" as the "coefficient" associated with the "first detection unit" and "divided area 401," which is the "divided area" associated with "OBJ001" in the first result information 503 from among the "divided areas" indicated in the area relation information 505. The identification unit 112 then multiplies the extracted "0.91" by "0.3" to obtain the multiplied value "0.273." Furthermore, the identification unit 112 extracts "0.98" as the "accuracy score" associated with "OBJ001" in the second result information 504. Furthermore, the identification unit 112 extracts "0.7", which is the "coefficient" associated with "second detection unit", and "delimited area 401", which is the "delimited area" associated with "OBJ001" in the second result information 504, from among the "delimited areas" indicated in the area relation information 505. The identification unit 112 then multiplies the extracted "0.98" by "0.7" to obtain the multiplied value "0.686". Furthermore, the identification unit 112 determines that the accuracy index for the target object is "0.959", which is the cumulative value of the multiplied value "0.273" for the first detection unit 111a and the multiplied value "0.686" for the second detection unit 111a. The accuracy index "0.959" calculated by the identification unit 112 is greater than the threshold "0.4" of the normal condition (see S306 in FIG. 4), and therefore satisfies the normal condition (YES in S306 in FIG. 4). In this case, the identification unit 112 identifies the type of object and the area in which the object is located from the result information, and generates identification information indicating the identification result (see S307 in FIG. 4).

[0043] Another example of calculation of the accuracy index by the specification unit 112 will be described. If the object whose "object ID" is "OBJ007" is the target object, the identification unit 112 extracts "0.89" as the "accuracy score" associated with "OBJ007" in the first result information 503. Furthermore, the identification unit 112 extracts "0.7" as the "coefficient" associated with the "first detection unit" and "delimited area 402," which is the "delimited area" associated with "OBJ007" in the first result information 503 from among the "delimited areas" indicated in the area relation information 505. The identification unit 112 then multiplies the extracted "0.89" by "0.7" to obtain the multiplied value "0.623." Furthermore, the "object ID" of "OBJ007" is not associated with the second result information 504, and the target object has not been detected by the second detection unit 111b. Therefore, the identification unit 112 sets the multiplied value "0.623" for the first detection unit 111a as the accuracy index for the target object. The accuracy index "0.623" calculated by the identification unit 112 is greater than the threshold value for the normal condition and therefore satisfies the normal condition (YES in S306 in FIG. 4). In this case, the identification unit 112 identifies the type of object and the area in which the object is located from the result information, and generates identification information indicating the identification result (see S307 in FIG. 4).

[0044] Another example of calculation of the accuracy index by the specification unit 112 will be described. If the object whose "object ID" is "OBJ008" is the target object, the identification unit 112 extracts "0.55" as the "accuracy score" associated with "OBJ008" in the first result information 503. Furthermore, the identification unit 112 extracts "0.3" as the "coefficient" associated with the "first detection unit" and "divided area 401," which is the "divided area" associated with "OBJ008" in the first result information 503, from among the "divided areas" indicated in the area relation information 505. The identification unit 112 then multiplies the extracted "0.55" by "0.3" to obtain the multiplied value "0.165." Furthermore, the "object ID" of "OBJ008" is not associated with the second result information 504, and the target object has not been detected by the second detection unit 111b. Therefore, the identification unit 112 sets the multiplication value "0.165" for the first detection unit 111a as the accuracy index for the target object. The accuracy index "0.165" calculated by the identification unit 112 is smaller than the threshold value for the normal condition and therefore does not satisfy the normal condition (NO in S306 in FIG. 4). In this case, the identification unit 112 determines that the detection of the target object by the detection unit 111 is an erroneous detection and does not generate identification information.

[0045] The above-described calculation of the accuracy index by the specifying unit 112 is performed using the following formula (1). TIFF2026019475000002.tif16160In formula (1), i represents a number that identifies whether the detection unit 111 is the first detection unit 111a or the second detection unit 111b. Also, in formula (1), w represents a "coefficient" in the area relation information 505, and s represents an "accuracy score" in the result information. From formula (1), "0.942" is calculated as the accuracy index for the object whose "object ID" is "OBJ002," and "0.962" is calculated as the accuracy index for the object whose "object ID" is "OBJ003." Also, from formula (1), "0.966" is calculated as the accuracy index for the object whose "object ID" is "OBJ004," and "0.932" is calculated as the accuracy index for the object whose "object ID" is "OBJ005." Furthermore, from equation (1), the accuracy index of the object whose "object ID" is "OBJ006" is calculated to be "0.962." Therefore, the accuracy index also satisfies the normal condition for all of the objects "OBJ002," "OBJ003," "OBJ004," "OBJ005," and "OBJ006." Then, for each object whose accuracy index satisfies the normal condition, the identification unit 112 identifies the type of object and the area in which the object is located from the result information, and generates identification information indicating the identification result.

[0046] In this manner, in this embodiment, the coefficient used to calculate the detection accuracy differs for each detection unit 111. Also, the coefficient used to calculate the detection accuracy differs for each partitioned area to which the area in which the object is detected belongs. The larger the coefficient, the higher the calculated detection accuracy for the detection unit 111 that uses this coefficient. As described above, the second detection unit 111b has a higher detection accuracy and detection rate for objects present in a monitoring area closer to the image capture device 120 than the first detection unit 111a. Furthermore, the first detection unit 111a has a higher detection accuracy and detection rate for objects present in a monitoring area farther from the image capture device 120 than the second detection unit 111b. Therefore, in the region relation information 505, a higher coefficient is assigned to the second detection unit 111b than to the first detection unit 111a for the "delimited area 401," which is a delimited area closer to the image capture device 120. Furthermore, in the region relation information 505, a higher coefficient is assigned to the second detection unit 111a than to the first detection unit 111b for the "delimited area 402," which is a delimited area farther from the image capture device 120.

[0047] Next, generation of identification information by the identification unit 112 will be described. As described above, the identification unit 112 generates identification information by identifying the object type and the area in which the object is located from the result information for an object whose accuracy index satisfies the normal condition. The identification unit 112 may identify the "object type" indicated in the result information as the object type. The identification unit 112 may also identify the "minimum coordinate" and "maximum coordinate" indicated in the result information as the area in which the object is located. If an object is detected by both detection units 111, the "minimum coordinate" and "maximum coordinate" detected by the detection unit 111 having a higher multiplication value of the "detection accuracy score" and the "coefficient" may be identified as the area in which the object is located. If an object is detected by both detection units 111, the average value of the "minimum coordinate" and the average value of the "maximum coordinate" for each detection unit 111 may be identified as the area in which the object is located.

[0048] Fig. 7(D) is a diagram showing the identification information 506. As shown in Fig. 7(D), the identification information 506 shows, for each object whose accuracy index satisfies the normal condition, the "object type," "minimum coordinates," "maximum coordinates," "accuracy index," and "object ID." The "object type" indicates the type of object identified by the identification unit 112. The "minimum coordinate" indicates the minimum coordinate among the coordinates corresponding to the area identified by the identification unit 112 as the position of the object. The "maximum coordinate" indicates the maximum coordinate among the coordinates corresponding to the area identified by the identification unit 112 as the position of the object. The "accuracy index" indicates the accuracy index calculated by the identifying unit 112. "Object ID" indicates identification information.

[0049] The above-mentioned specific information 506 is output by the output unit 116. This allows the specific information 506 to be utilized by a mobile body, which is the output destination, or a user of a display means on which the specific information 506 is displayed.

[0050] Second Embodiment Next, a monitoring system 100 according to a second embodiment will be described. Note that in the second embodiment, configurations different from those in the first embodiment will be described, and descriptions of configurations similar to those in the first embodiment will be omitted. FIG. 8 is a diagram illustrating an example of the functional configuration of a monitoring system 100 according to the second embodiment. The monitoring system 100 according to the second embodiment differs from the first embodiment in the number of detection units 111 provided in the monitoring server 110. More specifically, the monitoring server 110 of the monitoring system 100 according to the second embodiment includes not only a first detection unit 111a and a second detection unit 111b but also a third detection unit 111c. Hereinafter, the third detection unit 111c is assumed to detect an object from the difference between multiple images using a background subtraction method. However, the third detection unit 111c may also detect an object from the difference between multiple images using optical flow. Hereinafter, when the first detection unit 111a, the second detection unit 111b, and the third detection unit 111c are described without any particular distinction, they may be simply referred to as the detection unit 111.

[0051] Here, the third detection unit 111c, which detects an object from the difference between multiple images, has a lower detection accuracy and detection rate for objects present in a monitoring area closer to the image capture device 120 than the first detection unit 111a. Also, the third detection unit 111c has a lower detection accuracy and detection rate for objects present in a monitoring area farther from the image capture device 120 than the second detection unit 111b. On the other hand, the third detection unit 111c has a higher detection accuracy and detection rate for objects that are partially hidden in an image than the first detection unit 111a and the second detection unit 111b. Therefore, in this embodiment, in an area where a part of an object to be detected is hidden, the coefficients are determined in the area relation information 505 so that the accuracy index for the third detection unit 111c is highest. Note that an example of a case where an object is hidden in an image is when another object exists between the image capture device 120 and the object to be detected.

[0052] The output process (see FIG. 3) and the identification process (see FIG. 4) in the second embodiment are the same as those in the first embodiment, except that not only the first detection unit 111a and the second detection unit 111b but also the third detection unit 111c are used as the detection unit 111. That is, in the output process, each detection unit 111 generates result information in response to the detection of an object. Also, in the identification process, an object is identified from the result information for each detection unit 111.

[0053] FIG. 9 is a diagram showing the relationship between an image 800 generated by image capture by the imaging device 120 and divided areas in the second embodiment. As shown in Fig. 9, image 800 shows multiple vehicles and multiple people. In this embodiment, in surveillance system 100, the area shown in image 800 is divided into three areas, with dashed lines L2 and L3 as boundaries, depending on the area of ​​the image. More specifically, the area below dashed line L2 in image 800, excluding the area surrounded by dashed line L3, is divided as divided area 801. Furthermore, the area above dashed line L2 in image 800 is divided as divided area 802. Furthermore, the area surrounded by dashed line L3 in image 800 is divided as divided area 803. In the illustrated example, two cars and one person are shown in each of demarcated areas 801 and 802 of image 800. One car is shown in demarcated area 803 of image 800. Note that in image 800, the car shown in demarcated area 803 is partially hidden by a traffic light.

[0054] Image 800 shown in FIG. 9 is the same as image 400 (see FIG. 5) except for the inclusion of a traffic light. Additionally, the vehicles and people in image 800 and image 400 are in the same positional relationship. Also, demarcated area 801 is the area of ​​demarcated area 401 (see FIG. 5) excluding demarcated area 803. Also, demarcated area 802 is the same area as demarcated area 402.

[0055] Fig. 10 is a diagram showing the results of detection by the detection unit 111 for the image 800 shown in Fig. 9. More specifically, Fig. 10(A) is a diagram showing the results of detection by the first detection unit 111a, Fig. 10(B) is a diagram showing the results of detection by the second detection unit 111b, and Fig. 10(C) is a diagram showing the results of detection by the third detection unit 111c. 10(A), the first detection unit 111a detects a car C1, a car C2, a car C3, a car C5, a person P1, a person P2, and a person P3 from the image 800. In addition, the first detection unit 111a does not detect the car that was detected as "car C4" in the example shown in FIG. 6(A) because part of this car is hidden by a traffic light. 10(B), the second detection unit 111b detects cars Ca, Cb, Cc, Ce, and a person Pa from the image 800. In addition, the second detection unit 111b does not detect the car that was detected as "car Cd" in the example shown in FIG. 6(B) because part of this car is hidden by a traffic light.

[0056] 10(C), the third detection unit 111c detects objects B1, B2, B3, B4, B5, B6, B7, B8, and B9 from the image 800. Additionally, the third detection unit 111c has difficulty identifying the object types, and detects both the cars and people in the image as objects without a specific type. Furthermore, the third detection unit 111c erroneously detects objects B8 and B9 from an area in which neither a car nor a person is captured. Furthermore, the third detection unit 111c detects a car partially hidden by a traffic light, i.e., a car not detected by either the first detection unit 111a or the second detection unit 111b.

[0057] Fig. 11(A) is a diagram showing first result information 904, Fig. 11(B) is a diagram showing second result information 905, and Fig. 11(C) is information showing third result information 906. Note that the third result information 906 is result information generated by the third detection unit 111c. 11(A) to 11(C), the first result information 904, the second result information 905, and the third result information 906 each show an "object type," "minimum coordinates," "maximum coordinates," and an "accuracy score" associated with each detected object. In addition, the first result information 904, the second result information 905, and the third result information 906 each show a "divided area" and an "object ID" associated with each detected object.

[0058] 11(A) is the same information as the first result information 503, except that it does not include information about the car identified as "C4" in the first result information 503 (see FIG. 7(A)), in other words, the car shown in the defined area 803. In addition, the first result information 904 does not include information about the object located in the defined area 803. 11(B) is the same as the second result information 504, except that it does not include information about the car identified as "Cd" in the second result information 504 (see FIG. 7(B)), in other words, the car shown in the defined area 803. In addition, the second result information 905 does not include information about the object located in the defined area 803.

[0059] 11(C), "N / A" is displayed in the "Object Type" of the third result information 906. This "N / A" means that the type of object has not been identified. The number following "N / A" in the "Object Type" field indicates a number that identifies the detected object. Furthermore, the "accuracy score" of the third result information 906 is always set to "1." However, the "accuracy score" for the third detection unit 111c may be a value calculated based on the detection results, such as the size of the area where the object is detected and the proportion of the specific color in the area where the object is detected.

[0060] Furthermore, the third result information 906 indicates that "divided area 803" is the "divided area" to which the area in which an object with an "object type" of "N / A4" is detected belongs. The object with an "object type" of "N / A4" is a car located in the divided area 803 and detected by the third detection unit 111c (see FIG. 10(C)). Note that in the third result information 906, "N / A4" is associated with "OBJ004" as the "object ID." Furthermore, in the third result information 906, "N / A8" and "N / A9" are objects identified from area information that do not satisfy the same condition. In other words, "N / A8" and "N / A9" are objects that have not been detected by either the first detection unit 111a or the second detection unit 111b. Therefore, the "object ID" "OBJ009" associated with "N / A8" and the "object ID" "OBJ010" associated with "N / A9" are not associated with the first result information 904 and the second result information 905.

[0061] FIG. 11D is a diagram showing the area relation information 907 of the second embodiment. In the "divided area" of the area relation information 907, "divided area 801," "divided area 802," and "divided area 803" are indicated as divided areas to which the areas in which the object is detected by the detection unit 111 belong. The "coefficient" in the area relation information 907 indicates the coefficient used to calculate the accuracy index for each detection unit 111 and for each divided area. In the illustrated example, for the "first detection unit," the "coefficient" associated with the "divided area 801" is "0.3," the "coefficient" associated with the "divided area 802" is "0.7," and the "coefficient" associated with the "divided area 803" is "0.3." Furthermore, for the "second detection unit," the "coefficient" associated with the "divided area 801" is "0.7," the "coefficient" associated with the "divided area 802" is "0.3," and the "coefficient" associated with the "divided area 803" is "0.1." Furthermore, for the "third detection unit," the "coefficients" associated with the "divided area 801" and "divided area 802" are both "0," and the "coefficient" associated with the "divided area 803" is "0.6."

[0062] As described above, the detection accuracy and detection rate of the third detection unit 111c are lower than those of the first detection unit 111a and the second detection unit 111b for both objects present in a monitoring area close to the image capture device 120 and objects present in a monitoring area far from the image capture device 120. Therefore, in this embodiment, the "coefficient" of the area relation information 907 is determined so as to suppress the detection result of the third detection unit 111c from affecting the identification by the identification unit 112 of the objects detected in the divided area 801 and the objects detected in the divided area 802. Furthermore, the third detection unit 111c has a higher detection accuracy and detection rate for objects partially hidden in an image than the first detection unit 111a and the second detection unit 111b. Specifically, the detection unit 111 that detects objects using a learning model may experience a decrease in detection rate when the feature amount (e.g., texture or edge component) of the object is missing, such as when the object to be detected is partially hidden. Therefore, in this embodiment, not only the detection unit 111 that detects objects using a learning model but also the detection unit 111 that detects objects using a background subtraction method is used, thereby preventing a decrease in the detection rate of the detection unit 111 even when the object is partially hidden. Specifically, for the partitioned area 803 where it is previously recognized that the object to be detected by the detection unit 111 is hidden by a traffic light, the "coefficient" of the area relation information 907 is determined so that the detection result of the third detection unit 111c has the greatest effect on the identification of the object by the identification unit 112.

[0063] The method of calculating the accuracy index by the identification unit 112 is the same as in the first embodiment. Therefore, whether or not the accuracy index and the normality condition are satisfied for any of the objects identified from "OBJ001" to "OBJ003" and "OBJ005" to "OBJ008", which are the "object IDs" associated with the result information, is the same as in the first embodiment. Furthermore, the identification unit 112 calculates an accuracy index by considering the object having the "object ID" of "OBJ004" as the target object. Specifically, the identification unit 112 extracts "1," which is the "accuracy score" associated with "OBJ004" in the third result information 906. The identification unit 112 also extracts "0.6," which is the "coefficient" associated with the "third detection unit" and the "divided area 803," which is the "divided area" associated with "OBJ004" in the third result information 906, from among the "divided areas" indicated in the area relation information 907. The identification unit 112 then multiplies the extracted "1" by "0.6" to obtain the multiplied value "0.6." Furthermore, the target object has not been detected by the first detection unit 111a or the second detection unit 111b. Therefore, the identification unit 112 determines the multiplied value "0.6" for the third detection unit 111c as the accuracy index for the target object. The accuracy index "0.6" calculated by the identification unit 112 is greater than the threshold value for the normal condition and therefore satisfies the normal condition (YES in S306 in FIG. 4). In this case, the identification unit 112 identifies the type of object and the area in which the object is located from the result information, and generates identification information indicating the identification result (see S307 in FIG. 4).

[0064] Furthermore, the identification unit 112 calculates an accuracy index for each of the object having an "object ID" of "OBJ009" and the object having an "OBJ010." Here, the "coefficient" associated with the "delimited area 801," which is the "delimited area" associated with "OBJ009" and "OBJ010" in the third result information 906 among the "delimited areas" indicated in the area relation information 907, and the "third detection unit" is "0." Therefore, "0" is calculated as the accuracy index for both the object having an "OBJ009" and the object having an "OBJ010." Because this accuracy index does not satisfy the normality condition (NO in S306 of FIG. 4), the identification unit 112 determines that the detection by the third detection unit 111c of the object having an "OBJ009" and the object having an "OBJ010" is an erroneous detection, and does not generate identification information.

[0065] 11(E) is a diagram showing identification information 908. Similar to identification information 506 (see FIG. 7(D)), identification information 908 indicates "object type," "minimum coordinate," "maximum coordinate," "accuracy index," and "object ID." In this embodiment, the method by which the identification unit 112 identifies each item indicated in the identification information is the same as in the first embodiment. In the identification information 908, the "object type" associated with the "object ID" of "OBJ004" indicates "N / A," which means that the type of the object has not been identified. In this case, the identification unit 112 may perform a type identification process. The type identification process is a process in which the identification unit 112 identifies the type of an object that has not been detected by the detection unit 111.

[0066] 12 is a flowchart showing the flow of the type identification process. The type identification process is started when the type of the target object is not identified in step 307 of the identification process (see FIG. 4) and before proceeding to the process of step 308. In other words, the type identification process is performed while the identification process is being executed. In the following, the result information used in the currently executing identification process may be referred to as the current result information. Also, in the following, the result information used in the previously executed identification process may be referred to as the previous result information. The previous result information may also be regarded as result information indicating the detection result for the first image. Also, the current result information may also be regarded as result information indicating the detection result for the second image generated in association with the capture of a later time point than the first image. The identification unit 112 extracts the previous result information from the storage device 130 (S1201).

[0067] The identification unit 112 identifies the type of object not detected by the detection unit 111 by comparing the extracted previous result information with the current result information (S1202). Here, the detection unit 111 may detect object features, such as the color or shape of the object, from the image and calculate the detected features as indices, such as feature amounts. The detection unit 111 may also detect the movement speed of the object from multiple images captured at different times. The indices as object features and the movement speed of the object calculated by the detection unit 111 may then be included in the result information. In this case, the identification unit 112 may compare the indices as object features or the movement speed indicated in the previous result information with the features and the movement speed of the object indicated in the current result information. If the degree of change in the compared indices or the movement speed is equal to or less than a threshold, the identification unit 112 may identify the object indicated in the previous result information and the object indicated in the current result information as the same object. Furthermore, the identification unit 112 may identify the object indicated in the previous result information and the object indicated in the current result information as the same object by object tracking processing such as a Kalman filter. Furthermore, if the type of object is indicated in the previous result information for the identified object, the identification unit 112 may use this type as the type of the target object. Here, the identification unit 112 can also be regarded as a tracking means for tracking an object. Furthermore, the identification unit 112 may use at least one of type information, area information, object movement speed, and object feature amount to track the object. In this way, the type of object previously identified may be carried over. Furthermore, when the type identification process is completed, the process of step 308 of the identification process is started.

[0068] In the present disclosure, the monitoring system 100 is described as being provided with one imaging device 120, but the present disclosure is not limited to this. The monitoring system 100 may be provided with multiple image capturing devices 120. In this case, the monitoring system 100 may detect an object from an image for each image capturing device 120 and identify the object from the detection result. Furthermore, when the monitoring system 100 is provided with multiple image capturing devices 120, the coefficient used to calculate the accuracy index may be determined for each image capturing device 120.

[0069] 13 is a diagram showing area relation information 1000 when two imaging devices 120 are installed in the surveillance system 100. In the following example, it is assumed that the surveillance system 100 is installed with two imaging devices 120, namely, a first imaging device 120 and a second imaging device 120. As shown in FIG. 13, the area relation information 1000 indicates coefficients for each imaging device.

[0070] The "coefficients" shown in the region relation information 1000 will now be described in detail. First, the coefficients used for the detection accuracy of an object detected from an image generated by capturing an image with the "first imaging device" will be described. For the "first detection unit," the "coefficient" associated with the "delimited area 801" is "0.3," the "coefficient" associated with the "delimited area 802" is "0.7," and the "coefficient" associated with the "delimited area 803" is "0.3." For the "second detection unit," the "coefficient" associated with the "delimited area 801" is "0.7," the "coefficient" associated with the "delimited area 802" is "0.3," and the "coefficient" associated with the "delimited area 803" is "0.1." For the "third detection unit," the "coefficients" associated with the "delimited area 801" and "delimited area 802" are both "0," and the "coefficient" associated with the "delimited area 803" is "0.6."

[0071] Next, the coefficients used for the detection accuracy of an object detected from an image generated by capturing an image with the "second imaging device" will be described. For the "first detection unit," the "coefficient" associated with the "delimited area 801" is "0.4," the "coefficient" associated with the "delimited area 802" is "0.6," and the "coefficient" associated with the "delimited area 803" is "0.25." For the "second detection unit," the "coefficient" associated with the "delimited area 801" is "0.6," the "coefficient" associated with the "delimited area 802" is "0.4," and the "coefficient" associated with the "delimited area 803" is "0.15." For the "third detection unit," the "coefficients" associated with the "delimited area 801" and "delimited area 802" are both "0," and the "coefficient" associated with the "delimited area 803" is "0.6."

[0072] As described above, in the present disclosure, the identification unit 112 acquires information about the detection results of the detection unit 111. Information about the detection results includes result information. Therefore, the identification unit 112 can also be regarded as an acquisition unit that acquires information about the detection results. The identification unit 112 then identifies an object captured in the image based on the relationship between the detection results and whether or not the detection results satisfy a predetermined condition for each detection unit 111 from which the result information is acquired. Examples of the predetermined condition for each detection unit 111 include whether or not the partitioned area to which the area in which the object is detected belongs is a "partitioned area" associated with a higher "coefficient" in the area relationship information (see FIGS. 7(C) and 11(D)). Examples of the relationship between the detection results and whether or not the detection results satisfy a predetermined condition for each detection unit 111 include the multiplication of the accuracy score and the coefficient for each detection unit 111. In this case, the identification unit 112 obtains an identification result according to the detection unit 111 used to detect the object. Therefore, by utilizing a plurality of detection results according to the detection unit 111 used to detect the object, it is possible to identify the object shown in the image.

[0073] The predetermined conditions are not limited to the conditions determined for the partitioned area to which the area in which the object is detected by the detection unit 111 belongs. For example, the predetermined condition may be that the size of the object detected by the detection unit 111, in other words, the size of the area detected as the object by the detection unit 111, is equal to or larger than a predetermined size. In this case, the coefficient used to calculate the multiplication value may be set to be larger when the size of the area detected as the object by the detection unit 111 is equal to or larger than the predetermined size than when it is smaller than the predetermined size. Furthermore, the predetermined condition may be that the size of the object detected by detection unit 111, in other words, the size of the area detected as the object by detection unit 111, is equal to or smaller than a predetermined size. In this case, the coefficient used to calculate the multiplication value may be larger when the size of the area detected as the object by detection unit 111 is equal to or smaller than the predetermined size than when it exceeds the predetermined size. Furthermore, in cases where another object exists between the image capture device 120 and the object to be detected, the predetermined condition may be that the detection unit 111 detects or identifies that a part of the object detected is hidden. In this case, the coefficient used in calculating the multiplication value may be set to be larger when the detection unit 111 detects or identifies that a part of the object detected is hidden than when the detection unit 111 does not detect or identify that a part of the object detected is hidden. Note that the detection or identification that a part of the object detected by the detection unit 111 is hidden may be performed by the detection unit 111, or may be performed by an identification means (not shown) that identifies that a part of the object detected by the detection unit 111 is hidden.

[0074] Furthermore, the identification unit 112 identifies an object so that the detection result is more likely to be reflected in the identification result when the detection result satisfies a predetermined condition than when the detection result does not satisfy the predetermined condition. In this case, a specific result can be obtained depending on whether or not the detection result satisfies a predetermined condition.

[0075] Furthermore, the identification unit 112 identifies an object in the image when a value according to the detection result and a coefficient, which is determined depending on whether the detection result satisfies a predetermined condition, is equal to or greater than a predetermined threshold. Here, the coefficient may be a coefficient shown in the area relation information (see FIGS. 7(C) and 11(D)). Furthermore, the value according to the coefficient and the detection result may be a product of the accuracy score and the coefficient. Furthermore, the predetermined threshold may be a threshold determined as a normal condition. A coefficient that is greater when the detection result satisfies the predetermined condition is identified than when it does not. In this case, a specific result can be obtained depending on whether or not the detection result satisfies a predetermined condition.

[0076] Furthermore, at least one of the predetermined conditions for each detection unit 111 is determined for an area of ​​the image where an object is detected by the detection unit 111. In this case, it is possible to identify an object appearing in the image depending on the area in which the object was detected. Note that at least one of the predetermined conditions being defined for an area in an image where an object is detected by the detection unit 111 includes the predetermined condition being defined for a partitioned area to which the area where an object is detected by the detection unit 111 belongs. Also, at least one of the predetermined conditions being defined for an area in an image where an object is detected by the detection unit 111 includes the predetermined condition being defined for the size of the area detected as an object by the detection unit 111. For example, the predetermined condition may be that the size of the object detected by the detection unit 111, in other words, the size of the area detected as the object by the detection unit 111, is equal to or larger than a predetermined size. In this case, the coefficient used to calculate the multiplication value may be set to be larger when the size of the area detected as the object by the detection unit 111 is equal to or larger than the predetermined size than when it is smaller than the predetermined size. Furthermore, the predetermined condition may be that the size of the object detected by the detection unit 111, in other words, the size of the area detected as the object by the detection unit 111, is equal to or smaller than a predetermined size. In this case, it is more likely that the size of the area detected as the object by the detection unit 111 is equal to or smaller than the predetermined size than that it exceeds the predetermined size.

[0077] In addition, in the present disclosure, there is a region condition that is set for a plurality of detection results in which different detection units 111 are used for detection, and that is set for a region in the image in which an object is detected by the detection unit 111. Examples of the region condition include an identical condition. Then, for each of the plurality of detection results that satisfy the region condition, the identification unit 112 identifies whether an object is captured in the image according to the relationship between the detection result and whether the detection result satisfies the condition predetermined for each detection unit 111. In this case, it is possible to prevent the occurrence of a situation in which it is determined whether or not an object is captured in an image based on only one of a plurality of detection results that satisfy the region condition.

[0078] Furthermore, the identification unit 112 acquires information on indices identified from the relationship between the detection results and whether the detection results satisfy a condition predetermined for each detection unit 111. Examples of the information on indices include a multiplication value of an accuracy score and a coefficient indicated in the area relation information. Then, the identification unit 112 identifies that an object is captured in the image when the cumulative indexes related to a plurality of detection results that satisfy the area condition satisfy an index condition defined for the index. Examples of the cumulative indexes include an accuracy index. Examples of the index condition include a normality condition. In this case, it is possible to obtain an object identification result based on all detection results that satisfy the region condition.

[0079] Furthermore, for each detection result that does not satisfy the region condition, the identification unit 112 identifies whether an object is captured in the image depending on the relationship between the detection result and whether the detection result satisfies the condition predetermined for each detection unit 111. In this case, it is possible to prevent a detection result that satisfies the region condition from affecting a specific result corresponding to a detection result that does not satisfy the region condition.

[0080] Furthermore, for each detection result that does not satisfy the region condition, the identification unit 112 identifies whether an object is captured in the image based on whether the index related to the detection result satisfies the index condition. Then, the multiplied value, which is an index identified from the relationship related to one detection result that satisfies the predetermined condition, satisfies the index condition. In this case, even if a single detection result does not satisfy the area condition, if the result satisfies the predetermined condition, it is possible to identify that an object is captured in the image.

[0081] Furthermore, the identification unit 112 identifies the type of object, and the images that are the subject of detection by the detection unit 111 include a first image and a second image that was generated in association with capturing the first image at a later time than the first image. The detection results include a first detection result that is a detection result for the first image and a second detection result that is a detection result for the second image. The first detection result includes the detection result indicated in the previous result information. The second detection result includes the detection result indicated in the current result information. When the presence and type of an object are identified from the first detection result, and the presence of the object is identified from the second detection result without identifying the type of the object, the identification unit 112 regards the type identified from the first detection result as the type of the object whose presence is identified from the second detection result. In this case, even if the type of object cannot be detected from one image (see FIG. 11(C)), the type of object can be identified.

[0082] The detection means used as the first detection unit 111a, the second detection unit 111b, and the third detection unit 111c are not limited to the above examples. The first detection unit 111a may be any one of a detection means using machine learning, a detection means using the difference between multiple images, a detection means using a comparison between an object's feature amount and an object shown in an image, and a detection means using distance information. The second detection unit 111b may be any one of a detection means using machine learning, a detection means using the difference between multiple images, a detection means using a comparison between an object's feature amount and an object shown in an image, and a detection means using distance information, which is different from the first detection unit 111a. The third detection unit 111c may be any one of a detection means using machine learning, a detection means using the difference between multiple images, a detection means using a comparison between an object's feature amount and an object shown in an image, and a detection means using distance information, which is different from the first detection unit 111a and the second detection unit 111b.

[0083] At least some of the functional blocks shown as the configuration of the monitoring system 100 (see FIGS. 1 and 8) may be implemented by hardware, such as a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP). Furthermore, the functional blocks shown as the configuration of the monitoring system 100 (see FIGS. 1 and 8) may be provided in the same device, or may be provided in separate devices connected via a network.

[0084] In addition, in the present disclosure, the same divided area is set regardless of the detection unit 111, but this is not limiting. Different divided areas may be set depending on the detection unit 111. In addition, although the present disclosure has described that the area in which an object is detected is defined in a two-dimensional Cartesian coordinate system, the present disclosure is not limited to this. In cases where a detection means that detects an object from distance information is used, the area in which an object is detected may be defined in a world coordinate system, which is a three-dimensional coordinate system.

[0085] Furthermore, in the present disclosure, the detection target of the detection unit 111 has been described as a moving object or a human being, but the detection target of the detection unit 111 is not limited to this. The detection target of the detection unit 111 may be any object.

[0086] Furthermore, the trigger for starting the output process (see FIG. 3) is not limited to the image capture device 120 transmitting an image to the monitoring server 110. For example, the output process may be started when the monitoring server 110 or the image capture device 120 is powered on.

[0087] Furthermore, in the present disclosure, the result information indicates the "minimum coordinates" and "maximum coordinates" as area information, but is not limited to this. The region information may be, for example, the center coordinates of a rectangular region surrounding an object in an image, and the width and height of this region.

[0088] Furthermore, the "accuracy score" shown in the result information (see FIG. 7(A) and the like) is not limited to the above-mentioned example. For an object detected by the detection unit 111, the "accuracy score" may be "1", and for an object not detected by the detection unit 111, the "accuracy score" may be "0".

[0089] Furthermore, in the present disclosure, the same threshold value is defined as the normal condition regardless of the detection target, but this is not limiting. For example, the threshold value defined as the normal condition may differ for each divided area. To give one example, generally, an object located in a monitoring area far from the image capture device 120 is less likely to be detected by the detection unit 111 than an object located in a monitoring area close to the image capture device 120. Therefore, by setting a lower threshold value as the normal condition for a monitoring area far from the image capture device 120 than for a monitoring area close to the image capture device 120, it may be possible to prevent the detection unit 111 from failing to detect an object located in a monitoring area far from the image capture device 120.

[0090] Also, in the identification process, it has been explained that the divided area to which the area in which the object is detected belongs is identified by the identification unit 112 (see S303 in FIG. 4). Here, the identification unit 112 may identify the divided area to which the area in which the object is detected belongs as the divided area to which the entire detected area belongs, or may identify the divided area that is located most of the detected areas, or may identify the divided area to which the coordinates of the center of gravity of the detected area belong.

[0091] Furthermore, the information included in the specific information is not limited to the above-mentioned examples. For example, area information indicating the area detected by each detection unit 111 for the object identified by the identification unit 112 may be included in the identification information.

[0092] The present invention also includes cases where a software program that realizes the functions of each of the above-mentioned embodiments is supplied to a system or device having a computer that can execute the program directly from a recording medium or via wired / wireless communication, and the program is executed. Therefore, the program code itself, supplied to and installed on a computer to implement the functional processes described above, also embodies the present invention. In other words, the computer program itself for implementing the functional processes of the present invention is also included in the present invention. In this case, the program may take any form, such as object code, a program executed by an interpreter, or script data supplied to an OS, as long as it has the program's functionality. Recording media for providing the program may include, for example, a hard disk, a magnetic recording medium such as a magnetic tape, an optical / magneto-optical storage medium, or a non-volatile semiconductor memory. Another possible method for providing the program is to store the computer program forming the present invention on a server on a computer network, and then download the computer program to a connected client computer. Furthermore, an operating system or the like running on a computer may perform some or all of the actual processing based on the instructions of the program code, and the functions of the above-described embodiments may be realized by this processing. Furthermore, program code read from a storage medium may be written to memory provided on a function expansion board inserted into a computer or a function expansion unit connected to the computer. Then, a CPU or the like provided on the function expansion board or function expansion unit may perform some or all of the actual processing based on the instructions of the program code. Even in this case, the functions of the above-described embodiments are realized.

[0093] The disclosure of this embodiment includes the following configuration. (Configuration 1) an acquisition means for acquiring information about a detection result of a detection means for detecting an object captured in an image generated by the imaging device; an identification means for identifying an object shown in the image in accordance with a relationship between the detection result and whether or not the detection result satisfies a predetermined condition for each of the detection means, the information of which is acquired by the acquisition means; An information processing system comprising: (Configuration 2) 2. The information processing system according to claim 1, wherein the identification means identifies the object so that the detection result is more effectively reflected in the identification result when the detection result satisfies the predetermined condition than when the detection result does not satisfy the predetermined condition. (Configuration 3) the specifying means specifies that an object is captured in the image when a value according to a coefficient specified depending on whether the detection result satisfies the predetermined condition and the detection result is equal to or greater than a predetermined threshold; 3. The information processing system according to configuration 1 or 2, wherein the coefficient is specified to be larger when the detection result satisfies the predetermined condition than when the detection result does not satisfy the predetermined condition. (Configuration 4) 4. The information processing system according to any one of configurations 1 to 3, wherein at least one of the predetermined conditions for each of the detection means is determined for an area of ​​the image in which an object is detected by the detection means. (Configuration 5) The information processing system according to any one of configurations 1 to 4, wherein the identification means is a condition defined for a plurality of detection results in which the detection means used for detection are different, and for each of the plurality of detection results that satisfy a region condition defined for a region in the image in which an object is detected by the detection means, the identification means identifies whether or not an object is captured in the image according to the relationship. (Configuration 6) the acquiring means acquires information about an index identified from the relationship related to the detection result; The information processing system of configuration 5, wherein the identification means identifies that an object is captured in the image when a cumulative total of the indices related to the plurality of detection results that satisfy the area condition satisfies an index condition defined for the indices. (Configuration 7) 6. The information processing system according to claim 5, wherein the specifying unit specifies, for each of the detection results that do not satisfy the region condition, whether or not an object is captured in the image according to the relationship. (Configuration 8) the acquiring means acquires information about an index identified from the relationship related to the detection result; the specifying means specifies, for each of the detection results that do not satisfy the area condition, whether or not an object is captured in the image based on whether or not the index related to the detection result satisfies an index condition defined for the index; 8. The information processing system according to claim 7, wherein an index identified from the relationship related to one of the detection results that satisfies the predetermined condition satisfies the index condition. (Configuration 9) The identification means identifies the type of object, The images to be detected by the detection means include a first image and a second image generated in association with an image taken at a later time than the first image, the detection results include a first detection result that is a detection result for the first image and a second detection result that is a detection result for the second image; The information processing system according to any one of configurations 1 to 8, wherein, when the existence and type of an object are identified from the first detection result, and the existence of an object is identified from the second detection result without identifying the type of the object, the identification means sets the type identified from the first detection result as the type of the object whose existence is identified from the second detection result.

[0094] The present invention has been described in detail above based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications are possible based on the spirit of the present invention, and these modifications are not excluded from the scope of the present invention. Furthermore, a part of the configuration of one of the first and second embodiments may be applied to the other embodiment. [Explanation of symbols]

[0095] 100... monitoring system, 110... monitoring server, 111... detection unit, 112... identification unit, 120... imaging device, 130... storage device

Claims

1. an acquisition means for acquiring information about a detection result of a detection means for detecting an object captured in an image generated by the imaging device; an identification means for identifying an object shown in the image in accordance with a relationship between the detection result and whether or not the detection result satisfies a predetermined condition for each of the detection means, the information of which is acquired by the acquisition means; An information processing system comprising:

2. The information processing system according to claim 1 , wherein the identification means identifies the object so that the detection result is more effectively reflected in the identification result when the detection result satisfies the predetermined condition than when the detection result does not satisfy the predetermined condition.

3. the specifying means specifies that an object is captured in the image when a value according to a coefficient specified depending on whether the detection result satisfies the predetermined condition and the detection result is equal to or greater than a predetermined threshold; The information processing system according to claim 1 , wherein the coefficient is specified to be larger when the detection result satisfies the predetermined condition than when the detection result does not satisfy the predetermined condition.

4. 2. The information processing system according to claim 1, wherein at least one of the predetermined conditions for each of the detecting means is determined for an area of ​​the image in which an object is detected by the detecting means.

5. 2. The information processing system according to claim 1, wherein the identification means is configured to identify whether an object is present in the image for each of the plurality of detection results that satisfy a region condition defined for a region in the image in which an object is detected by the detection means, the region condition being a condition defined for a plurality of detection results that are different from each other using the detection means for detection, and the identification means identifies whether an object is present in the image for each of the plurality of detection results that satisfy a region condition defined for a region in the image in which an object is detected by the detection means, according to the relationship.

6. the acquiring means acquires information about an index identified from the relationship related to the detection result; 6. The information processing system according to claim 5, wherein the identification means identifies that an object is captured in the image when a cumulative total of the indices relating to the plurality of detection results that satisfy the area condition satisfies an index condition defined for the indices.

7. The information processing system according to claim 5 , wherein the specifying unit specifies, for each of the detection results that do not satisfy the region condition, whether or not an object is captured in the image according to the relationship.

8. the acquiring means acquires information about an index identified from the relationship related to the detection result; the specifying means specifies, for each of the detection results that do not satisfy the area condition, whether or not an object is captured in the image based on whether or not the index related to the detection result satisfies an index condition defined for the index; The information processing system according to claim 7 , wherein an index identified from the relationship related to one of the detection results that satisfies the predetermined condition satisfies the index condition.

9. The identification means identifies the type of object, The images to be detected by the detection means include a first image and a second image generated in association with an image taken at a later time than the first image, the detection results include a first detection result that is a detection result for the first image and a second detection result that is a detection result for the second image, 2. The information processing system according to claim 1, wherein, when the presence and type of an object are identified from the first detection result and the presence of an object is identified from the second detection result without identifying the type of the object, the identification means sets the type identified from the first detection result as the type of the object whose presence is identified from the second detection result.

10. acquiring information about the detection result of a detection means for detecting an object captured in an image generated by the imaging device; a method for identifying an object appearing in the image according to a relationship between the detection result and whether or not the detection result satisfies a predetermined condition for each of the detection means from which the information is acquired by the acquisition means; A method comprising:

11. On the computer, a function of acquiring information about the detection result of a detection means that detects an object captured in an image generated by the imaging device; a function of identifying an object appearing in the image in accordance with a relationship between the detection result and whether or not the detection result satisfies a predetermined condition for each of the detection means, the information of which is acquired by the acquisition means; A program to make this happen.

Images