Reporting support system and reporting support method

Abstract

One objective of the present invention is to provide a notification assistance system, etc., capable of assisting in transmitting location information to a notification destination. A notification assistance system according to one aspect of the present disclosure comprises: an acquisition means for acquiring captured images from a communication terminal in the possession of a notifying party; a detection means for detecting a plurality of objects included in the captured images; an identification means for identifying location information corresponding to each of the plurality of detected objects; and an estimation means for estimating the location of the notifying party on the basis of the identified location information.

Description

[Technical Field]

[0001] The present disclosure relates to a technology for supporting a response when a call is made. [Background technology]

[0002] When an accident occurs, people at the scene of the accident will make an emergency call to the police station, fire station, or other appropriate facility depending on the situation. The operator at the control room who receives the emergency call will usually understand the situation based on the caller's voice. For example, the operator will verbally ask the caller for their current location.

[0003] In order to identify the current location of the caller, it is conceivable that the control room at the call destination acquires location information measured from a communication terminal owned by the caller using a positioning system such as a Global Navigation Satellite System (GNSS). For example, Patent Document 1 discloses acquiring location information from a mobile terminal owned by the caller. Specifically, it discloses that the mobile terminal acquires location information using a Global Positioning System (GPS) and automatically transmits the location information to an emergency organization such as the police when an emergency call is made.

[0004] Furthermore, in relation to the acquisition of location information, Patent Document 2 discloses a technique for identifying the location of a mobile device based on an image captured by the mobile device. Specifically, the technique disclosed in Patent Document 2 asks the caller to capture a photo of a road sign and then asks the caller about nearby landmarks. The technique then identifies the caller's location based on the captured image and information about the landmarks provided in response to the question. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-189363 [Patent Document 2] Japanese Patent Application Laid-Open No. 2007-150681 Summary of the Invention [Problem to be solved by the invention]

[0006] When making an emergency call, the operator of the control room at the call destination is required to quickly determine the caller's current location. Here, for example, if the caller's current location is in an unfamiliar place, the caller may not know their current location. Furthermore, the caller may panic when making the emergency call and may not be able to quickly determine their current location.

[0007] The technology disclosed in Patent Document 1 is premised on the use of a positioning system in a communication terminal. Therefore, if the caller has a communication terminal that is not compatible with the positioning system, the method cannot identify the caller's current location. Furthermore, even if the caller has a communication terminal that can use the positioning system, the accuracy of the positioning system may be reduced depending on the caller's location.

[0008] The technology disclosed in Patent Document 2 requires the caller to answer questions. For example, if the caller is in a panic, there is a possibility that the caller will not be able to answer the questions appropriately.

[0009] The present disclosure has been made in consideration of the above-mentioned problems, and one of its objectives is to provide a reporting support system etc. that can support the transmission of location information to a reporting destination. [Means for solving the problem]

[0010] A reporting support system according to one aspect of the present disclosure includes an acquisition means for acquiring a captured image from a communication terminal owned by a reporter, a detection means for detecting multiple objects included in the captured image, an identification means for identifying location information corresponding to each of the detected multiple objects, and an estimation means for estimating the location of the reporter based on each of the identified location information.

[0011] A reporting support method according to one aspect of the present disclosure acquires a captured image from a communication terminal owned by a reporter, detects multiple objects contained in the captured image, identifies location information corresponding to each of the detected objects, and estimates the location of the reporter based on each of the identified location information.

[0012] A computer-readable storage medium according to one aspect of the present disclosure stores a program that causes a computer to execute the following processes: acquiring a captured image from a communication terminal owned by a reporter; detecting multiple objects contained in the captured image; identifying location information corresponding to each of the detected objects; and estimating the location of the reporter based on each of the identified location information. [Effects of the Invention]

[0013] According to the present disclosure, it is possible to assist in transmitting location information to a reporting destination. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating an example of a notification mechanism according to the present disclosure. [Figure 2] 1 is a block diagram illustrating an example of a functional configuration of a reporting support system according to a first embodiment of the present disclosure. [Figure 3] 4 is a flowchart illustrating an example of an operation of the reporting support system according to the first embodiment of the present disclosure. [Figure 4] FIG. 10 is a block diagram illustrating an example of a functional configuration of a reporting support system according to a second embodiment of the present disclosure. [Figure 5] FIG. 10 is a diagram illustrating an example of a captured image according to a second embodiment of the present disclosure. [Figure 6] FIG. 10 is a diagram illustrating an example of a map on which identified position information is superimposed according to the second embodiment of the present disclosure. [Figure 7A] FIG. 10 is a first diagram illustrating an example of the positional relationship between an object and a reporter according to the second embodiment of the present disclosure. [Figure 7B]FIG. 10 is a second diagram illustrating an example of the positional relationship between an object and a reporter according to the second embodiment of the present disclosure. [Figure 8] FIG. 10 is a sequence diagram illustrating an example of an operation of the reporting support system 100 according to the second embodiment of the present disclosure. [Figure 9] FIG. 13 is a diagram showing an example of a shooting screen according to a fourth modification of the present disclosure. [Figure 10] FIG. 10 is a block diagram illustrating an example of a functional configuration of a reporting support system according to a third embodiment of the present disclosure. [Figure 11] FIG. 11 is a diagram illustrating an example of output information according to the third embodiment of the present disclosure. [Figure 12] FIG. 11 is a diagram illustrating another example of output information according to the third embodiment of the present disclosure. [Figure 13] 10 is a flowchart illustrating an example of an operation of the reporting support system according to the third embodiment of the present disclosure. [Figure 14] 1 is a block diagram showing an example of a hardware configuration of a computer device that realizes a reporting support system according to a first, second, and third embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0016] First Embodiment First, the reporting mechanism will be explained. FIG. 1 is a diagram schematically illustrating an example of the reporting mechanism. A communication terminal 20 and a command system 10 are communicably connected via a wired or wireless network. A reporter uses the communication terminal 20 to report by, for example, inputting a predetermined phone number. The communication terminal 20 is, for example, a mobile terminal such as a mobile phone, a smartphone, or a tablet terminal. This example is not limiting, and the communication terminal 20 may also be a personal computer. The communication terminal 20 is a terminal that has at least the functions of reporting and taking pictures.

[0017] The report is connected to a command system 10 in a command room, which is the destination of the report. The command system 10 may be, for example, a server device or a group of devices including a server device. The command room refers to an organization that issues commands to dispatch those who should rush to the scene (for example, police officers, firefighters, and ambulance crews) depending on the content of the report. An operator in the command room communicates with the caller via the command system 10. Then, based on the information obtained from the caller, the operator gives instructions to the caller or uses the command system 10 to issue commands to those who should rush to the scene. An example of the report support system of the present disclosure is used in a situation where such a report is made.

[0018] Next, an overview of the reporting support system of the present disclosure will be described.

[0019] 2 is a block diagram showing an example of the functional configuration of the reporting support system 100. The reporting support system 100 is incorporated into, for example, the command system 10. However, the reporting support system 100 is not limited to this, and may be incorporated into the communication terminal 20, or may be a system realized across the communication terminal 20 and the command system 10. Furthermore, the reporting support system 100 may be realized in a device that is capable of communicating with the communication terminal 20 and is different from the command system 10.

[0020] As shown in FIG. 2, the reporting support system 100 includes an acquisition unit 110, a detection unit 120, an identification unit 130, and an estimation unit 140.

[0021] The acquisition unit 110 acquires an image captured by the communication terminal 20. For example, when a report is made by the communication terminal 20, the communication terminal 20 takes an image in response to an operation by the reporter. At this time, the reporting support system 100 may activate a camera mounted on the communication terminal 20 by transmitting a signal requesting the communication terminal 20 to take an image. Alternatively, the communication terminal 20 may activate the camera by receiving an input from the reporter. The acquisition unit 110 acquires, for example, a captured image generated by capturing an image from the communication terminal 20. In this way, the acquisition unit 110 acquires the captured image from the communication terminal 20 owned by the reporter. The acquisition unit 110 is an example of an acquisition means.

[0022] The detection unit 120 detects multiple objects included in the captured image. For example, the detection unit 120 detects the objects based on feature amounts extracted from the captured image. In this case, for example, a database containing information in which feature amounts and object information are associated is stored in a storage device (not shown) included in the reporting support system 100 or in an external device capable of communicating with the reporting support system 100. The detection unit 120 compares the feature amounts extracted from the captured image with the feature amounts included in the database. If the comparison results in a match, the detection unit 120 may detect the object by identifying the object information associated with the matching feature amount. The object information may be, for example, the name of the object or a code for identifying the object. The object information may be any information that can identify the object.

[0023] Note that various methods can be applied as an object detection method. For example, the object detection method may be a method based on various machine learning models such as deep learning. For example, the detection unit 120 extracts candidate areas including objects from the captured image and calculates feature amounts for each candidate area using a convolutional neural network (CNN) or the like. The detection unit 120 may then detect an object included in each candidate area by using a classifier such as a support vector machine (SVM) on the calculated feature amounts. Any method may be used for object detection as long as it is capable of identifying object information of an object included in a captured image.

[0024] In this way, the detection unit 120 detects a plurality of objects included in the captured image. The detection unit 120 is an example of a detection means.

[0025] The identification unit 130 identifies location information corresponding to the detected object. For example, the identification unit 130 identifies the location information based on object information related to the detected object. More specifically, a database containing information in which object information and location information are associated is stored in advance in a storage device (not shown) included in the reporting support system 100 or in an external device capable of communicating with the reporting support system 100. The identification unit 130 may identify location information associated with object information similar to the object information related to the detected object from the database. Note that various methods can be applied as a method for identifying the location information. For example, the identification unit 130 may identify the location information by searching for object information using a search engine.

[0026] In this way, the identification unit 130 identifies the position information corresponding to each of the detected objects. The identification unit 130 is an example of an identification means.

[0027] The estimation unit 140 estimates the position of the reporter. Specifically, the estimation unit 140 estimates the position of the reporter based on each piece of identified position information, i.e., the position information of each detected object. For example, the estimation unit 140 may estimate any of the identified position information as the position of the reporter. Furthermore, the estimation unit 140 may, for example, exclude, from the identified position information, position information that is separated from other position information by a predetermined distance or more, i.e., position information that is an outlier. Then, the estimation unit 140 may estimate any of the identified position information other than the excluded position information as the position of the reporter. Furthermore, the estimation unit 140 may, for example, estimate the position of the reporter based on the positional relationship of the detected object in the captured image and the identified position information. For example, the estimation unit 140 may estimate the position of the reporter from the position information of an object that appears in the foreground of a captured image among multiple objects, based on a depth map generated from the captured image.

[0028] In this way, the estimation unit 140 estimates the location of the reporter based on each piece of identified location information. The estimation unit 140 is an example of an estimation means.

[0029] Next, an example of the operation of the reporting support system 100 will be described with reference to Fig. 3. In this disclosure, each step in the flowchart will be represented by a number assigned to each step, such as "S1".

[0030] 3 is a flowchart illustrating an example of the operation of the reporting support system 100. The acquisition unit 110 acquires a captured image from the communication terminal 20 owned by the reporter (S1). The detection unit 120 detects a plurality of objects included in the captured image (S2). The identification unit 130 identifies location information corresponding to each of the detected plurality of objects (S3). The estimation unit 140 estimates the location of the reporter based on each of the identified location information (S4).

[0031] As described above, the reporting support system 100 of the first embodiment acquires a captured image from the communication terminal 20 owned by the reporter and detects multiple objects included in the captured image. The reporting support system 100 then identifies location information corresponding to each of the detected objects and estimates the location of the reporter based on the identified location information. This allows the reporting support system 100 to provide the location of the reporter estimated from the image captured by the communication terminal 20. This allows the operator to quickly grasp the current location of the reporter. Furthermore, since the reporting support system 100 estimates the location of the reporter from the image captured by the communication terminal 20, it is possible to estimate the location of the reporter even when the communication terminal 20 does not have a positioning system that can measure its own location. Furthermore, the reporting support system 100 does not necessarily require the reporter to answer questions for estimating the location of the reporter. As described above, the reporting support system 100 of the first embodiment can support the transmission of location information to the reporting destination.

[0032] <Second embodiment> Next, a reporting support system according to a second embodiment will be described. In the second embodiment, the reporting support system 100 described in the first embodiment will be described in more detail. Note that some of the content that overlaps with the content described in the first embodiment will not be described again.

[0033] Fig. 4 is a block diagram showing an example of the functional configuration of the reporting support system 100. As shown in Fig. 4, for example, the reporting support system 100 is provided in the command system 10. As described above, the configuration of the reporting support system 100 is not limited to this example. In the present disclosure, an example in which the reporting support system 100 is provided in the command system 10 will mainly be described.

[0034] The reporting support system 100 includes an acquisition unit 110, a detection unit 120, a specification unit 130, and an estimation unit 140. The communication terminal 20 also includes an image capture unit 210 and a request reception unit 220.

[0035] The acquisition unit 110 includes an image capture control unit 1101 and a captured image acquisition unit 1102. The image capture control unit 1101 causes the communication terminal 20 to capture an image in response to a notification. For example, when the communication terminal 20 makes a notification, the image capture control unit 1101 detects the notification. Then, the image capture control unit 1101 transmits an image capture request. When the request receiving unit 220 of the communication terminal 20 receives the image capture request, the image capture unit 210 of the communication terminal 20 starts capturing an image. The image capture unit 210 captures an image and generates a captured image. In other words, the image capture unit 210 has the function of a camera mounted on the communication terminal 20. In this way, the image capture control unit 1101 causes the communication terminal 20 to start capturing an image in response to detecting a notification from the communication terminal 20. The image capture control unit 1101 is an example of an image capture control means.

[0036] The captured image acquisition unit 1102 acquires captured images from the communication terminal 20. For example, in response to a capture request from the capture control unit 1101, the capture unit 210 captures images. The capture unit 21 transmits the generated captured images to the reporting support system 100. The captured image acquisition unit 1102 acquires the captured images transmitted by the capture unit 21. In this way, the captured image acquisition unit 1102 acquires the captured images captured by the capture started by the capture control unit 1101 from the communication terminal 20. The captured image acquisition unit 1102 is an example of a captured image acquisition means.

[0037] Here, various methods are conceivable for the captured image acquisition unit 1102 to acquire the captured image. For example, suppose that the communication terminal 20 makes a report via a telephone line. In other words, suppose that the report is made via a line that cannot transmit captured images. The image capture control unit 1101 sends a message using, for example, the telephone number of the communication terminal 20. At this time, the image capture control unit 1101 may send the message using SMS (Short Message Service). The message may include, for example, a URL (Uniform Resource Locator). When the URL of the message is opened on the communication terminal 20, data communication between the communication terminal 20 and the reporting support system 100 (or the command system 10) is initiated. The image capture control unit 1101 may, for example, transmit such a message as an image capture request. The request receiving unit 220 receives such a message as an image capture request. Then, the URL included in the message is opened by an operation of the caller, thereby starting data communication. This data communication may be realized, for example, by WebRTC (Web Real Time Communication). That is, the caller may open a URL included in the message, which may initiate data communication via a browser between the call support system 100 (or the command system 10) and the communication terminal 20. The image capturing unit 210 transmits the captured image using the data communication thus initiated. The captured image acquiring unit 1102 acquires the captured image using the data communication.

[0038] Also, for example, suppose that a report from communication terminal 20 is made using data communication rather than a telephone line. That is, suppose that the report is made using data communication that allows a captured image to be transmitted. In this case, upon detecting the report, photographing control unit 1101 transmits a photographing request to communication terminal 20. At this time, the photographing request may be a signal that controls photographing unit 210 of communication terminal 20. That is, photographing control unit 1101 may activate the camera function of communication terminal 20. Then, photographing unit 210 transmits the photographed image to reporting support system 100 using the data communication. Photographed image acquisition unit 1102 acquires the photographed image using the data communication.

[0039] The detection unit 120 detects multiple objects from the acquired captured image. The objects are landmarks whose locations can be identified. For example, the objects may be signs and labels, or landmarks such as parking lots, stores, distinctive buildings, statues, and monuments. The objects may also be signs, utility poles, manholes, vending machines, and signs showing the control numbers attached to these.

[0040] Fig. 5 is a diagram showing an example of a captured image. In the example of Fig. 5, a group of buildings, a river, a train, an overpass, etc. are captured in the captured image. The detection unit 120 detects, for example, a tower, a river billboard, an advertising billboard, and a bronze statue from the captured image. That is, the detection unit 120 identifies object information of the tower, the river billboard, the advertising billboard, and the bronze statue.

[0041] The identification unit 130 identifies location information of objects detected by the detection unit 120. In the example of FIG. 5, it is assumed that the detection unit 120 detects a tower, a river sign, an advertising billboard, and a bronze statue. In this case, the identification unit 130 identifies the location information of each of the tower, the river sign, the advertising billboard, and the bronze statue. For example, the identification unit 130 identifies the location information associated with the object information corresponding to each of the tower, the river sign, the advertising billboard, and the bronze statue from a database containing information in which object information and location information are associated. Here, the identification unit 130 may output the identified location information by superimposing it on a map. FIG. 6 is a diagram showing an example of a map on which the identified location information is superimposed. More specifically, FIG. 6 shows a diagram in which the location information of objects detected from the captured image of FIG. 5 is superimposed on a map. In the example of FIG. 6, the locations of the detected objects are indicated by dots. In addition, in FIG. 6, the hatched area indicates a river, and the striped line indicates a railroad track.

[0042] The estimation unit 140 estimates the location of the reporter based on the identified location information. For example, in the example of FIG. 5, the detection unit 120 detects a tower, a river sign, an advertising billboard, and a bronze statue, and the identification unit 130 identifies the location information of the tower, the river sign, the advertising billboard, and the bronze statue. In this case, the estimation unit 140 estimates the location of the reporter based on the location information of the tower, the river sign, the advertising billboard, and the bronze statue. For example, the estimation unit 140 may estimate a location indicating the vicinity of any one of the tower, the river sign, the advertising billboard, and the bronze statue as the location of the reporter.

[0043] Furthermore, the estimation unit 140 may estimate the position of the reporter based on the positional relationship of the position information of the detected objects and the positional relationship of the detected objects in the captured image. For example, in the example of FIG. 5, the detected objects appear in the captured image in the following order from the left side: a tower, a river sign, an advertising billboard, and a bronze statue. That is, the reporter is located in a position where the tower, the river sign, the advertising billboard, and the bronze statue are visible from the left side as viewed from the front. An example of a method for estimating the position of the reporter in this case will be described with reference to FIGS. 7A and 7B. FIG. 7A is a first diagram illustrating an example of the positional relationship between the objects and the reporter. FIG. 7B is a second diagram illustrating an example of the positional relationship between the objects and the reporter. As shown in FIGS. 7A and 7B, each detected object is represented by a dot on a map. FIG. 7A shows lines connecting point A and each detected object. Assume that the reporter is at point A. In this case, as shown by the lines, the objects should appear in the captured image in the following order from the left side as viewed from the front. Therefore, the estimation unit 140 can estimate that there is no reporter near point A. Meanwhile, FIG. 7B shows line segments connecting point B and each of the detected objects. Assuming that the reporter is at point B, the captured image should show the following objects in order from left to right: a tower, a river sign, an advertising billboard, and a bronze statue. Therefore, the estimation unit 140 can estimate that there is a high possibility that the reporter is near point B. In this way, the estimation unit 140 can estimate the range within which the captured image acquired by the acquisition unit 110 can be captured. Then, the estimation unit 140 may, for example, estimate the estimated range as the location of the reporter. Furthermore, the estimation unit 140 may identify the object closest to the estimated range among the detected objects. Then, the estimation unit 140 may estimate the location of the reporter as a range within the estimated range that is within a predetermined distance from the identified object. In this way, the estimation unit 140 may estimate the position of the reporter based on the positional relationship of the identified positional information and the positional relationship of the detected plurality of objects on the captured image.

[0044] The method of estimating the position of the caller is not limited to this example. For example, the estimation unit 140 may extract information about the distance from the communication terminal 20 to each of a plurality of detected objects from the captured image. At this time, for example, the estimation unit 140 generates a depth map from the captured image. The depth map is information about the distance from the camera to the object, corresponding to each pixel of the image. Then, the estimation unit 140 uses the depth map to obtain the distance from the communication terminal 20 to each of the detected objects. In this way, the estimation unit 140 may extract information about the distance from the captured image.

[0045] For example, the estimation unit 140 may estimate, as the position of the reporter, the position indicated by the position information of an object that is the shortest distance from the communication terminal 20 among the detected objects. Alternatively, the estimation unit 140 may estimate, as the position of the reporter, a predetermined range that includes the position indicated by the position information of the object that is the shortest distance from the communication terminal 20 among the detected objects. In this way, the estimation unit 140 may estimate the position of the reporter based on the positional relationship of the specified position information and information on the distance from the communication terminal 20 to each of the multiple objects, which is extracted from an image captured by the communication terminal 20.

[0046] Note that various methods may be used to generate the depth map. For example, assume that the communication terminal 20 has a multi-lens camera and the imaging unit 210 takes images using the multi-lens camera. At this time, the captured image acquisition unit 1102 acquires multiple captured images taken by the multi-lens camera. The estimation unit 140 may, for example, generate a depth map from the multiple acquired captured images. Furthermore, the depth map may be generated using machine learning. For example, the relationship between the captured images and the depth maps corresponding to the captured images (i.e., ground truth data) may be learned in advance. The estimation unit 140 may generate a depth map from the acquired captured images using the model learned in this way.

[0047] [Example of operation of the reporting support system 100] Next, an example of the operation of the reporting support system 100 will be described with reference to FIG.

[0048] 8 is a sequence diagram illustrating an example of the operation of the reporting support system 100. First, the photographing control unit 1101 detects a report (S101). Then, the photographing control unit 1101 transmits a photographing request to the communication terminal 20 (S102). Then, the request receiving unit 220 receives the photographing request (S103). In response to the photographing request received by the request receiving unit 220, the photographing unit 210 takes a photograph (S104). The photographing unit 210 transmits the photographed image to the reporting support system 100 (S105).

[0049] The captured image acquisition unit 1102 acquires a captured image (S106). The detection unit 120 detects multiple objects from the acquired captured image (S107). The identification unit 130 identifies position information of the multiple detected objects (S108). Then, the estimation unit 140 estimates the position of the reporter based on the identified position information (S109).

[0050] As described above, the reporting support system 100 of the second embodiment acquires a captured image from the communication terminal 20 owned by the reporter and detects multiple objects included in the captured image. The reporting support system 100 then identifies location information corresponding to each of the detected objects and estimates the location of the reporter based on the identified location information. This allows the reporting support system 100 to provide the location of the reporter estimated from the image captured by the communication terminal 20. Therefore, the operator can quickly grasp the current location of the reporter. Furthermore, since the reporting support system 100 estimates the location of the reporter from the image captured by the communication terminal 20, it is possible to estimate the location of the reporter even when the communication terminal 20 does not have a positioning system that can measure its own location. Furthermore, the reporting support system 100 does not necessarily require the reporter to answer questions for estimating the location of the reporter. As described above, the reporting support system 100 of the second embodiment can support the transmission of location information to the reporting destination.

[0051] Furthermore, the reporting support system 100 of the second embodiment may estimate the position of the reporter based on the positional relationship of the specified position information and the positional relationship of the detected plurality of objects on the captured image. Furthermore, the reporting support system 100 may estimate the position of the reporter based on the positional relationship of the specified position information and information on the distance from the communication terminal 20 to each of the plurality of objects, which is extracted from the captured image captured by the communication terminal 20. This allows the reporting support system 100 to improve the accuracy of estimating the position of the reporter.

[0052] [Variation 1] The functional configuration of the communication terminal 20 may be included in the reporting support system 100. That is, the reporting support system 100 may include an image capturing unit 210 and a request receiving unit 220.

[0053] [Variation 2] The reporting support system 100 may be provided in the communication terminal 20. That is, the acquisition unit 110, the detection unit 120, the identification unit 130, and the estimation unit 140 may be provided in the communication terminal 20. In this case, the photographing control unit 1101 may detect that the communication terminal 20 has made a report, and may cause the photographing unit 210 to start photographing in response to the start of data communication between the communication terminal 20 and the command system 10. The photographed image acquisition unit 1102 may acquire the photographed image taken by the photographing unit 210. Then, the estimation unit 140 may transmit information indicating the estimated position of the reporter to the command system 10.

[0054] [Variation 3] When there are multiple candidate objects to be detected from the same region of the captured image, the detection unit 120 may detect the multiple candidates. For example, the detection unit 120 performs matching on the captured image to detect objects. In this case, in the example of FIG. 5, it is determined that the matching matches multiple types of objects in the tower region on the captured image. In this case, the detection unit 120 may detect multiple types of objects in the tower region. The identification unit 130 then identifies location information for each of the multiple types of objects. The estimation unit 140 may then estimate the location of the reporter for each of the multiple types of detected objects. For example, it is assumed that objects "Tower X" and "Tower Y" are detected in the tower region of FIG. 5. In this case, the estimation unit 140 may estimate the location of the reporter when the tower region on the captured image is "Tower X" and the location of the reporter when the tower region on the captured image is "Tower Y."

[0055] [Variation 4] The photographing control unit 1101 may superimpose various information on the photographing screen of the communication terminal 20. The photographing screen is, for example, a screen that appears on a display or the like provided in the communication terminal 20 when the communication terminal 20 photographs. The photographing control unit 1101 may, for example, display information on the photographing screen that prompts the user to photograph a specific object. FIG. 9 is a diagram showing an example of the photographing screen. As shown in FIG. 9, the photographing control unit 1101 may, for example, display a message on the photographing screen saying, "Please include a signboard in the photograph." This allows the photographing control unit 1101 to prompt the caller to include a signboard in the photograph. Note that the object to be photographed does not have to be a signboard. For example, it is desirable for the photographing control unit 1101 to prompt the caller to photograph an object that is easy to detect by the detection unit 120. In this way, the photographing control unit 1101 may superimpose information indicating an object recommended as a photographing target on the photographing screen of the communication terminal 20.

[0056] [Variation 5] If the direction information when the communication terminal 20 took the photograph can be acquired, the estimation unit 140 may further use the direction information to estimate the position of the reporter.

[0057] The direction information is information indicating the direction in which the communication terminal 20 was facing when the communication terminal 20 captured an image. For example, the communication terminal 20 is equipped with a sensor capable of measuring orientation, such as a magnetic sensor and a gyro sensor. In this case, the photographing unit 210 generates a photographed image and acquires direction information indicating the orientation at the time of photographing. The photographing unit 210 then transmits the photographed image and the direction information to the reporting support system 100. The photographed image acquisition unit 1102 of the acquisition unit 110 acquires the direction information including the orientation at the time of photographing.

[0058] The estimation unit 140 estimates the position of the reporter using the direction information. For example, assume that the captured image of FIG. 5 is acquired and the position information is shown as in FIG. 6. Here, assume that the direction information indicates northeast. This indicates that the reporter took the image while facing northeast. For example, if it is assumed that the reporter took the image at point A shown in FIG. 7A, the reporter needs to face east-southeast when taking the image. Therefore, the estimation unit 140 can estimate that the reporter is not near point A. On the other hand, if the reporter took the image from point B shown in FIG. 7B, the reporter needs to face northeast when taking the image. Therefore, the estimation unit 140 can estimate that there is a high possibility that the reporter is near point B.

[0059] In this way, the reporting support system 100 may acquire direction information indicating the direction in which the communication terminal 20 was facing when the communication terminal 20 captured the captured image, and may further use the direction information to estimate the position of the reporter, thereby enabling the reporting support system 100 to more accurately estimate the position of the reporter.

[0060] <Third embodiment> Next, a reporting support system according to a third embodiment will be described. In the third embodiment, an example of further functions of the reporting support system will be described. Note that some of the contents overlapping with those described in the first and second embodiments will not be described.

[0061] [Details of Reporting Support System 101] FIG. 10 is a block diagram showing an example of the functional configuration of the reporting support system 101. The reporting support system 101 may be provided in the command system 10 instead of the reporting support system 100 shown in FIG. 4. In this embodiment, an example will be described in which the command system 10 and the communication terminal 20 are capable of communicating with each other as shown in FIG. 4, and the reporting support system 101 is provided in the command system 10. Note that the reporting support system 101 may be incorporated in the communication terminal 20, similar to the reporting support system 100, or may be a system realized across the communication terminal 20 and the command system 10. Furthermore, the reporting support system 101 may be realized in a device different from the command system 10, which is capable of communicating with the communication terminal 20. The reporting support system 101 performs, for example, the processing described below in addition to the processing of the reporting support system 100.

[0062] As shown in FIG. 10, the reporting support system 101 includes an acquisition unit 110, a detection unit 120, an identification unit 130, an estimation unit 140, and an output control unit 150.

[0063] The output control unit 150 outputs various types of information. The output control unit 150 outputs various types of information to a display device such as a display that can be viewed by an operator in a command room. The display device may be provided in the command system 10, or may be provided in a personal computer, smartphone, tablet, or the like that is communicatively connected to the command system 10 or the reporting support system 101. The output control unit 150 may also output various types of information to a display device provided in the communication terminal 20.

[0064] The output control unit 150 outputs, for example, information indicating the location of the reporter estimated by the estimation unit 140. At this time, the output control unit 150 may output output information indicating the estimated location of the reporter on a map. FIG. 11 is a diagram showing an example of the output information. In the example of FIG. 11, a star-shaped mark indicating the estimated location of the reporter is superimposed on the map. Furthermore, the output information displays an address such as "XX, A City, Kanagawa Prefecture" as information indicating the location of the reporter.

[0065] Furthermore, the output control unit 150 may output position information of a plurality of detected objects. In this case, the output control unit 150 may output output information indicating the locations indicated by the position information of the plurality of objects on a map. In the example of FIG. 11, dots are superimposed on the locations indicating the locations of the detected objects. Furthermore, the output control unit 150 may output output information that associates each of the detected objects on the captured image with the location of the object indicated on the map. For example, FIG. 11 shows a line segment connecting a tower on the captured image with the location indicating the tower on the output information (i.e., on the map).

[0066] In this way, the output control unit 150 outputs to the display device output information showing on a map the estimated position of the reporter and the point indicated by the position information. Furthermore, the output control unit 150 may output to the display device information in which each of the multiple objects in the captured image is associated with each of the points indicated by the position information in the output information. The output control unit 150 is an example of an output control means.

[0067] Furthermore, if the location of the reporter cannot be uniquely estimated, the output control unit 150 may output output information including information prompting the reporter to take another photograph to the display device. A case in which the location of the reporter cannot be uniquely estimated refers to a case in which multiple locations of the reporter are estimated or the location of the reporter cannot be estimated. For example, assume that multiple locations of the reporter are estimated by the estimation unit 140. In this case, the output control unit 150 outputs information prompting the reporter to take another photograph to the display device. At this time, the output control unit 150 can display information prompting the reporter to take another photograph on a display visible to the operator in the control room, thereby causing the operator to request the reporter to take an photograph. Furthermore, the output control unit 150 can prompt the reporter to take a photograph by outputting information prompting the reporter to take another photograph to the communication terminal 20. FIG. 12 is a diagram illustrating another example of output information. More specifically, FIG. 12 is an example of output information displayed on a display visible to the operator, including information prompting the reporter to take a photograph. The example of FIG. 12 illustrates a captured image and four types of estimated candidate locations of the reporter. 12, the message "Please request shooting from a different direction" is displayed. In this way, the output control unit 150 may output information to the display device that prompts shooting from a different direction. By visually checking this information, the operator can instruct the caller to shoot again.

[0068] [Example of operation of Reporting Support System 101] Next, an example of the operation of the reporting support system 101 will be described with reference to FIG.

[0069] Fig. 13 is a sequence diagram illustrating an example of the operation of the reporting support system 101. Note that the processes of S201 to S209 are similar to the processes of S101 to S109 in Fig. 8, and therefore description thereof will be omitted. The output control unit 150 outputs output information based on the estimated position of the reporter (S210). Here, when the position of the reporter is uniquely estimated, the output control unit 150 outputs output information, for example, in which the estimated position of the reporter and the point indicated by the position information are displayed on a map.

[0070] Furthermore, if the position of the reporter cannot be uniquely estimated, the output control unit 150 outputs, for example, output information including information prompting the user to take another photograph to the display device. At this time, the photographing control unit 1101 may control the communication terminal 20 to start taking photographs again.

[0071] In this way, the reporting support system 101 of the third embodiment outputs to a display device output information showing on a map the estimated position of the reporter and the point indicated by the position information, and information in which each of the multiple objects in the captured image is associated with each of the points indicated by the position information in the output information. This allows the reporting support system 101 to allow, for example, an operator in a control room to grasp the estimated position of the reporter. Furthermore, because the operator can grasp the information in which each of the multiple objects in the captured image is associated with each of the points indicated by the position information in the output information, the reporting support system 101 can more easily grasp the position of the reporter.

[0072] Furthermore, when the position of the reporter cannot be uniquely estimated, the reporting support system 101 of the third embodiment may output information to the display device that prompts the reporter to take another photograph. This allows the reporting support system 101 to, for example, have an operator request the reporter to take a photograph. Furthermore, when information prompting the reporter to take another photograph is output to the communication terminal 20, the reporting support system 101 can directly prompt the reporter to take another photograph. Furthermore, at this time, the reporting support system 101 may output information prompting the reporter to take a photograph from a different direction to the display device. This allows the reporting support system 101 to re-estimate the position of the reporter from an image captured from a different direction.

[0073] [Variation 6] The reporting support system 101 may estimate the position of the reporter by using further information. For example, the photographing unit 210 of the communication terminal 20 collects background sound when photographing. The background sound is sound occurring around the communication terminal 20 when photographing. The photographing unit 210 transmits the photographed image and the background sound to the reporting support system 101. The photographed image acquisition unit 1102 of the acquisition unit 110 acquires the photographed image and the background sound.

[0074] Then, the estimation unit 140 further uses the background sound to estimate the location of the reporter. For example, it is assumed that the estimation unit 140 has estimated multiple locations as candidates for the location of the reporter. Here, if the captured image shown in FIG. 5 has been captured, the background sound includes the sound of a running train. The estimation unit 140 estimates, for example, a location with a railroad track nearby from among the multiple estimated locations as the location of the reporter.

[0075] In this way, the reporting support system 101 may acquire background sound generated when the communication terminal 20 captures an image, and further use the background sound to estimate the location of the reporter. This allows the reporting support system 101 to more accurately estimate the location of the reporter.

[0076] <Example of hardware configuration for reporting support system> The hardware constituting the reporting support systems of the first, second, and third embodiments described above will now be described. Fig. 14 is a block diagram showing an example of the hardware configuration of a computer device that realizes the reporting support systems of the embodiments. The reporting support systems and reporting support methods described in the embodiments and modifications are realized in a computer device 90.

[0077] 14, a computer device 90 includes a processor 91, a RAM (Random Access Memory) 92, a ROM (Read Only Memory) 93, a storage device 94, an input / output interface 95, a bus 96, and a drive device 97. The reporting support system may be realized by a plurality of electric circuits.

[0078] The storage device 94 stores a program (computer program) 98. The processor 91 executes the program 98 of the reporting support system using the RAM 92. Specifically, for example, the program 98 includes a program that causes a computer to execute the processes shown in FIGS. 3, 8, and 13. The processor 91 executes the program 98 to realize the functions of the components of the reporting support system. The program 98 may be stored in the ROM 93. The program 98 may also be recorded in the storage medium 80 and read out using the drive device 97, or may be transmitted to the computer device 90 from an external device (not shown) via a network (not shown).

[0079] The input / output interface 95 exchanges data with peripheral devices (such as a keyboard, a mouse, and a display device) 99. The input / output interface 95 functions as a means for acquiring or outputting data. The bus 96 connects the components.

[0080] There are various variations in the method for realizing the reporting support system. For example, the reporting support system can be realized as a dedicated device. Furthermore, the reporting support system can be realized based on a combination of multiple devices.

[0081] The scope of each embodiment also includes a processing method for recording a program for realizing each component of the function of each embodiment on a storage medium, reading the program recorded on the storage medium as code, and executing it on a computer. That is, a computer-readable storage medium is also included in the scope of each embodiment. Furthermore, the storage medium on which the above-mentioned program is recorded and the program itself are also included in each embodiment.

[0082] The storage medium may be, but is not limited to, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD (Compact Disc)-ROM, a magnetic tape, a non-volatile memory card, or a ROM. The programs recorded on the storage medium are not limited to standalone programs that execute processes, but also include programs that run on an OS (Operating System) in cooperation with other software and functions of an expansion board.

[0083] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0084] The above-described embodiments and modifications can be combined as appropriate.

[0085] Some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0086] <Additional Notes> [Appendix 1] an acquisition means for acquiring a captured image from a communication terminal owned by a reporter; a detection means for detecting a plurality of objects included in the captured image; an identification means for identifying position information corresponding to each of the plurality of detected objects; and an estimation means for estimating the location of the reporter based on each of the identified location information. Reporting support system.

[0087] [Appendix 2] the estimation means estimates the position of the reporter based on the positional relationship of the specified position information and the positional relationship of the detected plurality of objects on the captured image. 1. A reporting support system as described in Appendix 1.

[0088] [Appendix 3] the acquiring means acquires direction information indicating a direction in which the communication terminal was facing when the communication terminal captured the photographed image; the estimation means further utilizes the direction information to estimate the position of the caller; 2. A reporting support system as described in Appendix 2.

[0089] [Appendix 4] the estimation means estimates the position of the reporter based on the positional relationship of the specified position information and information on the distance from the communication terminal to each of the plurality of objects, the information being extracted from the captured image taken by the communication terminal; A reporting support system according to any one of appendices 1 to 3.

[0090] [Appendix 5] The apparatus further includes an output control means for outputting to a display device output information showing on a map the estimated position of the reporter and the points indicated by the position information, and information in which each of the plurality of objects on the captured image is associated with each of the points indicated by the position information in the output information. A reporting support system according to any one of appendices 1 to 4.

[0091] [Appendix 6] When the position of the caller cannot be uniquely estimated, the output control means outputs information to a display device prompting the caller to take another photograph. Attachment 5: A reporting support system.

[0092] [Appendix 7] the output control means outputs information to the display device to prompt the user to take pictures from different directions; 6. A reporting assistance system as described in Appendix 6.

[0093] [Appendix 8] the acquiring means acquires background sound when the communication terminal captures the photographed image, the estimation means further utilizes the background sound to estimate the location of the caller. A reporting support system according to any one of appendices 1 to 7.

[0094] [Appendix 9] The acquisition means an image capturing control means for causing the communication terminal to start capturing an image in response to detection of a notification from the communication terminal; and a captured image acquisition means for acquiring the captured image from the communication terminal. A reporting support system according to any one of appendices 1 to 8.

[0095] [Appendix 10] the photographing control means superimposes information indicating an object recommended as a photographing subject on the photographing screen of the communication terminal; 9. A reporting support system as described in Appendix 9.

[0096] [Appendix 11] The captured image is acquired from the communication device owned by the reporter, Detecting a plurality of objects included in the captured image; Identifying position information corresponding to each of the plurality of detected objects; Estimating the location of the reporter based on each of the identified location information. How to support reporting.

[0097] [Appendix 12] A process of acquiring a captured image from a communication terminal owned by the reporter; A process of detecting a plurality of objects included in the captured image; a process of identifying position information corresponding to each of the plurality of detected objects; and a process of estimating the location of the reporter based on each of the identified location information. A computer-readable storage medium. [Explanation of symbols]

[0098] 10 Command System 20. Communication terminals 100, 101 reporting support system 110 Acquisition Department 120 Detector 130 Specific section 140 Estimation part 150 Output control section 1101 Imaging control unit 1102 Image acquisition unit 210 Photography Department 220 Request Receiver

Claims

1. an acquisition means for acquiring a captured image from a communication terminal owned by a reporter; a detection means for detecting a plurality of objects included in the captured image; an identification means for identifying position information corresponding to each of the plurality of detected objects; an estimation means for estimating the location of the reporter based on each of the identified location information, the estimation means estimates the position of the reporter based on the positional relationship of the specified position information and information on the distance from the communication terminal to each of the plurality of objects, the information being extracted from the captured image taken by the communication terminal. Reporting support system.

2. the estimation means estimates the position of the reporter based on the positional relationship of the specified position information and the positional relationship of the detected plurality of objects on the captured image. The reporting support system according to claim 1 .

3. the acquiring means acquires direction information indicating a direction in which the communication terminal was facing when the communication terminal captured the photographed image; the estimation means further utilizes the direction information to estimate the position of the caller; The reporting support system according to claim 2 .

4. The apparatus further includes an output control means for outputting to a display device output information showing on a map the estimated position of the reporter and the location indicated by the location information, and information in which each of the plurality of objects on the captured image is associated with each of the locations indicated by the location information in the output information. The reporting support system according to any one of claims 1 to 3.

5. When the position of the caller cannot be uniquely estimated, the output control means outputs information to a display device prompting the caller to take another photograph. The reporting support system according to claim 4.

6. the output control means outputs information to the display device to prompt the user to take pictures from different directions; The reporting support system according to claim 5 .

7. the acquiring means acquires background sound when the communication terminal captures the photographed image, the estimation means further utilizes the background sound to estimate the location of the caller. The notification support system according to any one of claims 1 to 6.

8. The acquisition means an image capturing control means for causing the communication terminal to start capturing an image in response to detection of a notification from the communication terminal; and a captured image acquisition means for acquiring the captured image from the communication terminal. The notification support system according to any one of claims 1 to 7.

9. The captured image is acquired from the communication device owned by the reporter, Detecting a plurality of objects included in the captured image; Identifying position information corresponding to each of the plurality of detected objects; Estimating the location of the reporter based on each of the identified location information; In the estimating process, the position of the reporter is estimated based on the positional relationship of the specified position information and information on the distance from the communication terminal to each of the plurality of objects, the information being extracted from the captured image taken by the communication terminal. How to support reporting.

Images