Information processing system and its control method, program

Abstract

To improve the efficiency of recording information related to specific objects. [Solution] The system identifies a moving object that has moved within a predetermined range including specified location information, and controls the system to play back a video captured by an imaging device attached to the identified moving object. The system then controls the system to extract and display an image containing the object from the video.

Description

Technical Field

[0001] The present invention relates to an information processing system, its control method, and a program.

Background Art

[0002] When some case (such as a road depression or illegal dumping) occurs within a specific area, the local government in that area needs to handle them. In that case, the situation must be recorded by methods such as going to the site and taking pictures.

[0003] Patent Document 1 discloses a technique of displaying an icon indicating a driving history on a map and playing back a captured video from the selected point when accepting the selection of the icon.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Disclosure of the Invention

Problems to be Solved by the Invention

[0005] When going to the site to take pictures, it is necessary to search for where the site is and go there, resulting in a decrease in work efficiency in recording the situation. Therefore, a method of utilizing the video by an imaging means provided in a moving body such as a vehicle is conceivable. Although Patent Document 1 discloses playing back a captured video from the selected point when accepting the selection of an icon indicating a driving history, it does not disclose a method for recording the situation.

[0006] [[ID=,42]] Therefore, an object of the present invention is to improve the efficiency of the recording work of information related to a specific object.

Means for Solving the Problems

[0007] To solve the above problems, the present invention provides A means for identifying a moving object that has moved within a predetermined range including specified location information, A control means for controlling the playback of a video captured by an imaging means provided on a mobile body identified by the aforementioned identification means, A display control means that controls the video to extract and display an image containing the object, It is characterized by having the following features. [Effects of the Invention]

[0008] According to the present invention, the efficiency of recording information related to a specific object can be improved. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of a system configuration. [Figure 2] This figure shows an example of the hardware configuration of various devices. [Figure 3] This flowchart shows an example of a basic process for saving video and location information. [Figure 4] This flowchart shows an example of the detailed process for saving video and location information. [Figure 5] Here is a flowchart showing an example of video search. [Figure 6] This is a flowchart showing an example of a movement trajectory display. [Figure 7] This figure shows an example of the main screen. [Figure 8] This is a diagram showing an example of a video list screen. [Figure 9] This figure shows an example of a video playback screen. [Figure 10] This figure shows an example of the note registration process. [Figure 11] This is a diagram showing an example of a vehicle tracking screen. [Figure 12] This is an example of a data table. [Modes for carrying out the invention]

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0011] In this embodiment, a system for recording illegal waste, which is one of the application examples of the present invention, will be used for explanation. For example, when large household appliances such as refrigerators and washing machines are illegally discarded in a certain area, a citizen who finds them reports to the local government of that area.

[0012] The staff member who received the report checks the video recorded by the edge device 101 (for example, a drive recorder) installed in the vehicle that passed near the illegal waste to see if the reported illegal waste is shown in the video. The vehicle refers to a vehicle that regularly travels within the area, such as a cleaning vehicle or a garbage collection vehicle. Since such vehicles regularly travel within the area, there is a high possibility that the illegal waste is shown in the video recorded by the edge device 101.

[0013] When the staff member who received the report checks the video and the illegal waste is shown, the staff member records the information about the illegal waste in an arbitrary format. In this embodiment, an arbitrary format is called a "note", and by inputting information into the note, a plurality of information about one illegal waste can be recorded.

[0014] As a result, in the past, it was necessary to go to the site to take pictures of the situation, but now, by utilizing the video of the vehicle that passed nearby, the situation can be recorded, leading to an improvement in work efficiency.

[0015] In addition, in this embodiment, illegal waste is used as an example for explanation, but any case may be applicable as long as it is a recording of a situation when an unusual situation occurs, such as the impact of natural disasters such as typhoons or infrastructure inspections such as road subsidence.

[0016] Although this explanation uses a vehicle that regularly travels within a region as an example, any vehicle equipped with an edge device 101 that can communicate with server 102 via network 104 will suffice. Furthermore, "vehicle" includes not only cars but also trains, bicycles, motorcycles, etc. In addition, it is not limited to vehicles; it can also include flying objects such as drones.

[0017] First, with reference to Figure 1, an example of the configuration of an information processing system in an embodiment of the present invention will be described. An edge device 101 installed in a vehicle and a server 102 are connected to each other via a network 104 so as to be able to communicate. Furthermore, the server 102 is connected to a client terminal 103 via the network 104 so as to be able to communicate.

[0018] The edge device 101 is a device equipped with an imaging mechanism, such as a dashcam. Furthermore, by incorporating a GPS, it can acquire the vehicle's location information. Although an example using a single device has been described, it may consist of multiple devices. For example, the vehicle may be equipped with an imaging device and a GPS receiving terminal, and the system may consist of a server 102 and a personal computer (hereinafter referred to as PC) that can be connected via a network 104. In addition, any method that can measure the vehicle's location information is acceptable, not limited to GPS.

[0019] Server 102 is a device that, for example, stores various types of data and performs processing in response to requests from client terminals 103. The server can be a single server or composed of multiple servers. Furthermore, it can operate in a cloud environment or an on-premises environment.

[0020] The client terminal 103 is an information processing device such as a PC or smartphone. Staff members use the client terminal 103 to check video footage and record the situation.

[0021] Next, referring to Figure 2, an example configuration of a server 102 and a client terminal 103, which is an example of a device to which the present invention can be applied, is shown. Note that the server 102 does not necessarily have to have some components such as the display 205 and the camera unit 212.

[0022] In Figure 2, the CPU 201, memory 202, non-volatile memory 203, image processing unit 204, display 205, operation unit 206, recording medium interface 207, external interface 209, and communication interface 210 are connected to the internal bus 250. Each component connected to the internal bus 250 is configured to exchange data with each other via the internal bus 250.

[0023] Memory 202 consists of, for example, RAM (volatile memory using semiconductor elements). The CPU 201 controls the operator terminal 101, server 103, and supervisor terminal 104 using memory 202 as work memory, according to a program stored in, for example, non-volatile memory 203. Non-volatile memory 203 stores image data, audio data, other data, and various programs for the operation of the CPU 201. Non-volatile memory 203 consists of, for example, a hard disk (HD) or ROM.

[0024] The image processing unit 204 performs various image processing operations on image data stored in the non-volatile memory 203 and recording medium 208, video signals acquired via the external I / F 209, image data acquired via the communication I / F 210, and captured images, based on the control of the CPU 201. The image processing operations performed by the image processing unit 204 include A / D conversion, D / A conversion, image data encoding, compression, decoding, resizing, noise reduction, and color conversion. The image processing unit 204 may be composed of dedicated circuit blocks for performing specific image processing operations. Furthermore, depending on the type of image processing, the CPU 201 may perform image processing according to a program without using the image processing unit 204.

[0025] The display 205 displays images and GUI (Graphical User Interface) screens based on the control of the CPU 201. The CPU 201 generates display control signals according to the program and controls the various parts of the operator terminal 101 and supervisor terminal 104 to generate video signals for display on the display 205 and output them to the display 205. The display 205 displays images based on the output video signals. Note that the operator terminal 101 and supervisor terminal 104 themselves are configured to have an interface for outputting video signals for display on the display 205, and the display 205 may be configured as an external monitor (such as a television).

[0026] The operation unit 206 is an input device for receiving user input, including text input devices such as keyboards, pointing devices such as mice and touch panels, buttons, dials, joysticks, touch sensors, and touchpads. The touch panel is configured planarly by being superimposed on the display 205, and is an input device that outputs coordinate information corresponding to the position of contact.

[0027] The recording medium interface (I / F207) allows for the insertion of recording media 208 such as memory cards, CDs, and DVDs. Based on the control of the CPU 201, it reads data from the inserted recording media 208 and writes data to the recording media 208. The external interface (I / F209) is an interface for connecting to external devices via wired or wireless cables and for inputting and outputting video and audio signals. The communication interface (I / F210) is an interface for communicating with external devices and the internet 211 to send and receive various data such as files and commands.

[0028] The camera unit 212 is a camera unit composed of an image sensor (imaging sensor) such as a CCD or CMOS element that converts an optical image into an electrical signal.

[0029] Next, referring to Figure 3, an example of the basic processing for saving video and location information in this embodiment is shown. Each step is executed by the CPU 201 of the edge device 101 or server 102. The processing in this flowchart starts when the edge device 101 is powered on (when the vehicle engine is started).

[0030] In S301, the CPU 201 of edge device 101 requests permission to use the server function using authentication information stored on edge device 101. The format of the authentication information used here is not restricted; it can be a username and password, access key, device certificate, etc.

[0031] In S302, the CPU 201 of server 102 returns an authentication result with the necessary permissions granted according to the received authentication information. The authentication result returned here is attached to the request when calling the server function, and its format is not restricted.

[0032] In S303, the CPU 201 of the edge device 101 receives the authentication result from the server 102 and starts saving the video and location information.

[0033] In S304, the CPU 201 of the edge device 101 records video and continuously sends the recorded video to the server 102 while the edge device 101 is powered on. Details are explained using Figure 4.

[0034] In S305, the CPU 201 of the edge device 101 records location information and continuously transmits the recorded location information to the server 102 while the edge device 101 is powered on. Further details are explained using Figure 4.

[0035] Although the processes in S304 and S305 are explained using examples where they are executed in parallel, they can also be executed sequentially.

[0036] Alternatively, location information can be saved together with the video as metadata. Here, we explain examples of saving them separately for reasons such as individual use of location information or optimizing costs by setting separate retention periods for video and location information.

[0037] In S306, the CPU 201 of edge device 101 detects that the engine has stopped and that the power supply to edge device 101 has been cut off. If the power supply is cut off, the process in this flowchart is terminated; otherwise, processes S304 and S305 are executed until the power supply is cut off.

[0038] Next, with reference to Figure 4, an example of the detailed processing for saving video and location information in this embodiment will be shown. First, with reference to Figure 4(a), an example of the specific processing for saving video will be described. The processing of each step is executed by the CPU 201 of the edge device 101 or server 102.

[0039] In S401, the CPU 201 of the edge device 101 activates the camera and begins parallel execution of acquiring and transmitting the video stream.

[0040] In S402, the CPU201 of edge device 101 acquires a video stream from the camera.

[0041] In S403, CPU201 of edge device 101 compresses the video and saves it as a file.

[0042] In S404, the CPU 201 of edge device 101 adds information such as the destination of the saved video to the transmission queue.

[0043] In S405, the CPU 201 of edge device 101 returns to S402 and repeats subsequent processing until it detects engine failure (loss of power).

[0044] In S406, the CPU 201 of edge device 101 checks the transmission queue for any video information waiting to be transmitted, and if there is video information, it starts transmitting that video.

[0045] In S407, the CPU 201 of the edge device 101 retrieves the video from the storage location and sends it to the server 102. When sending, the authentication result obtained from the server 102 is incorporated into the transmitted data using a predetermined method.

[0046] In S408, the CPU 201 of server 102 verifies that the authentication result included in the received data is valid, saves the video included in the received data, and returns a save result indicating whether the save was successful or not.

[0047] Figure 12(c) shows an example of a video information table stored on server 102. The video information table 1220 consists of vehicle ID 1201, start time 1221, and video file path.

[0048] Information about the vehicles is registered in advance and recorded in the vehicle table 1200. Each vehicle is assigned a unique vehicle ID 1201.

[0049] The start time 1221 indicates the start time of video recording, and the video file path is the path to the location where the video file is saved.

[0050] In S409, the CPU 201 of the edge device 101 receives the saving result from the server 102, and if successful, deletes the video information from the transmission queue.

[0051] In S410, the CPU 201 of edge device 101 returns to S406 and repeats subsequent processing until it detects engine failure (loss of power).

[0052] Next, with reference to Figure 4(b), an example of a specific process for saving location information will be described. Each step of the process is executed by the CPU 201 of the edge device 101 or the server 102.

[0053] In S411, the CPU 201 of the edge device 101 activates the location information sensor and starts parallel execution of acquiring and transmitting location information.

[0054] In S412, the CPU 201 of the edge device 101 acquires location information from the location sensor.

[0055] In S413, the CPU 201 of the edge device 101 stores the location information in the memory or storage of the edge device 101.

[0056] In S414, the CPU 201 of the edge device 101 adds information such as the location where the location information is stored to the transmission queue.

[0057] In S415, the CPU 201 of edge device 101 returns to S412 and repeats subsequent processing until it detects engine failure (loss of power).

[0058] In S416, the CPU 201 of the edge device 101 checks the transmission queue for any location information waiting to be transmitted, and if location information is available, it starts transmitting that location information.

[0059] In S417, the CPU 201 of the edge device 101 obtains location information from the storage location and sends it to the server 102. When sending, the authentication result obtained from the server 102 is incorporated into the transmitted data using a predetermined method.

[0060] In S418, the CPU 201 of server 102 verifies that the authentication result included in the received data is valid, saves the location information included in the received data, and returns a save result indicating whether or not the saving was successful.

[0061] Figure 12(b) shows an example of a location information table stored in server 102. The location information table 1210 consists of vehicle ID 1201, start time 1211, recording time 1212, latitude and longitude 1213, and geohash 1214.

[0062] Vehicle ID 1201 in location information table 1210 is linked to vehicle ID 1201 in vehicle table 1200.

[0063] The start time 1211 is the time when the location information sensor of the edge device 101 was activated, and the recording time 1212 is the time when the location information was recorded.

[0064] The latitude and longitude coordinates 1213 indicate the vehicle's position at the recording time 1212, and geohash 1214 is the value used when searching for the vehicle. Geohash is explained in detail in Figure 5.

[0065] In S419, the CPU 201 of the edge device 101 receives the save result from the server 102, and if successful, removes the location information from the transmission queue.

[0066] In S420, the CPU 201 of edge device 101 returns to S416 and repeats subsequent processing until it detects engine failure (loss of power).

[0067] Next, with reference to Figure 5, an example of the video search process in this embodiment is shown. Each step of the process is executed by the CPU 201 of the client terminal 103 or the server 102. After receiving an instruction to start the application installed on the client terminal 103, and after accepting a predetermined period of time on the main screen, the process of this flowchart is started.

[0068] In S501, the CPU 201 of client terminal 103 requests server 102 to retrieve a list of notes for a specified period. Methods for accepting the period include setting the period from the current day to a predetermined number of days prior, having the user specify at least one of the start or end dates and times of the period, or allowing the entire period without specifying a particular timeframe.

[0069] In S502, CPU 201 of server 102 searches for and returns a list of notes for the specified period.

[0070] In S503, the CPU 201 of client terminal 103 displays a list of acquired notes and overlays an object indicating the location of each note on a map. When the user makes a selection regarding an item in the note list or an object overlaid on the map, the information for that note is displayed.

[0071] Figure 7 shows an example of the main screen. The main screen includes a map display area 701, a location object 702, a center position cursor 703, a time period area 704, a note list 705, and a video search button 706.

[0072] When the application is launched, the map display area 701 displays a map based on the latitude and longitude 1213 registered in advance. Users can also search for any location using addresses, facility names, etc. Multiple types of maps can be registered; for example, different latitude and longitude 1213 can be registered for the first map information and the second map information to manage notes in different regions. Alternatively, the first and second map information can be registered with the same latitude and longitude 1213, and notes can be managed by different types (for example, notes related to illegal dumping can be managed in the first map information, and notes related to disaster situations can be managed in the second map information).

[0073] Location object 702 indicates the location corresponding to the registered note. It is displayed so that the location of illegally dumped waste can be recognized on the map, in order to register the location of the illegal dump in the note. When a selection is made for a location object, the detailed information of that note is displayed. Location object 702 may also be displayed in different colors.

[0074] The center position cursor 703 is a cursor that indicates the center position of the map display area. The video search described later searches for vehicles in the area surrounding the center position and checks whether illegal dumping is visible in the video footage taken from those vehicles.

[0075] The period area 704 is an area where a specified period can be entered. Notes registered within the specified period will be displayed in the map display area 701 and the note list 705.

[0076] Note list 705 is an area where registered notes are displayed in a list. When an operation is performed on a note in the list, the detailed information of the selected note is displayed. You can also search for a specific note within the note list using the subject filter.

[0077] In the example in Figure 7, the registration date and subject are displayed, but this is not the only option. For example, you could filter the results to show the person in charge and display notes for each person in charge, or display them in chronological order of when the report was received.

[0078] Furthermore, the position object 702 and the corresponding note may be displayed in a way that clearly indicates their relationship. For example, the corresponding note and position object 702 may be displayed in the same color. Additionally, when a first selection is received for a note or position object 702, the relationship between the selected note and the position object may be made clear (for example, by displaying them in the same color or highlighting the position object by making it blink), and when a second selection is received, detailed information about the note may be displayed.

[0079] The video search button 706 is used to execute the process of registering notes from videos taken from a vehicle. The specific process will be described later.

[0080] In S504, the CPU 201 of client terminal 103 receives coordinate and time period input from the user.

[0081] There are several ways to input coordinates, including specifying coordinates on the map, manipulating the map's display position so that the target coordinates are at the center of the map, and selecting candidate coordinates (addresses) generated by geocoding from the input of a building name or part of an address. Here, we will explain using the point of the center position cursor 703 as the specified coordinate, but you can also specify the coordinates of any point. For example, you could specify the location of illegally dumped waste on the map and accept input of those coordinates.

[0082] There are several ways to input a period, including specifying a start and end date and time, specifying days / weeks / months, or specifying a number of days from the present to a specified past date.

[0083] In S505, when the CPU 201 of the client terminal 103 receives a press of the video search button 706, it requests the server 102 to retrieve the video list, using the coordinates and time period specified in S504 as parameters.

[0084] In S506, the CPU 201 of server 102 searches the location information table 1210 (Figure 12(b)) for records where the start time 1211 falls within the period specified in S504 and the latitude and longitude are within a predetermined range from the coordinates specified in S504. One specific search method involves using geohash, which enables high-speed searching.

[0085] Therefore, this embodiment will explain a search method using geohash, but first, we will explain the information that is handled when searching using geohash.

[0086] A geohash is a value that represents an area divided into fixed-size sections using coordinates, with a larger number of digits indicating a smaller area. For example, the 8-digit area "hogehoge" is contained within the 7-digit area "hogehog". Let's assume the 8-digit area is 30m x 30m and the 7-digit area is 300m x 300m. Note that these are simplified numbers and strings used for explanation purposes and do not represent actual geohash values.

[0087] The specified coordinates are the coordinates received from the user in S504, and the specified period is the period received from the user in S504. In this case, it is set from February 10, 2025 to February 20, 2025.

[0088] Figures 3 and 4 show the video and location information stored. The information stored on server 102 is shown in Figures 12(b) and (c), respectively.

[0089] Next, we will explain how to perform a specific search using geohash.

[0090] First, limit the number of digits in the geohash of the specified coordinates according to the size of the area you want to search. For example, if you want to search around 30m from the specified coordinates, limit it to 8 digits, "hogehoge", and if you want to search around 300m, limit it to 7 digits, "hogehog".

[0091] Next, the system searches the location information table 1210 for geohash1214 that starts with the specified coordinate geohash with a limited number of digits. For example, to search for locations around 30m, it searches for location information that starts with the 8 digits "hogehoge". Records where Geohash1214 is "xn76fgrf" do not match the beginning and are therefore not included in the search. Also, the search is limited to the specified period, so in this case, records with a start time 1211 from February 10, 2025 to February 20, 2025 are included in the search.

[0092] Specifically, the system searches the location information table 1210 for records where the geohash of the specified coordinates matches the geohash 1214 of the location information table 1210, and where the start time 1211 falls within the specified period. Then, it retrieves the latitude and longitude 1213 of the found record.

[0093] Note that the specified coordinates may be located at the edge rather than the center within the first X-digit area; in such cases, it is advisable to also search the geohash of adjacent areas.

[0094] Parameters related to the search area can be specified through system settings or as request parameters.

[0095] In S507, the CPU 201 of server 102 groups the location information of records found through the search in S506 in order to display the vehicle's movement trajectory on a map. Specifically, it groups records that have the same "vehicle ID 1201" and the same "start time 1211" from the records found through the search. In other words, by grouping the location information of a particular vehicle from the time the location information sensor is activated until it is stopped, the movement trajectory of that particular vehicle can be displayed.

[0096] While we've explained how to group location information with the same start time (12:11), there are other methods, such as grouping locations that are continuity and not separated by more than a specified time in the system settings or request parameters.

[0097] In S508, the CPU 201 of server 102 returns a list with one record per group to client terminal 103 as a result of the grouping process in S507.

[0098] Note that the processing related to S507 and S508 may also be executed on the client terminal 103 side. That is, the client terminal 103 may receive the search results from the server 102 and generate a video list using the search results.

[0099] In S509, the CPU 201 of the client terminal 103 displays the generated video list. An example of the displayed video list is shown in Figure 8, item 801. Note that the movement trajectory 802 and marker 803 are already displayed in Figure 8, but these will be displayed in the processing of S513.

[0100] In S510, the CPU 201 of client terminal 103 accepts selections for the video list 801. The selected row in the list is displayed in a way that makes it clear that it is currently selected.

[0101] In S511, the CPU 201 of the client terminal 103 requests the server 102 to obtain location information for the vehicle and period corresponding to the selected video list 801.

[0102] In S512, the CPU 201 of server 102 searches for and returns the location information of the vehicle and period corresponding to the selected video list 801. Specifically, it retrieves and returns the recording time 1212 and latitude and longitude (location information) 1213 from the records of the group corresponding to the selected video list.

[0103] In S513, the CPU 201 of the client terminal 103 plots the locations corresponding to the returned location information on a map and displays the movement trajectory 802, which is created by connecting these locations, on the map.

[0104] The movement trajectory is highlighted at its start and end points, and markers 803 are displayed to indicate the direction of travel of the vehicle. When markers 803 are displayed at regular time intervals using the recording time, it is easy to visually determine the speed of movement from the interval, and it can also be used to specify the playback start position and playback time (the time between markers 803).

[0105] The direction of travel can be calculated from the relationship between the recording time and location information, or the direction of travel can be saved along with the location information when the dashcam records it.

[0106] Now, let's explain the screen shown in Figure 8.

[0107] The screen in Figure 8 includes a video list 801, a movement trajectory of a specific vehicle 802, and a marker 803. Additionally, the map display area 701 displays an enlarged map based on the center position cursor 703 in the map of Figure 7. In other words, the map is displayed enlarged around the center position cursor 703.

[0108] Video list 801 displays the video list generated in S509. The video list displays, for example, the shooting date and time and the vehicle name. The shooting date and time refers to the recording time 1212 when the coordinates closest to the specified coordinates obtained in S508 were recorded.

[0109] The movement trajectory 802 of a specific vehicle displays the movement trajectory of the vehicle specified in the video list 801.

[0110] Marker 803 is an object that indicates the direction of travel of the vehicle specified in the video list 801. This object also indicates the start of playback of the video shot from the vehicle. Video playback can start not only when the camera is activated but also from partway through, and the video will be played from the point in time when the operation on the object is received.

[0111] In S514, the CPU 201 of the client terminal 103 accepts the selection of either the movement trajectory 802 or the marker 803.

[0112] In S515, the CPU 201 of the client terminal 103 sends a video acquisition request to the server 102 based on the movement trajectory or marker selected in S514.

[0113] Specifically, video identification can be performed using the location information of the selected marker, the recording time of that location information, and the time until the next marker (the time interval between set markers). Alternatively, the time until the next marker may be set to a predetermined time or to the end point of the trajectory.

[0114] In S516, the CPU 201 of server 102 sends the video related to the request sent in S515 to client terminal 103. Specifically, it retrieves and sends the video taken by the vehicle related to the video list selected in S510, with the recording time 1212 where the location information 1213 related to the selected marker is recorded as the start time 1221, up to the time until the next marker, from the video information table 1220.

[0115] If the location information for the selected marker is not registered in the location information table, the video may be identified by obtaining the recording time 1212 for the latitude and longitude 1213 closest to that location information.

[0116] In S517, the CPU 201 of the client terminal 103 plays the video stream received from the server 102.

[0117] An example of video stream playback is shown in Figure 9. The user checks if illegal dumping is visible in the video. If the user presses the cancel button 902, the screen returns to Figure 8, so if illegal dumping is not visible, the user returns to the previous screen and checks videos from a different starting point or videos taken from a different vehicle.

[0118] Although the example described uses videos filmed before the search time, it is also possible to play videos in real time. That is, videos filmed from a moving vehicle are acquired sequentially and played back. Similarly, the movement trajectory can be updated by acquiring location information from the edge device 101 in real time.

[0119] In S518, the CPU 201 of the client terminal 103 determines whether or not the registration button 901 has been pressed. If it has been pressed, the process proceeds to S519; otherwise, the video continues to play.

[0120] In S519, the CPU 201 of the client terminal 103 extracts a still image from the video being played when the registration button 901 is pressed. For example, the user presses the registration button 901 when they confirm that illegally dumped waste is visible in the video. As a result, a still image showing the illegally dumped waste is extracted.

[0121] Although it is currently set to a still image, it is also possible to extract a video clip. For example, if the video being played is 3 minutes long, when the registration button 901 is pressed, a few seconds before and after that point will be extracted. In that case, in the note registration process described in S520 and later, a few seconds of video will be registered instead of a still image. The method of extracting the video clip can be any method other than being triggered by the registration button being pressed; for example, the user can specify the range to be extracted.

[0122] In S520, the CPU 201 of the client terminal 103 displays the extracted still image 1005 as a candidate image 1004 on the note registration screen 1000 shown in Figure 10. It also displays a map 1002 with the coordinate information object 1003 superimposed on it.

[0123] Here, we will explain the note registration screen 1000 shown in Figure 10. The note registration screen 1000 includes a subject 1001, a map 1002, a candidate image for registration 1004, a remarks field 1005, and a save button 1006.

[0124] Subject 1001 accepts the title of the note as input. The subject may also be automatically generated using information such as the vehicle used to film the video or the address obtained by reverse geocoding from the location information.

[0125] Map 1002 displays a map with coordinate information object 1003 superimposed on it. The coordinate information may be determined according to the center position cursor 703 or according to the coordinates of the point where the video was played (i.e., the coordinates of marker 803).

[0126] The coordinate information object 1003 is movable and accepts modifications from the user. It may also be modified by accepting the specification of an address. Furthermore, the coordinate information object 1003 is the information before it is determined as the location object 702 in Figure 7. When a note is registered, it is displayed as the location object 702 in the map display area 701.

[0127] Image candidate 1004 is an image that can be registered in the notebook. It accepts registration of 0 to multiple images. Possible methods of image registration include using the camera on a smartphone to take a picture, or specifying an image saved in storage. Images may also be cropped or otherwise edited.

[0128] The remarks column 1005 accepts input of information you wish to report or share in this note. You may also automatically input a pre-registered template of explanatory text. Additionally, you may automatically input information about the vehicle used to film the video, the address obtained by reverse geocoding from the location information, or a document automatically generated based on registered images.

[0129] The save button 1006 is a button that accepts the registration of the entered content.

[0130] In S521, the CPU 201 of client terminal 103 accepts input for each item, that is, it accepts input such as subject 1001.

[0131] In S522, when the CPU 201 of the client terminal 103 receives a press of the save button 1006, it requests the server 102 to register the note using the input as a parameter. Upon receiving a press of the save button 1006, it displays the registration confirmation screen 1007, and upon receiving a press of the save button, it sends a request to the server 102.

[0132] In S523, the CPU 201 of server 102 performs note registration based on the input parameters.

[0133] In S524, the CPU 201 of the client terminal 103 receives a notification of the registration result, and if the registration is successful, updates the note list 705 and the location object 702 to the latest state.

[0134] When an operation is performed on the note list 705 or the location object 702, the note details screen 1008 is displayed. The details screen displays the registration date and time 1009, and allows for confirmation, editing, and printing of the registration details.

[0135] In this way, by searching for objects based on videos taken from a vehicle, it is unnecessary to travel to the site to confirm the object, thus reducing travel time and effort. Furthermore, if the object is not clearly visible due to lighting conditions or other reasons when filming on-site, it may be necessary to return to the site to film again. Therefore, confirming the object using videos reduces this risk.

[0136] Furthermore, if the target object is not clearly visible in the video, or if the target object cannot be seen in any of the videos, the option to go to the location and film it there becomes available. In other words, it becomes possible to perform the registration process more efficiently than going to the location from the start.

[0137] Furthermore, when reviewing video footage to search for a target object, if the time of filming is significantly later than the desired time, the target object may not be visible at all, or even if it is visible, the situation may have changed since the time of filming. In contrast, with this invention, by attaching the edge device 101 to vehicles that regularly travel within a region, such as street sweepers, the likelihood of obtaining video footage at the desired time increases.

[0138] Next, referring to Figure 6, an example of the processing related to the display of movement trajectories in this embodiment is shown. The processing of each step is executed by the CPU 201 of the client terminal 103 or server 102. This flowchart assumes processing on a screen different from the main screen. That is, when a predetermined operation is received on the main screen, the movement trajectory screen shown in Figure 11 is displayed, and the processing of this flowchart begins.

[0139] In S601, the CPU 201 of client terminal 103 displays map 1201 based on the previously registered latitude and longitude 1213. Alternatively, users can search for any location using an address, facility name, etc.

[0140] In S602, the CPU 201 of the client terminal 103 accepts input for the date and vehicle. The date 1202 accepts input for a predetermined date, and the vehicle 1203 accepts selection by any method (for example, from a pull-down list).

[0141] In S603, the CPU 201 of the client terminal 103 requests the server 102 to obtain the entered date and location information related to the vehicle.

[0142] In S604, the CPU 201 of server 102 searches for and returns the entered date and vehicle location information. Specifically, it searches for the start time 1211 in the location information table 1210 based on the entered date 1202, and searches for the vehicle ID 1201 based on the entered vehicle 1203. Since the vehicle ID 1201 is associated with the vehicle name in the vehicle table 1200, it can find the vehicle ID 1201 corresponding to the entered vehicle 1203. It searches for the target record using "start time 1211" and "vehicle ID 1201" and obtains the latitude and longitude 1213 of the found record. Then, it sends the obtained latitude and longitude 1213 to the client terminal 103.

[0143] In S605, the CPU 201 of client terminal 103 displays the movement trajectory, which is a compilation of location information, on a map. The method for generating the movement trajectory is the same as the method described in Figure 5.

[0144] On the movement trajectory, a start (end) object 1205 is displayed, indicating the start or end point of the edge device 101 (in other words, the point when the engine was started or stopped). Although shown as the same shape in Figure 11, the start and end may be displayed as objects of different shapes. Additionally, an endpoint object 1206 is displayed. This object indicates the turning point.

[0145] In S606, the CPU 201 of the client terminal 103 determines whether or not it has accepted the selection of a marker display interval 1207. If it has accepted, proceed to S607; otherwise, proceed to S608.

[0146] In S607, the CPU 201 of client terminal 103 displays markers at specified intervals. For example, if the marker interval can be selected from 1 minute to 5 minutes, the system accepts the selection of which interval to display. It is also possible to accept input of an arbitrary value for the marker interval. As explained in Figure 5, once a marker selection is accepted, the video is played from the selected point to the next marker, so the marker interval is essentially the video playback time.

[0147] Figures 12(a) and 12(b) show examples of screens with changed marker intervals. Figure 12(a) displays markers at 5-minute intervals, while Figure 12(b) shows the intervals becoming closer when set to 1 minute. Figure 12(b) results in a shorter video playback time.

[0148] In S608, CPU201 of client terminal 103 displays markers at the default interval.

[0149] In S609, the CPU 201 of the client terminal 103 accepts the selection of a movement trajectory or marker, and determines the vehicle and video playback period.

[0150] In S610, the CPU 201 of the client terminal 103 requests the server 102 to acquire a video, using the specified vehicle and specified video playback period as parameters.

[0151] In S611, the CPU 201 of server 102 searches for video files stored during a specified video playback period for a specified vehicle and delivers them as a video stream.

[0152] In S612, the CPU 201 of the client terminal 103 plays the video stream received from the server 102.

[0153] Once the video has finished playing, you can either stop it or return to the beginning and repeat the playback.

[0154] By playing the video from a selected marker to the next marker, it becomes unnecessary to review the video for irrelevant parts. For example, suppose a video exists from point A to point C, and a report is received that there is illegal dumping between point A and point B. In this case, only the section from point A to point B needs to be reviewed, but if the illegal dumping is not visible in the video, the user might end up reviewing the footage up to point C without realizing it. Therefore, by playing the video up to the next marker, only the specific section of the video can be reviewed, leading to increased efficiency in the registration process.

[0155] Although this was explained as processing on a screen different from the main screen, it could also be described as another example of the video list screen shown in Figure 8. In other words, the screen in Figure 8 could be configured to accept the specification of the marker interval.

[0156] As described above, this embodiment makes it possible to improve the efficiency of recording information related to a specific object.

[0157] In this embodiment, the use of the present invention by local governments for purposes such as illegal dumping, infrastructure inspection, and disaster situation assessment has been described as an example, but it is not limited to these. In other words, the present invention can be applied to any method of searching for a specific object using video footage taken from a vehicle and recording it. For example, when searching for a specific person, it is conceivable that the approximate address could be determined based on eyewitness accounts, and then video footage taken from a vehicle driving around that area could be reviewed and recorded.

[0158] In this embodiment, steps S517-520 in Figure 5 illustrate an example where the user searches for an object in the video and presses the registration button when the object is visible. However, AI may be used in this process. For example, an anomaly in the video can be detected using a trained model that has learned images with annotation information added to anomalies such as road collapses and corresponding labels. By identifying and displaying the detected anomalies in the video, such as by adding a rectangle, the user's chances of overlooking anomalies can be reduced. The trained model may run on the client terminal 103 or on the server 102. If processing is performed on the server 102, it is done before distributing the video to the client terminal 103 in step S516.

[0159] Furthermore, reports from citizens can be made via telephone, email, or by utilizing a specific application. In other words, an application linked to the information processing system according to the present invention can be prepared, and citizen reports can be received through the application. For example, a citizen who finds illegally dumped waste in an area can report it by transmitting its location information. When a report is received, a notification is sent to the client terminal 103, and the user can check a video of a vehicle that has taken pictures of the area around the location information. To transmit location information, for example, the user can open a map on the application and plot a pin at a designated location, or input the general area where the waste is dumped in text. In addition, when citizens report waste, they may also be able to transmit images that have been taken.

[0160] In this embodiment, we have described an example in which the user checks whether the target object is visible in the video footage taken from the vehicle, but a system that directs the vehicle directly to the scene is also possible. For example, assuming that video and location information are acquired from the vehicle in real time, the system searches the area around the reported location information to identify vehicles traveling in that area. By sharing the reported location information with the edge device 101 of the identified vehicle, that vehicle can be directed to the scene.

[0161] The present invention can take the form of, for example, a system, apparatus, method, program, or recording medium. Specifically, it may be applied to a system consisting of multiple devices, or to an apparatus consisting of a single device.

[0162] Furthermore, the various controls described above, which are performed by CPU201, may be performed by a single piece of hardware, or multiple pieces of hardware (for example, multiple processors or circuits) may share the processing to control the entire device.

[0163] Furthermore, although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0164] (Other embodiments) The present invention can also be realized by performing the following process: supplying software (programs) that realize the functions of the embodiments described above to a system or device via a network or various storage media, and having the computer (or CPU, MPU, etc.) of that system or device read and execute the program code. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0165] 101 Edge Devices 102 Servers 103 Client terminals 104 Network

Claims

1. A means for identifying a moving object that has moved within a predetermined range including location information specified on a map, A control means for controlling the playback of a video captured by an imaging means provided on a mobile body identified by the aforementioned identification means, A receiving means that receives an operation to extract an image containing an object from a video played back under the control of the aforementioned control means, A display control means controls the video to extract and display an image from the video when the reception means receives the operation, Equipped with, The identification means is characterized by identifying a moving object that has moved within a predetermined range including the location information specified on the map, while an object indicating the location of the report items related to the object, which is stored in association with the image cut out and displayed by the display control means, is displayed on the map.

2. The information processing system according to claim 1, characterized in that the display control means controls the display to display a display screen that can display together the image at the time the operation was received by the reception means and the report items related to the image.

3. The information processing system according to claim 2, characterized in that the display screen is a screen for receiving input of the reported items.

4. A management means for managing the aforementioned mobile body, Trajectory display control means that controls the display of the movement trajectory of each moving object managed by the management means, Equipped with, The information processing system according to claim 1, characterized in that the identifying means identifies a moving object that has moved along the trajectory by receiving an operation to specify the trajectory.

5. The mobile entity managed by the management means is a vehicle that travels periodically within a region in order to use video footage taken within a predetermined time period and range. The aforementioned information processing system is Video management means for managing videos captured by the imaging means installed in the vehicle. Furthermore, The identification means identifies a vehicle that has moved within a predetermined range including the specified location information, The information processing system according to claim 4, characterized in that the control means controls the reading and playback of a video corresponding to the identified vehicle from the video management means.

6. A detection means for detecting objects included in the aforementioned video, The information processing system according to any one of claims 1 to 5, further comprising an identification display means for controlling the detection means to identify and display an object detected by the detection means.

7. The information processing system according to claim 1, characterized in that the aforementioned object includes at least one of illegally dumped waste, a person, a disaster situation, and an infrastructure situation.

8. The information processing system according to claim 1, characterized in that the mobile body includes at least one of a car, a train, a bicycle, a motorcycle, and a drone.

9. A step of identifying a moving object that has moved within a predetermined range including location information specified on a map, A control step that controls the playback of a video captured by an imaging means provided on the moving object identified in the aforementioned specific step, A reception step that accepts an operation to extract an image containing an object from a video played back by the control step described above, A display control step controls the video to extract and display the image taken when the operation was received in the reception step, Equipped with, The aforementioned identification step is a control method for an information processing system characterized by identifying a moving object that has moved within a predetermined range including the location information specified on the map, while an object indicating the location of the report items related to the object, which is stored in association with the image extracted and displayed by the display control step, is displayed on the map.

10. A program for causing at least one computer to function as one of the means of the information processing system described in any one of claims 1 to 5.

11. A program for causing at least one computer to function as each of the means of the information processing system described in claim 6.

12. A program for causing at least one computer to function as each of the means of the information processing system described in claim 7 or 8.

Images