Information processing device, information processing system, information processing method, and program

Abstract

This makes it possible to suppress the occurrence of false positives. [Solution] The information processing device includes an object detection means for detecting objects in an captured image, a determination means for determining whether an object exists in a predetermined area in the captured image, a storage control means for saving a predetermined dummy image as a background image corresponding to the predetermined area if it is determined that an object exists in the predetermined area, and saving an image cut out from the captured image according to the predetermined area as a background image corresponding to the predetermined area if it is determined that no object exists, and an extraction means for extracting the difference between the image cut out from a later captured image according to the predetermined area in the time series and the background image as the foreground. The determination means determines whether an object exists based on the image cut out from a later captured image according to the predetermined area in the time series and the object detection result of the object detection means if the foreground is extracted, and determines that no object exists if the foreground is not extracted.

Description

Technical Field

[0001] The present invention relates to information processing technology using images.

Background Art

[0002] Conventionally, there has been a technology for detecting whether a vehicle (for example, an automobile) is parked in a parking area of a parking lot. As a method for detecting the presence or absence of a vehicle in each parking area of a parking lot, there is a method of installing a vehicle detection device such as an infrared sensor to detect the vehicle. Further, Patent Document 1 discloses a technology for determining the presence or absence of parking based on a captured image and a background image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, even if a determination technique based on a captured image and a background image as described in Patent Document 1 is adopted to detect whether a vehicle is parked, false detection may occur.

[0005] Therefore, an object of the present invention is to suppress the occurrence of false detection.

Means for Solving the Problems

[0006] The information processing apparatus of the present invention comprises: object detection means for detecting an object in an captured image; determination means for determining whether or not the object exists in a predetermined area in the captured image; storage control means for saving a predetermined dummy image as a background image corresponding to the predetermined area if the determination means determines that an object exists in the predetermined area, and saving an image cut out from the captured image according to the predetermined area as a background image corresponding to the predetermined area if the determination means determines that an object does not exist; and extraction means for extracting the difference between an image cut out from a later captured image in chronological order according to the predetermined area and the background image as a foreground. The determination means is characterized in that, if the foreground is extracted, it determines whether or not the object exists based on the image cut out from the later captured image in chronological order according to the predetermined area and the detection result of the object detection means, and if the foreground is not extracted, it determines that the object does not exist. [Effects of the Invention]

[0007] According to the present invention, the occurrence of false detections can be suppressed. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example configuration of an information processing system. [Figure 2] This figure shows an example of the configuration of an imaging device. [Figure 3] This figure shows an example of the functional configuration of an imaging device. [Figure 4] This figure shows an example of the hardware configuration for the first and second servers. [Figure 5] This figure shows an example of the functional configuration of the first server. [Figure 6] This figure shows an example of the functional configuration of the second server. [Figure 7] This figure shows an example of a vehicle interior settings screen. [Figure 8] This is an explanatory diagram of the vehicle detection area and vehicle detection image. [Figure 9] This figure shows an example of vehicle compartment setting information. [Figure 10] This figure shows an example of a detection result display screen. [Figure 11] This is a flowchart of the information processing according to the first embodiment. [Figure 12] This figure shows an example image used to explain the information processing according to the embodiment. [Figure 13] This figure shows an example of a screen displaying the results of the false detection suppression process according to the first embodiment. [Figure 14] This is an explanatory diagram illustrating an example of a false detection occurring in the second embodiment. [Figure 15] This is a flowchart of the information processing according to the second embodiment. [Figure 16] This figure shows an example of a screen displaying the results of the false detection suppression process according to the third embodiment. [Figure 17] This is a flowchart of the information processing according to the third embodiment. [Figure 18] This figure shows an example of input / output data from the second server. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. The following embodiments are not limiting to the present invention, and not all of the features described in these embodiments are essential to the solution of the present invention; these features may be combined arbitrarily. The configuration of the embodiments may be modified or changed as appropriate depending on the specifications of the apparatus to which the present invention is applied and various conditions (usage conditions, usage environment, etc.). Furthermore, parts of each embodiment described later may be combined as appropriate. In each of the following embodiments, redundant descriptions of the same configuration and processing steps will be omitted.

[0010] <System Configuration> FIG. 1 is a diagram showing a configuration example of an information processing system 101 according to the present embodiment. The information processing system of the present embodiment is a system that analyzes whether an object exists in a predetermined area within a predetermined location using an image captured at the predetermined location. In the following embodiments, an example in which the information processing system is applied to a parking situation determination system will be described. Here, as a method for detecting the presence or absence of a vehicle in one or more individual parking areas provided in a parking lot, there is a method of installing a vehicle detection device such as an infrared sensor as described above to detect the vehicle. However, generally, such a vehicle detection device has a limited range in which it can detect a vehicle, and there is a risk that the number of installed vehicle detection devices will become extremely large in a large-scale parking lot. On the other hand, as a method for detecting the presence or absence of a vehicle in an individual parking area, there is a method of capturing an image of the parking area with a camera and analyzing the captured image to detect whether a vehicle is parked. In this method, there is an advantage that the presence or absence of a vehicle in a plurality of parking areas can be detected from the images of a small number of cameras by capturing images of a plurality of parking areas with one camera. In the present embodiment, an example in which the presence or absence of a vehicle in a plurality of parking areas provided in a parking lot is detected from the captured images of a small number of cameras by capturing images of the plurality of parking areas with one camera will be described. That is, in the present embodiment, an example will be described in which it is determined whether a vehicle is parked or an empty vehicle is not parked in each individual parking area within the parking lot using an image captured by an imaging device of the parking lot. Here, although a parking situation determination system is taken as an example, the information processing system according to the present embodiment is not limited to the parking situation determination system and can be applied to any system that analyzes an image and outputs predetermined information.

[0011] The information processing system 101 of the present embodiment includes imaging devices 110a to 110d, a network 120, a first server 130, and a second server 131. In the following description, the imaging devices 110a to 110d will be referred to as "imaging device 110". The imaging device 110 is an imaging device such as a network camera. In this embodiment, the imaging device 110 is assumed to incorporate an arithmetic unit for processing images, but it is not limited to this. For example, an external computer such as a PC (personal computer) connected to the imaging device 110 may perform operations such as image processing, and a combination with such a PC may be treated as the imaging device 110.

[0012] The first server 130 and the second server 131 are computers such as PCs and have information processing functions according to the present embodiment described later. Also, it is assumed that the first server 130 has a function of receiving input from a user and outputting information (for example, displaying information) to the user. The imaging device 110, the first server 130, and the second server 131 are connected to each other so as to be communicable via the network 120.

[0013] The network 120 includes, for example, a plurality of routers, switches, cables, etc. that satisfy a communication standard such as Ethernet (registered trademark). Also, the network 120 may be any network that enables communication between the imaging device 110, the first server 130, and the second server 131, and may be constructed with any scale, configuration, and communication standard to which it conforms. For example, the network 120 may be the Internet, a wired LAN (Local Area Network), a wireless LAN, a WAN (Wide Area Network), etc. Also, the network 120 may be configured to enable communication using a communication protocol conforming to, for example, the ONVIF (Open Network Video Interface Forum) standard. However, it is not limited to these. The network 120 may be configured to perform communication using other communication protocols such as its own communication protocol.

[0014] <Configuration of Imaging Device> Figure 2 shows an example of the configuration of the imaging device 110. As shown in Figure 2, the imaging device 110 is configured to include, for example, an imaging unit 201, an image processing unit 202, an arithmetic processing unit 203, and a communication unit 204. The imaging unit 201 includes a lens unit for forming a light image of a subject, etc., and an image sensor for converting the formed light image into an analog signal. The lens unit has a zoom function for adjusting the angle of view and an aperture function for adjusting the amount of light. The image sensor has a gain function for adjusting the sensitivity when converting light into an analog signal. These functions are adjusted based on setting values ​​notified by the image processing unit 202. The analog signal acquired by the imaging unit 201 is converted into a digital signal by an analog-to-digital conversion circuit (not shown) and sent to the image processing unit 202 as an image signal.

[0015] The image processing unit 202 consists of an image processing engine and its peripheral devices. Peripheral devices include, for example, RAM (Random Access Memory) and drivers for each I / F (interface). The image processing unit 202 generates image data by applying image processing such as development, filtering, sensor correction, and noise reduction to the image signal from the imaging unit 201. The image processing unit 202 also sends set values ​​to the lens unit and image sensor and performs exposure adjustment to obtain an appropriately exposed image. The image data generated by the image processing unit 202 is sent to the arithmetic processing unit 203.

[0016] The arithmetic processing unit 203 consists of one or more processors such as a CPU or MPU, memory such as RAM or ROM, and drivers for each I / F. Note that CPU is an acronym for Central Processing Unit, MPU for Micro Processing Unit, RAM for Random Access Memory, and ROM for Read Only Memory.

[0017] The communication unit 204 consists of a network communication engine and peripheral devices such as RAM and an ETH PHY module. The ETH PHY module is a module that performs processing on the physical (PHY) layer of Ethernet. The communication unit 204 converts image data and processing result data acquired from the arithmetic processing unit 203 into a format that can be transmitted over the network 120, and outputs the converted data to the network 120.

[0018] Figure 3 shows an example of the functional configuration of the imaging device 110. As shown in Figure 3, the imaging device 110 includes an imaging control unit 301, a signal processing unit 302, a storage unit 303, a control unit 304, an analysis unit 305, and a network communication unit 306.

[0019] The imaging control unit 301 controls the imaging operation of the imaging unit 201. The signal processing unit 302 performs predetermined signal processing on the image captured by the imaging unit 201 to generate captured image data. In the following description, this captured image data will simply be referred to as "captured image." The signal processing unit 302 also encodes the captured image. For example, if the captured image is a still image, the signal processing unit 302 encodes the still image using an encoding method such as JPEG (Joint Photographic Experts Group). If the captured image is a moving image, the signal processing unit 302 encodes the moving image using an encoding method such as H.264 / MPEG-4 AVC or HEVC (High Efficiency Video Coding). The signal processing unit 302 can also encode the image using an encoding method selected by the user via an operation unit (not shown) of the imaging device 110 from among a plurality of pre-set encoding methods.

[0020] The memory unit 303 stores temporary data for various processes. The control unit 304 controls the signal processing unit 302, the storage unit 303, the analysis unit 305, and the network communication unit 306 to perform predetermined processes. The analysis unit 305 performs various image analysis processes on the captured image. The network communication unit 306 communicates with the first server 130 shown in Figure 1 via the network 120.

[0021] <Server Configuration> Figure 4 shows an example of the hardware configuration of the first server 130 and the second server 131. The first server 130 and the second server 131 are general-purpose computers such as PCs, and as shown in Figure 4, each includes a processor 401 such as a CPU, memory such as RAM 402 and ROM 403, storage devices such as an HDD 404, and a communication I / F 405, etc. The first server 130 can perform various functions described later by having its processor 401 execute the information processing program of this embodiment stored in memory and storage devices. The second server 131 can perform various functions described later by having its processor 401 execute the information processing program of the third embodiment described later stored in memory and storage devices.

[0022] Figure 5 is a diagram showing an example of the functional configuration of the first server 130 according to this embodiment. In this embodiment, the processor 401 of the first server 130 realizes each of the functional units shown in Figure 5 by executing the information processing program according to this embodiment stored in the RAM 402. However, this is merely an example, and some or all of the functional units shown in Figure 5 may be realized not only by the first server 130, but also by the imaging device 110 or dedicated hardware.

[0023] As shown in Figure 5, the first server 130 includes a network communication unit 501, a control unit 502, a display unit 503, an operation unit 504, an analysis unit 505, a storage unit 506, and a setting processing unit 507, etc. As will be described in detail later, the analysis unit 505 performs object detection processing to detect objects from captured images, confidence acquisition processing to obtain a confidence level indicating the likelihood of the detected objects, calculation of the foreground area ratio, and comparison of the foreground area ratio with a predetermined area ratio threshold. Similarly, as will be described in detail later, the control unit 502 performs storage control processing to save images etc. to the storage unit 506, and judgment processing based on the results of the object detection processing and confidence acquisition processing by the analysis unit 505 and the comparison of the foreground area ratio with a predetermined area ratio threshold, etc.

[0024] The network communication unit 501 communicates with external devices such as the imaging device 110 and the second server 131, for example, via the network 120. However, this is just one example, and the network communication unit 501 may also establish a direct connection with the imaging device 110 and communicate with it without going through the network 120 or other devices. The display unit 503 presents information to the user, for example, via a display device. In this embodiment, the display unit 503 presents various information to the user by displaying the results rendered by the browser on the display device. Information may also be presented to the user by methods other than screen display, such as sound or vibration. Details of the information presented to the user by the display unit 503 will be described later. The operation unit 504 receives input from the user. In this embodiment, the operation unit 504 is a mouse or keyboard, and the user operates them to input user input into the browser. However, it is not limited to this, and the operation unit 504 may be any device that can detect the intentions of other users, such as a touch panel or a microphone.

[0025] The analysis unit 505 performs the process of detecting objects from the captured image. In this embodiment, the analysis unit 505 detects vehicles from the captured image by the vehicle detection process included in the information processing according to this embodiment. In this embodiment, an example is given in which the vehicle detection process is performed by a vehicle detector. As an example, the vehicle detector used in the vehicle detection process is an object detection model trained by a method called machine learning, which finds hidden patterns in a large amount of data through iterative calculations. As an object detection model, for example, a machine learning model to which the technology described in Reference 1 below is applied can be used.

[0026] Reference 1: J. Redmon, A. Farhadi, “YOLO9000: Bett er Faster Stronger”,Computer Vision and Pattern Recognition (CVPR) 2016.

[0027] Furthermore, the analysis unit 505 obtains the region in the input captured image where the vehicle is located, and a confidence level indicating the likelihood that it is a vehicle. In this embodiment, the region where the vehicle is located is represented by a rectangular region surrounding the vehicle (called a bounding box). Details of the analysis process by the analysis unit 505 will be described later.

[0028] The setting processing unit 507 sets area information corresponding to one or more predetermined areas in the captured image. In this embodiment, the setting processing unit 507 performs a vehicle space setting process (parking frame setting process) to set vehicle space setting information (parking frame setting information) indicating a vehicle space corresponding to each individual parking area, as area information indicating one or more parking areas in the parking lot. The setting processing unit 507 also performs setting processing for suppressing false detections, as described later, in response to instructions from the user.

[0029] The storage unit 506 stores area information (vehicle compartment setting information) set by the setting processing unit 507, information from the vehicle detector used in the vehicle detection process of the analysis unit 505, vehicle detection images detected in the vehicle detection process, and background images (described later) used to determine whether or not a vehicle is parked.

[0030] The control unit 502 controls the operation and processing of the network communication unit 501, display unit 503, operation unit 504, analysis unit 505, storage unit 506, and setting processing unit 507 to realize the information processing according to this embodiment. Details of the control and processing by the control unit 502 will be described later.

[0031] Figure 6 shows an example of the functional configuration of the second server 131 according to this embodiment. Note that the second server 131 is used in the third embodiment, so the details of the operation and processing of each configuration will be explained in the third embodiment. As shown in Figure 6, the second server 131 includes, for example, a network communication unit 601, a control unit 602, and an analysis unit 603. The network communication unit 601 performs communication with an external device, such as a second server 131, via the network 120.

[0032] As will be described in the third embodiment later, the analysis unit 603 acquires the image, vehicle detection area, and prompt transmitted from the first server 130 via the network communication unit 601, and performs image generation processing based on them. The image generator used by the analysis unit 603 of the second server 131 for image generation processing is, as an example, an image generation model trained by a technique called machine learning, which finds patterns hidden in a large amount of data through iterative calculations. As an image generation model, for example, a machine learning model to which the technology described in Reference 2 below is applied can be used.

[0033] Reference 2: Alan D. Thompson,'Inside language models (from GPT-4 to PaLM)

[0034] The control unit 602 controls the operation and processing of the network communication unit 601 and the analysis unit 603, respectively, to realize the information processing according to the third embodiment described later.

[0035] Figure 7 shows an example of a vehicle room setting screen 700 that the display unit 503 displays on the display device screen under the control of the control unit 502 when vehicle room setting processing is performed in the setting processing unit 507 of the first server 130. The vehicle room setting screen 700 displays the captured image 701 taken by the imaging device 110. The captured image 701 is, for example, a 1080 x 720 pixel image. Since the imaging device 110 is imaging the parking lot, the captured image 701 shown in Figure 7 shows multiple parking areas within the parking lot, and also shows vehicles parked in some of the parking areas.

[0036] On the parking space setting screen 700, the user can specify the position and size of the parking space frame for each parking area shown in the captured image 701 by operating the operation unit 504. When the user specifies the coordinates of four vertices of a rectangular parking space frame, the setting processing unit 507 sets the rectangular area represented by the set of these four specified vertex coordinates as the parking space area. For example, the parking space area 704 consists of four parking space vertex coordinates 708, 709, 710, and 711. In the parking space setting screen 700 illustrated in Figure 7, an example is shown in which parking space areas 702, 703, 704, 705, 706, and 707 are set as parking space areas corresponding to each parking area in the parking lot.

[0037] In this embodiment, an example was given in which each parking space area is set by the user manually specifying the coordinates of the four vertices of each parking space while looking at the captured image 701 displayed on the parking space setting screen 700, but the embodiment is not limited to this. The setting processing unit 507 may be configured to detect the white lines of the parking space area on the captured image 701 using, for example, a known Hough transform, and to detect the parking space area using the intersection of the white lines as vertices.

[0038] The OK button 712 on the cabin setting screen 700 is a virtual button that the user operates when instructing the user to save the cabin area setting information (cabin setting information), which is the area information mentioned above. After the cabin area has been set, if the user virtually presses the OK button 712, the setting processing unit 507 saves the cabin setting information to the storage unit 506. The Cancel button 713 is a virtual button that the user operates when instructing the user to cancel the cabin area set on the cabin setting screen 700. For example, after the operation to specify a cabin area has been performed, if the user virtually presses the Cancel button 713, the setting processing unit 507 cancels the specified cabin area.

[0039] Figures 8(a) and 8(b) are diagrams used to explain the vehicle detection process by the analysis unit 505. Here, the vehicle detection process for the passenger compartment area 704 shown in Figure 7 will be used as an example. As shown in Figure 8(a), the analysis unit 505 estimates a three-dimensional vehicle area (e.g., a rectangular area) where a vehicle is expected to be parked in relation to the vehicle compartment area 704 as the vehicle position estimation area 805. For example, the analysis unit 505 uses the vehicle compartment area 704 as the base and calculates a three-dimensional vehicle area as the vehicle position estimation area 805 by connecting the coordinates of each vertex of the vehicle compartment area 704 with the coordinates of each point at a vertical height equivalent to, for example, 2.0m, which is assumed to be the height of the vehicle. In other words, the vehicle position estimation area 805 is a three-dimensional vehicle area consisting of eight vertices: the coordinates of each vertex of the vehicle compartment area 704 (708, 709, 710, 711) and the coordinates of each vertex (801, 802, 803, 804) at a vertical height equivalent to 2.0m from those vertex coordinates.

[0040] Furthermore, the analysis unit 505 calculates the area indicated by the vertex coordinates corresponding to the circumscribing rectangle in the captured image as the vehicle detection area 806, relative to the vehicle position estimation area 805. That is, as shown in Figure 8(b), the analysis unit 505 calculates the area indicated by the vertex coordinates 709, 710, and 711 of the passenger compartment area 704 and the vertex coordinates 804, 803, and 801 of the vehicle position estimation area 805 as the vehicle detection area 806. Furthermore, the analysis unit 505 acquires an image obtained by cropping the vehicle detection region 806 from the captured image, which is then used as the vehicle detection image 807. The use of the vehicle detection image 807 will be described later.

[0041] Figure 9 shows an example of the vehicle room setting information that is set in the setting processing unit 507 and stored in the storage unit 506. The vehicle compartment setting information 900 consists of a vehicle compartment ID 901, a vehicle vacancy status 902, and a background image 903, which will be described later, and is stored in the storage unit 506 in a table format as shown in Figure 9. The vehicle compartment ID 901 is an identifier that identifies each vehicle compartment area, and in this embodiment, vehicle compartment IDs 1 to 6 are set as identifiers that individually identify the six vehicle compartment areas 702 to 707 exemplified in Figure 7. Records 904 to 909 store information that can be identified by the vehicle compartment ID 901. Record 904 corresponds to vehicle compartment area 702, and similarly, record 905 corresponds to vehicle compartment area 703, record 906 to vehicle compartment area 704, record 907 to vehicle compartment area 705, record 908 to vehicle compartment area 706, and record 909 to vehicle compartment area 707. The vacancy status 902 stores the parking or vacant status determination result, which will be described later, as information indicating whether each vehicle compartment area is vacant or parked. In addition, records 904 to 909 for each vehicle compartment ID contain records for the corresponding vertex ID 911, X coordinate 912, and Y coordinate 913 of each vehicle compartment area, and records for the vertex ID 918, X coordinate 919, and Y coordinate 920 of each vehicle detection area. In the example in Figure 9, the vertex coordinates of vehicle compartment area 702 correspond to records 914 to 917, and the vertex coordinate information of vehicle detection area 806 corresponds to records 921 to 925.

[0042] Figure 10 shows an example of a detection result display screen 1000 that the display unit 503 displays on the display device screen under the control of the control unit 502 when vehicle detection processing is performed in the analysis unit 505 of the first server 130 according to this embodiment. The detection result display screen 1000 displays the captured image 701 taken by the imaging device 110. In the example of Figure 10, as in the example of Figure 7 described above, the captured image 701 shows multiple parking areas provided within the parking lot, and also shows vehicles parked in some of the parking areas.

[0043] The detection result display screen 1000 displays the parked or vacant vehicle determination result for each vehicle compartment area set by the setting processing unit 507. The false detection suppression button 1001 is a virtual button operated by the user to indicate whether the false detection suppression function for parked or vacant vehicles is on or off. In the example in Figure 10, the false detection suppression button 1001 is in the off state. When the false detection suppression button 1001 is set to on, the first server 130 executes the false detection suppression process included in the information processing of this embodiment.

[0044] The judgment result boxes 1002, 1003, 1004, 1005, 1006, and 1007 are areas where information regarding the determination result of whether a vehicle is parked or vacant in the corresponding vehicle compartment area is displayed. As will be described in detail later, the analysis unit 505 in this embodiment performs a vehicle detection process to obtain a confidence level of the likelihood that it is a vehicle. The control unit 502 then determines whether the confidence level obtained by the analysis unit 505 is equal to or greater than a first confidence threshold, and if it is less than the first confidence threshold, it determines that it is an vacant vehicle that is not parked. On the other hand, if the confidence level is equal to or greater than the first confidence threshold, the control unit 502 determines that a vehicle is parked in the vehicle compartment area and displays this by making the border of the judgment result box corresponding to that vehicle compartment area thicker. In this embodiment, the confidence level of the likelihood that it is a vehicle is represented by a value from 1 to 1000, for example, and the first confidence threshold is 500.

[0045] In the example shown in Figure 10, the confidence level for parking space 702 is 5, so it is determined to be empty, and the result box 1002 corresponding to parking space 702 displays the confidence level of 5. Similarly, the confidence levels for parking space 703 are 11, 705 are 2, and 707 are 3, so they are determined to be empty, and the result boxes 1003, 1005, and 1007 display the confidence levels of 11, 2, and 3, respectively. On the other hand, the confidence level for parking space 704 is 970 and for parking space 706 is 950. Since the confidence levels are above the first confidence threshold, parking space 704 and 706 are determined to be parked. Therefore, the borders of the corresponding result boxes 1004 and 1006 for parking space 704 and 706 are displayed in a thicker format.

[0046] <Flowchart of information processing according to the first embodiment> The following describes the processes performed in the information processing system 101 according to the first embodiment. Figure 11 is a flowchart showing the flow of false detection suppression processing, which is part of the information processing performed by the first server 130 of the information processing system 101. Each processing step shown in the following flowchart is executed by the respective functional units in Figure 5, which are formed by the processor 401 of the first server 130 based on the information processing program according to this embodiment stored in RAM 402. However, this is merely an example, and some or all of the processing steps described later may be implemented not only by the first server 130 but also by the imaging device 110 or dedicated hardware. The processor 401 of the first server 130 starts the information processing shown in the flowchart of Figure 11 when the user turns on the false detection suppression button 1001 on the detection result display screen 1000 shown in Figure 10.

[0047] The false detection suppression process will be explained below with reference to Figures 11 and 12. The captured images 1201, 1202, 1203, 1204, and 1205 in Figure 12(a) are examples of images taken by the imaging device 110 of the parking lot after the start of information processing according to this embodiment, and are shown in chronological order at regular intervals. These captured images 1201 to 1205 show each parking area within the parking lot. Here, it is assumed that the setting of vehicle areas corresponding to each parking area by the setting processing unit 507 as described above has already been performed. In the example of captured images 1201 to 1205, captured image 1203 is an image taken when a vehicle was newly parked in vehicle area 704, and in the subsequent captured images 1204 and 1205, it is assumed that the state of a vehicle being parked in vehicle area 704 continues. Also, in the vehicle area 706 of captured images 1201 to 1205, a vehicle has been parked there since before the start of the false detection suppression process. Furthermore, in the example of captured images 1201-1205, it is shown that shadows 1221, 1222, 1223, 1224, and 1225, cast by structures not shown, are gradually moving across the vehicle interior area 705.

[0048] The vehicle detection images 1206, 1207, 1208, 1209, and 1210 in Figure 12(b) show the results of vehicle detection processing performed by the analysis unit 505 on the vehicle interior region 704 of the corresponding captured images 1201 to 1205. The analysis unit 505 of the first server 130 obtains vehicle detection images 1206, 1207, 1208, 1209, and 1210 by performing vehicle detection processing using a vehicle detector on the captured images 1201 to 1205. In the example of Figure 12(b), the result of vehicle detection processing on vehicle detection images 1206 and 1207 is considered to be empty, and the result of vehicle detection processing on vehicle detection images 1208, 1209, and 1210 is considered to be parked.

[0049] The vehicle detection images 1211, 1212, 1213, 1214, and 1215 in Figure 12(c) show the results of vehicle detection processing performed by the analysis unit 505 on the vehicle interior region 705 of the captured images 1201 to 1205. In the example of Figure 12(c), the result of vehicle detection processing on vehicle detection images 1211 and 1212 is assumed to be empty. On the other hand, the result of vehicle detection processing on vehicle detection images 1213, 1214, and 1215 should ideally be empty, but it is assumed that they were mistakenly detected as parked due to the influence of shadows 1223, 1224, and 1225, for example.

[0050] The vehicle detection images 1216, 1217, 1218, 1219, and 1220 in Figure 12(d) are examples showing the results of vehicle detection performed by the analysis unit 505 on the vehicle interior region 706 of the captured images 1201 to 1205. In the example of Figure 12(d), the results of the vehicle detection processing for vehicle detection images 1216 to 1220 are all assumed to be parked.

[0051] In the flowchart of Figure 11, the control unit 502 of the first server 130 controls the network communication unit 501 and acquires an image from the imaging device 110 as step S1101. In the example shown in Figure 12, the control unit 502 acquires the image 1201 in step S1101. After step S1101, the processing of the first server 130 proceeds to the first determination process by loop processing from steps S1102 to S1105. The processing from steps S1102 to S1105 is carried out as loop processing for each vehicle compartment area (parking area) set by the setting processing unit 507.

[0052] When the process proceeds to step S1102, the analysis unit 505 of the first server 130 performs vehicle detection processing using the captured image 1201 acquired via the control unit 502 and the vehicle room setting information 900 stored in the storage unit 506. The analysis unit 505 performs vehicle detection processing using a vehicle detector on the vehicle detection image obtained by cropping the portion of the vehicle detection area identified by the vehicle room setting information from the captured image 1201, and obtains the confidence level of the likelihood that the detection result from the vehicle detection processing is a vehicle. In this embodiment, as described above, the analysis unit 505 outputs a value from 1 to 1000 that represents the confidence level. For example, suppose the confidence level of the result of the vehicle detection processing for the vehicle detection image 1206 shown in Figure 12(b) is 2. Also, suppose the confidence level of the result of the vehicle detection processing for the vehicle detection image 1211 shown in Figure 12(c) is 5, and the confidence level of the result of the vehicle detection processing for the vehicle detection image 1216 shown in Figure 12(d) is 960.

[0053] Next, in step S1103, the control unit 502 determines that the vehicle is parked if the confidence level obtained by the analysis unit 505 is equal to or greater than the first confidence level value, and proceeds to step S1105. On the other hand, if the confidence level is less than the first confidence level threshold, the control unit 502 determines that the vehicle is vacant and proceeds to step S1104. The first confidence level threshold is 500, as mentioned above. For example, in the case of the vehicle detection image 1206 shown in Figure 12(b), the confidence level is 2, as mentioned above, so the control unit 502 determines that the vehicle is vacant. Also, for example, in the case of the vehicle detection image 1211 shown in Figure 12(c), the confidence level is 5, so the control unit 502 determines that the vehicle is vacant. On the other hand, in the case of the vehicle detection image 1216 shown in Figure 12(d), the confidence level is 960, so the control unit 502 determines that the vehicle is parked.

[0054] If it is determined in step S1103 that the vehicle is empty and the process proceeds to step S1104, the control unit 502 performs a save control process to cause the save unit 506 to save the vehicle detection image used in the vehicle detection process as the background image 903 of the vehicle compartment ID 901 shown in Figure 9. On the other hand, if it is determined in step S1103 that the vehicle is parked and the process proceeds to step S1105, the control unit 502 performs a save control (referred to as background image registration processing) to register a dummy image as a background image. In this background image registration processing, the control unit 502 saves the dummy image to the storage unit 506 as background image 903 for vehicle space ID 901. The dummy image may be, for example, a single white image or a single black image. Alternatively, the dummy image may be, for example, an image created by duplicating another vehicle space area determined to be empty, transforming it to match the vehicle space area determined to be parked, and further adjusting the brightness and saturation to match, and then superimposing it onto the vehicle space area.

[0055] For example, as mentioned above, if vehicle detection image 1206 is determined to be empty, the control unit 502 saves the vehicle detection image 1206 as the background image 903 for the corresponding vehicle space ID 901 in the storage unit 506. Similarly, if vehicle detection image 1211 is determined to be empty, the control unit 502 saves the vehicle detection image 1211 as the background image 903 for the corresponding vehicle space ID 901 in the storage unit 506. On the other hand, as mentioned above, if vehicle detection image 1216 is determined to be parked, the control unit 502 saves a dummy image as the background image 903 for the corresponding vehicle space ID 901 in the storage unit 506. Then, after the loop processing described above in steps S1102 to S1105 has been performed on all interior areas of the captured image 1201, the control unit 502 proceeds to step S1106.

[0056] When the process proceeds to step S1106, the control unit 502 acquires the next image in chronological order from the imaging device 110. In the example in Figure 12, the control unit 502 acquires image 1202, which is acquired a certain period after image 1201. Then, the processing of the first server 130 proceeds to the second determination process by loop processing in steps S1107 to S1115. The processing in steps S1107 to S1115 is performed as loop processing for each vehicle compartment (parking area) for the image acquired in step S1106. Once the loop processing for each vehicle compartment by steps 1107 to S1115 is completed, the control unit 502 determines, as step S1116, whether or not to stop the process in the flowchart of Figure 11. For example, the control unit 502 stops the process in the flowchart of Figure 11 if a stop instruction is input from the user or the like, or if imaging by the imaging device is stopped. On the other hand, if there is no stop command input from the user or the imaging device, or if imaging is not stopped by the imaging device, the control unit 502 returns to step S1116 and acquires the next image in chronological order. In other words, the processing in steps S1107 to S1115 is performed on the images acquired from the imaging device in chronological order at regular intervals, unless it is determined in step S1116 that processing should be stopped.

[0057] In step S1107, the analysis unit 505 of the first server 130 compares the vehicle detection image acquired in step S1106 with the background image saved in step S1104 or S1105, and extracts the foreground portion. That is, the analysis unit 505 extracts the difference between the vehicle detection image and the background image as the foreground portion. Next, in step S1108, the analysis unit 505 calculates the area ratio of the foreground portion to the vehicle detection image (image area ratio of the foreground portion, hereinafter referred to as the foreground area ratio).

[0058] Here, if the captured image 1202 is acquired in step S1106, the analysis unit 505 compares the vehicle detection image acquired corresponding to the passenger compartment area of ​​the captured image 1202 with the background image stored in the storage unit 506 in step S1104 or S1105. For example, in the case of Figure 12(b), as mentioned above, the vehicle detection image 1206 acquired corresponding to the passenger compartment area 704 of the captured image 1201 was determined to be empty, so the vehicle detection image 1206 is stored in the storage unit 506 as the background image. For this reason, the foreground portion, which is the difference between the vehicle detection image 1207 corresponding to the passenger compartment area 704 of the captured image 1202 and the background image (vehicle detection image 1206), is small, and the foreground area ratio is also a small value. For example, in the case of Figure 12(c), as mentioned above, the vehicle detection image 1211 corresponding to the vehicle compartment area 705 of the captured image 1201 was determined to be empty, so the vehicle detection image 1211 is stored as a background image in the storage unit 506. Therefore, the foreground portion, which is the difference between the vehicle detection image 1212 corresponding to the vehicle compartment area 705 of the captured image 1202 and the background image (vehicle detection image 1211), is small, and the foreground area ratio is also a small value. On the other hand, for example, in the case of Figure 12(d), the vehicle detection image 1216 corresponding to the vehicle compartment area 706 of the captured image 1201 was determined to be parked, so a dummy image is stored as a background image in the storage unit 506. Therefore, the foreground portion extracted as the difference between the vehicle detection image 1217 corresponding to the vehicle compartment area 705 of the captured image 1202 and the background image (dummy image) is large, and the foreground area ratio is a large value.

[0059] Next, in step S1109, the analysis unit 505 determines whether the foreground area ratio calculated in step S1108 is equal to or greater than a predetermined area ratio threshold. In this embodiment, the predetermined area ratio threshold is, for example, 20%. The analysis unit 505 determines that the state of the vehicle interior area has changed if the foreground area ratio is equal to or greater than the predetermined area ratio threshold, and that the state of the vehicle interior area has not changed if it is less than the predetermined area ratio threshold. In other words, if the foreground area ratio, which corresponds to the difference between the background image corresponding to the previous captured image and the vehicle detection image corresponding to the next captured image in a time series sequence over a certain period, is equal to or greater than the area ratio threshold, it is considered that the state of the vehicle interior area has changed from empty to parked, or from parked to empty. On the other hand, if the foreground area ratio, which corresponds to the difference between the background image corresponding to the previous captured image and the vehicle detection image corresponding to the next captured image in a time series sequence, is less than the area ratio threshold, it is considered that the state of the vehicle interior area has not changed, remaining either empty or parked.

[0060] For example, in the case of Figure 12(b), as mentioned above, the foreground portion is small between the vehicle detection image 1207 corresponding to the vehicle interior region 704 of the captured image 1202 and the background image (vehicle detection image 1206), so the foreground area ratio is less than the area ratio threshold. Also, for example, in the case of Figure 12(c), as mentioned above, the foreground portion is small between the vehicle detection image 1212 corresponding to the vehicle interior region 705 of the captured image 1202 and the background image (vehicle detection image 1211), so the foreground area ratio is less than the area ratio threshold. For this reason, in step S1109, the analysis unit 505 determines that the foreground area ratio is equal to or greater than a predetermined area ratio threshold, that is, that the state of the vehicle interior region has changed, and proceeds to step S1110. On the other hand, in the case of Figure 12(d), the foreground portion is large between the vehicle detection image 1217 corresponding to the vehicle interior region 706 of the captured image 1202 and the background image (dummy image), so the foreground area ratio is equal to or greater than the area ratio threshold. Therefore, the analysis unit 505 determines that the area ratio threshold is below the threshold and that the state of the vehicle compartment area has not changed, meaning it remains either empty or parked, and proceeds to step S1112.

[0061] When the process proceeds to step S1110, the analysis unit 505 performs vehicle detection processing using the vehicle detector on the vehicle detection image and obtains the confidence level of the likelihood that it is a vehicle. Next, in step S1111, the control unit 502 determines whether the confidence level obtained by the analysis unit 505 in step S1110 is equal to or greater than the second confidence threshold. In this embodiment, the second confidence threshold is, for example, 200. If the confidence level is equal to or greater than the second confidence threshold, the control unit 502 proceeds to step S1113; on the other hand, if it is less than the second confidence threshold, the control unit 502 proceeds to step S1112.

[0062] When the process proceeds to step S1112, the control unit 502 calculates the image average of the background image and the vehicle detection image, and saves the resulting image as a new background image to the storage unit 506, i.e., updates the background image. By calculating the image average of the background image and the vehicle detection image, changes in the parking detection image can be gradually reflected in the previously registered background image.

[0063] If the process proceeds to step S1113, the control unit 502 determines whether the confidence level obtained by the analysis unit 505 in step S1110 is equal to or greater than the first confidence threshold (500). If the control unit 502 determines that it is equal to or greater than the first confidence threshold, it proceeds to step S1114. On the other hand, if it determines that it is less than the first confidence threshold, it proceeds to step S1115. If the process proceeds to step S1114, the control unit 502 determines that parking has occurred. On the other hand, if the process proceeds to step S1115, the control unit 502 determines that the vehicle is empty. Subsequently, as described above, the control unit 502 determines whether or not to stop the process in step S1116. If it does not stop the process, it returns to step S1116 and thereafter performs the processing in steps S1107 to S1115 for each image acquired next, as described above.

[0064] As mentioned above, in steps S1106 to S1115 of Figure 12, after loop processing is performed on the captured image 1202, the time-series captured images 1203, 1204, and 1205 are processed in order at regular intervals. In this case, for the vehicle detection images 1206-1210 in Figure 12(b) detected from the time-series captured images 1201-1205, the image changes significantly in the scene of vehicle detection image 1208, so the foreground area ratio, which represents the difference between vehicle detection image 1208 and the background image, becomes large. On the other hand, in the case of vehicle detection image 1208, the confidence level detected in step S1110 is determined to be above the second confidence threshold in step S1111, and further determined to be above the first confidence threshold in step S1113. Therefore, the vehicle compartment area 704 in the scene of vehicle detection image 1208 is determined to be parked. Also, in this case, the background image is not updated, so the background image remains the same as vehicle detection image 1207. Furthermore, for example, in vehicle detection images 1209 and beyond, the foreground area ratio, which represents the difference between the vehicle detection image and the background image, becomes large, while the confidence level is above the first confidence threshold, so the vehicle compartment area 704 is determined to be parked. Therefore, the background image is not updated, and the background image remains the vehicle detection image 1207. In other words, if the vehicle continues to be parked in the vehicle compartment area 704 thereafter, the vehicle detection image is compared with the background image (vehicle detection image 1207), and the vehicle detection process is performed, which determines that the vehicle is parked.

[0065] In the example shown in Figure 2(c), after processing is performed on captured image 1202, processing is carried out on captured images 1203, 1204, and 1205 in that order. Here, in the case of vehicle detection images 1211 to 1215 in Figure 12(c) detected from captured images 1201 to 1205 in chronological order, the foreground area ratio, which represents the difference with the background image, becomes large in the scene of vehicle detection image 1213 (it exceeds the area ratio threshold). On the other hand, in the case of vehicle detection image 1213, the confidence level of the likelihood of it being a vehicle is below the second confidence threshold. Therefore, for the vehicle compartment area 705 at this time, the background image is updated in step S1112. The same processing continues thereafter unless a new vehicle is parked or the shadow decreases. That is, in the example shown in Figure 2(c), even in the scenes of vehicle detection images 1213, 1214, and 1215, which could be mistakenly detected as parked, the foreground area ratio becomes below the second confidence threshold, and it is determined that the vehicle is empty.

[0066] In the example in Figure 2(d), after processing of captured image 1202, processing is performed on captured images 1203, 1204, and 1205 in that order. Here, in the case of vehicle detection images 1216 to 1220 in Figure 12(d), which are detected from captured images 1201 to 1205 in chronological order, the foreground area ratio, which represents the difference from the background image, becomes large in all cases (it exceeds the area ratio threshold). On the other hand, in the case of vehicle detection images 1216 to 1220, the confidence level of the likelihood that it is a vehicle is above the first confidence threshold. Therefore, the vehicle compartment area 706 at this time is determined to be parked. In the example in Figure 2(d), the background image is not updated, so the background image remains a dummy image. In other words, as long as the vehicle detection process continues to determine that it is parked, the background image remains a dummy image, and the vehicle detection process continues, so it is determined to be parked.

[0067] Figure 13 shows an example of a detection result display screen 1300 displayed on the display device screen by the display unit 503 under the control of the control unit 502 when false detection suppression processing is performed in the first server 130 of the first embodiment. On the detection result display screen 1300, the false detection suppression button 1301 is the false detection suppression button 1001 shown in Figure 10, and in the example in Figure 13, it is switched to the ON state. When the user switches the false detection suppression button 1301 from OFF to ON, the first server 130 starts the false detection suppression process. If the false detection suppression button 1301 remains ON thereafter, the user can recognize that the false detection suppression process is in progress.

[0068] The judgment result boxes 1302, 1303, 1304, 1305, 1306, and 1307, like the judgment result boxes 1002 to 1007 shown in Figure 10 above, display information regarding the determination result of whether the vehicle is parked or empty in the corresponding vehicle compartment area. For example, if the determination result from the vehicle detection process is empty, and no vehicle detection process is performed by the vehicle detector, the judgment result box displays "-" to indicate that the confidence level of the vehicle's certainty has not been obtained. Also, for example, as mentioned above, if the foreground area ratio is small and there is no change in the vehicle compartment area, or if there is little change such as the movement of a shadow, the vehicle detection process by the vehicle detector is not performed, and the same judgment result as before is displayed. In the judgment result boxes 1302, 1303, 1305, and 1307 in Figure 13, "-" is displayed to indicate that the confidence level has not been obtained. On the other hand, if the determination result from the vehicle detection process is parked, the judgment result box displays the confidence level value, similar to the example in Figure 10 above. For example, as mentioned above, if the foreground area ratio is large and there is a high probability that a vehicle is parked, the vehicle detector performs a vehicle detection process, and if it is determined that a vehicle is parked, a confidence value is displayed in the judgment result box. In Figure 13, judgment result boxes 1304 and 1306 are determined to be parked by the vehicle detection process of the vehicle detector, and the confidence values ​​obtained at that time are displayed. Since the vehicle compartment area 704 and vehicle compartment area 706 were determined to be parked, the borders of the judgment result boxes 1304 and 1306 corresponding to the vehicle compartment area 704 and vehicle compartment area 706 are displayed in a thicker color.

[0069] As explained above, if the foreground area ratio based on a comparison of the vehicle detection image of the time-series captured images with the saved background image is less than the area ratio, the first server 130 does not perform vehicle detection processing, updates the background image, and determines that the vehicle is empty. On the other hand, if the foreground area ratio is equal to or greater than the area ratio threshold, the first server 130 performs vehicle detection processing. If the confidence level obtained in the vehicle detection processing is equal to or greater than the second confidence level threshold but less than the first confidence level threshold, the first server 130 updates the background image, and if it is equal to or greater than the first confidence level threshold, it determines that the vehicle is parked. If the vehicle is parked, the first server 130 saves a dummy image as the background image.

[0070] In other words, in the first embodiment, the characteristic that the way the foreground portion of the vehicle detection image changes differs depending on whether a vehicle is newly parked or a shadow or other object enters the vehicle compartment area is utilized to suppress false detections in the vehicle detection process by the vehicle detector. For example, if the foreground area ratio is small, that is, if there is no change or the change is gradual, such as a shadow, the vehicle detection process by the vehicle detector is not performed, and the same result as before (empty) is determined to continue. On the other hand, if the foreground area ratio is large (there is a high possibility that a vehicle has been parked), the vehicle detection process by the vehicle detector is performed. Also, if a vehicle is parked when the background image is initially registered, the image of the vehicle compartment area where the vehicle is parked will be registered as the background image, and in a later stage, the foreground area ratio will be determined to be small, and the vehicle will be determined to be empty. To prevent this, if it is determined that the vehicle is parked when the background image is initially registered, a dummy image is registered as the initial registration image, so that in subsequent processing, the foreground area ratio will be determined to be large, and the vehicle detection process by the vehicle detector will be performed, and the vehicle will be determined to be parked. By performing this series of processes, the first server 130 can suppress false detections caused by slowly changing foreground elements such as shadows, which would otherwise be falsely detected by the vehicle detection process using the vehicle detector.

[0071] <Second Embodiment> In the first embodiment, an example of suppressing false detections was described by utilizing the characteristic that shadows and other elements prone to false detection change gradually. In the second embodiment, a process for suppressing false detections for captured images that are relatively prone to false detections will be described. In the second embodiment, the system configuration and the configuration of each device are the same as in the first embodiment. The following description will mainly explain the differences from the first embodiment.

[0072] Figures 14(a) and 14(b) are used to illustrate examples in which false detections are relatively likely to occur during vehicle detection processing by the vehicle detector. Figure 14(a) shows an example of an image 1400 in which false detections are likely to occur during vehicle detection processing by the vehicle detector, in which a so-called wheelchair mark is drawn in the parking area (vehicle compartment area). Figure 14(b) shows an example of a detection result display screen 1402 that the display unit 503 displays on the display device screen under the control of the control unit 502 when vehicle detection processing is performed in the first server 130 according to the second embodiment. In the detection result display screen 1402, the false detection suppression button 1001 and the judgment result boxes 1002 to 1006 are the same as in the example in Figure 10 described above.

[0073] In the example shown in Figure 14, the vehicle compartment area 1401 is an area where, although it is actually empty, the vehicle detection process by the vehicle detector results in a confidence score of 640 (i.e., a value greater than the confidence threshold of 500) because a wheelchair symbol is drawn on the parking area. In other words, the confidence score for the likelihood of a vehicle being present in the vehicle compartment area 1401 is 640, and it was mistakenly detected as a parked vehicle, so the border of the judgment result box 1403 is displayed in a thicker line.

[0074] If the false detection suppression button 1001 is turned ON and the false detection suppression process is started, the first server 130 executes the flowchart shown in Figure 11 described above. Then, if, in step S1102, the vehicle detection process for the vehicle compartment area 1401 is falsely detected as a parked vehicle in step S1103, a dummy image will be saved as the background image in step S1105. Subsequently, in step S1107, the foreground area is extracted, and in step S1108, it is determined that the foreground area ratio is above a predetermined area ratio threshold (20% or more). Furthermore, in step S1110, a confidence value of 640 is obtained in the vehicle detection process. In this case, in step S113, the confidence is determined to be above the first confidence threshold (500), and therefore, in step S1114, it is incorrectly detected as a parked vehicle.

[0075] Therefore, in the first server 130 of the second embodiment, a false detection suppression process is performed as shown in the flowchart of Figure 15. In the first server 130 of the second embodiment, triggered by the user turning on the false detection suppression button 1001 on the detection result display screen 1402 of Figure 14, the false detection suppression process shown in the flowchart of Figure 15 is started. Note that in the flowchart of Figure 15, the processes of steps S1101 to S1115 and step S1116 are the same as the corresponding steps in Figure 11. In the flowchart of Figure 15, after the loop processing of steps S1107 to S1115, the processing of the first server 130 proceeds to step S1501.

[0076] When the process proceeds to step S1501, the control unit 502 of the first server 130 determines whether the passenger compartment area displayed on the detection result display screen 1402 shown in Figure 14 has been selected by the user. In the second embodiment, the passenger compartment area selected by the user is assumed to be passenger compartment area 1401, which is prone to false detection as described above. If the control unit 502 determines that passenger compartment area 1401 has been selected by the user, the process proceeds to step S1502; otherwise, the process proceeds to step S1116.

[0077] When the process proceeds to step S1501, the control unit 502 saves the vehicle detection image as the background image for the vehicle ID 901 corresponding to the vehicle compartment area 1401 selected by the user. This ensures that the vehicle detection image, rather than a dummy image, is saved as the background image. In other words, the vehicle detection image corresponding to the vehicle compartment area 1401 with the wheelchair mark shown in Figure 14 is saved as the background image. As a result, the foreground area ratio calculated in step S1108 will be less than, for example, 20% of the area ratio threshold, and in step S1109, it will be determined that the foreground area ratio is less than the area ratio threshold, and in step S1115, it will be determined that the vehicle is empty.

[0078] Figure 16 shows an example of a detection result display screen 1600 displayed on the display device screen by the display unit 503 under the control of the control unit 502 when false detection suppression processing is performed in the first server 130 of the second embodiment. In the detection result display screen 1600 of Figure 16, the false detection suppression button 1301 and the judgment result boxes 1302 to 1006 are the same as in the example of Figure 13 described above.

[0079] In this embodiment, the user selects the vehicle compartment area 1401, and a vehicle detection image is saved as the background image instead of a dummy image. As a result, the foreground area ratio falls below the second area ratio threshold, and the system determines in step S1115 that the vehicle is empty. In this embodiment, the determination result box 1601 corresponding to the vehicle compartment area 1401 displays "-", indicating that the confidence level of the likelihood that it is a vehicle has not been obtained.

[0080] As described above, in the second embodiment, the background image can be changed from a dummy image to a vehicle detection image depending on the user's selection of the vehicle interior area, and the occurrence of false detections of vehicle interior areas that would otherwise be falsely detected by the vehicle detection process by the vehicle detector can be suppressed.

[0081] <Third Embodiment> The first and second embodiments described above explained examples of suppressing false detections. The third embodiment describes a background image registration process that replaces step S1105. In the third embodiment, the system configuration and the configuration of each device are the same as in the embodiments described above. The following description will mainly explain the differences from the embodiments described above.

[0082] Figure 17 is a flowchart showing the flow of the background image registration process performed in the information processing system 101 according to the third embodiment. First, in step S1701, the control unit 502 of the first server 130 transmits the captured image, the vehicle detection area, and the generation AI (artificial intelligence) prompt to the second server 131 via the network communication unit 501.

[0083] Figure 18(a) shows an example of input data transmitted from the first server 130 and received by the second server 131, the input data being an image capture 1800, a vehicle detection area 1801, and a prompt 1802. Next, in step S1702, the control unit 602 of the second server 131 receives the captured image 1800, the vehicle detection area 1801, and the prompt 1802 via the network communication unit 601. Furthermore, in step S1703, the control unit 602 inputs the received captured image 1800, the vehicle detection area 1801, and the prompt 1802 to the analysis unit 603.

[0084] In step S1704, the analysis unit 603 performs image generation processing using the generated AI with the captured image 1800, the vehicle detection area 1801, and the prompt 1802. In this embodiment, the prompt 1802 of the generated AI is, for example, "remove vehicles in the parking detection area from the image."

[0085] Figure 18(b) shows an example of an image 1803 generated by the AI ​​generated by the analysis unit 603 using the input captured image 1800, the vehicle detection region 1801, and the prompt 1802. Specifically, the analysis unit 603 generates an image 1803 from the captured image 1800 by removing the vehicle in the vehicle detection region 1801.

[0086] Then, in step S1705, the control unit 602 transmits the image 1803 generated by the analysis unit 603 to the first server 130 via the network communication unit 601. The control unit 502 of the first server 130 sends the image 1803 received via the network communication unit 501 to the analysis unit 505. The analysis unit 505 cuts out the vehicle detection area portion from the received image 1803 and saves the cut-out vehicle detection image as a background image to the storage unit 506.

[0087] As described above, in the third embodiment, the image generation process removes the image of the area where the vehicle is parked, and the vehicle detection image extracted from that image is saved as a background image, thereby enabling more accurate foreground extraction than saving a dummy image.

[0088] In the first to third embodiments described above, we gave examples of determining whether individual parking spaces in a parking lot are parked or empty in order to know the degree of congestion or vacancy of the parking lot. However, the example is not limited to parking lots. For example, it can also be applied to situations such as determining whether individual tables and chairs are being used in order to know the degree of congestion or usage of areas such as tables and chairs set up in a designated facility, such as a food court in a shopping center.

[0089] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more of the functions. The embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various ways without departing from its technical concept or its main features.

[0090] This embodiment includes the following configurations, methods, and programs. (Composition 1) An object detection means for detecting objects that appear in the captured image, A determination means for determining whether or not the object exists in a predetermined area shown in the captured image, If the determination means determines that an object exists in the predetermined area, a predetermined dummy image is saved as the background image corresponding to the predetermined area; if the determination means determines that no object exists, an image cut out from the captured image according to the predetermined area is saved as the background image corresponding to the predetermined area. Extraction means for extracting the difference between an image cut out from a later captured image in time series according to a predetermined region and the background image as the foreground, It has, The determination means determines whether or not the object exists based on the image cropped from the later captured images in the time series according to the predetermined region and the detection result of the object detection means when the foreground is extracted, and determines that the object does not exist when the foreground is not extracted. (Configuration 2) The system has setting means for setting region information corresponding to one or more predetermined regions that appear in the captured image, The determination process by the determination means for determining whether or not the object exists is as follows: A first determination process that determines whether or not the object exists in the predetermined region based on the region information and the detection result of the object detection means, The information processing apparatus according to configuration 1, characterized by including a second determination process, in which, if the foreground is extracted, it determines whether or not the object exists based on the image cut out from the later captured images in the time series according to the region information and the detection result of the object detection means, and determines that the object does not exist if the foreground is not extracted. (Composition 3) The information processing apparatus according to configuration 1 or 2, characterized in that the determination means determines that the foreground has been extracted if the area ratio of the foreground to the predetermined area of ​​the imaging device in the time series is equal to or greater than a predetermined area ratio threshold, and determines that the foreground has not been extracted if the area ratio of the foreground is less than the area ratio threshold. (Composition 4) The information processing apparatus according to configuration 3, wherein the storage control means updates the background image based on the background image and an image cropped from later captured images in the time series according to the predetermined region, if the area ratio of the foreground with respect to the predetermined region is equal to or greater than the area ratio threshold. (Composition 5) The information processing device according to any one of configurations 1 to 4, characterized in that the object detection means acquires a confidence level indicating the likelihood of the detected object. (Composition 6) The determination means is If the confidence level is equal to or greater than the first confidence threshold, it is determined that an object exists in the predetermined area. If the confidence level is greater than or equal to a second confidence threshold that is less than the first confidence threshold, and less than the first confidence threshold, it is determined that no object exists in the predetermined area. The information processing apparatus according to configuration 5, characterized in that the storage control means updates the background image based on the background image and an image cut out from a later captured image in the time series according to a predetermined region when the reliability is greater than or equal to the second reliability threshold and less than the first reliability threshold. (Composition 7) The information processing apparatus according to configuration 4 or 6, characterized in that the storage control means updates the background image using an image generated based on the background image and an image cropped from later captured images in the time series according to a predetermined region. (Composition 8) The information processing apparatus according to configuration 7, characterized in that the storage control means generates an image as the background image to be updated by averaging the background image and an image cut out from a later captured image in the time series according to a predetermined area. (Composition 9) The information processing apparatus according to any one of configurations 1 to 8, characterized in that the storage control means generates an image by duplicating an image corresponding to another predetermined region in which the determination means has determined that the object does not exist, and adjusting at least one of the brightness and saturation, as the background image of the predetermined region in which the determination means has determined that the object does not exist. (Composition 10) The information processing apparatus according to any one of configurations 1 to 9, characterized in that the storage control means stores, for a predetermined region specified by the user within the captured image, an image cut out from the captured image according to the predetermined region as the background image. (Composition 11) The information processing apparatus according to configuration 1, characterized by having an image generation means that generates a background image corresponding to the predetermined region based on the captured image, the predetermined region, and a prompt from the generating AI. (Composition 12) The information processing device according to configuration 11, characterized in that the prompt of the generating AI is a prompt that instructs the generation of an image in which an object in the predetermined area has been removed. (Composition 13) The information processing device according to any one of configurations 1 to 12, characterized in that the predetermined area is one or more parking areas provided in a parking lot, and the object is a vehicle. (Composition 14) One or more imaging devices, An information processing device described in any one of configurations 1 to 13, An information processing system characterized by having the following features. (Method 1) An object detection process for detecting objects in the captured image, A determination step of determining whether or not the object exists in a predetermined area shown in the captured image, If the determination step determines that an object exists in the predetermined area, a predetermined dummy image is saved as the background image corresponding to the predetermined area; if the determination step determines that no object exists, an image cut out from the captured image according to the predetermined area is saved as the background image corresponding to the predetermined area; Extraction step of extracting the difference between the background image and an image cropped from a later image in time series from the aforementioned image, according to a predetermined region, as the foreground; It has, The information processing method is characterized in that, in the determination step, if the foreground is extracted, it is determined whether or not the object exists based on the image cut out from the later captured images in the time series according to the predetermined region and the detection result of the object detection step, and if the foreground is not extracted, it is determined that the object does not exist. (Program 1) A program that causes a computer to function as an information processing device described in any one of configurations 1 through 14. [Explanation of Symbols]

[0091] 110a~110d: Imaging device, 120: Network, 130: First server, 131: Second server, 502: Control unit, 505: Analysis unit, 506: Storage unit, 507: Setting processing unit

Claims

1. An object detection means for detecting objects that appear in the captured image, A determination means for determining whether or not the object exists in a predetermined area shown in the captured image, If the determination means determines that an object exists in the predetermined area, a predetermined dummy image is saved as the background image corresponding to the predetermined area; if the determination means determines that no object exists, an image cut out from the captured image according to the predetermined area is saved as the background image corresponding to the predetermined area. Extraction means for extracting the difference between an image cut out from a later captured image in time series according to a predetermined region and the background image as the foreground, It has, The determination means determines whether or not the object exists based on the image cropped from the later captured images in the time series according to the predetermined region and the detection result of the object detection means when the foreground is extracted, and determines that the object does not exist when the foreground is not extracted.

2. The system has setting means for setting region information corresponding to one or more predetermined regions that appear in the captured image, The determination process by the determination means for determining whether or not the object exists is as follows: A first determination process that determines whether or not the object exists in the predetermined region based on the region information and the detection result of the object detection means, The information processing apparatus according to claim 1, further comprising a second determination process, in which, if the foreground is extracted, the apparatus determines whether or not the object exists based on an image cut out from a later captured image in the time series according to the region information and the detection result of the object detection means, and determines that the object does not exist if the foreground is not extracted.

3. The information processing apparatus according to claim 1, characterized in that the determination means determines that the foreground has been extracted if the area ratio of the foreground to the predetermined area of ​​the imaging device in the time series is equal to or greater than a predetermined area ratio threshold, and determines that the foreground has not been extracted if the area ratio of the foreground is less than the area ratio threshold.

4. The information processing apparatus according to claim 3, wherein the storage control means updates the background image based on the background image and an image cropped from a later captured image in the time series according to the predetermined region if the area ratio of the foreground with respect to the predetermined region is equal to or greater than the area ratio threshold.

5. The information processing apparatus according to claim 1, characterized in that the object detection means acquires a confidence level indicating the likelihood of the detected object.

6. The determination means is If the confidence level is equal to or greater than the first confidence threshold, it is determined that an object exists in the predetermined area. If the confidence level is greater than or equal to a second confidence threshold that is less than the first confidence threshold, and less than the first confidence threshold, it is determined that no object exists in the predetermined area. The information processing apparatus according to claim 5, wherein the storage control means updates the background image based on the background image and an image cropped from a later captured image in the time series according to a predetermined region when the reliability is greater than or equal to the second reliability threshold and less than the first reliability threshold.

7. The information processing apparatus according to claim 4 or 6, wherein the storage control means updates the background image using an image generated based on the background image and an image cropped from later captured images in the time series according to a predetermined region.

8. The information processing apparatus according to claim 7, characterized in that the storage control means generates an image as the background image to be updated by averaging the background image and an image cut out from a later captured image in the time series according to a predetermined area.

9. The information processing apparatus according to claim 1, characterized in that the storage control means generates an image by duplicating an image corresponding to another predetermined region in which the determination means has determined that the object does not exist, and adjusting at least one of the brightness and saturation, as the background image of the predetermined region in which the determination means has determined that the object does not exist.

10. The information processing apparatus according to claim 1, wherein the storage control means saves, for a predetermined region specified by the user within the captured image, an image cut out from the captured image according to the predetermined region, as the background image.

11. The information processing apparatus according to claim 1, further comprising an image generation means for generating a background image corresponding to the predetermined region based on the captured image, the predetermined region, and a prompt from the generating AI.

12. The information processing apparatus according to claim 11, characterized in that the prompt of the generating AI is a prompt that instructs the generation of an image in which an object in the predetermined area has been removed.

13. The information processing device according to claim 1, characterized in that the predetermined area is one or more parking areas provided in a parking lot, and the object is a vehicle.

14. One or more imaging devices, The information processing apparatus according to claim 1, An information processing system characterized by having the following features.

15. An object detection process for detecting objects in the captured image, A determination step of determining whether or not the object exists in a predetermined area shown in the captured image, If the determination step determines that an object exists in the predetermined area, a predetermined dummy image is saved as the background image corresponding to the predetermined area; if the determination step determines that no object exists, an image cut out from the captured image according to the predetermined area is saved as the background image corresponding to the predetermined area; Extraction step of extracting the difference between the background image and an image cropped from a later image in time series from the aforementioned image, according to a predetermined region, as the foreground; It has, The information processing method is characterized in that, in the determination step, if the foreground is extracted, it is determined whether or not the object exists based on the image cut out from the later captured images in the time series according to the predetermined region and the detection result of the object detection step, and if the foreground is not extracted, it is determined that the object does not exist.

16. Computers, An object detection means for detecting objects that appear in the captured image, A determination means for determining whether or not the object exists in a predetermined area shown in the captured image, If the determination means determines that an object exists in the predetermined area, a predetermined dummy image is saved as the background image corresponding to the predetermined area; if the determination means determines that no object exists, an image cut out from the captured image according to the predetermined area is saved as the background image corresponding to the predetermined area. Extraction means for extracting the difference between an image cut out from a later captured image in time series according to a predetermined region and the background image as the foreground, It has, The determination means is a program that functions as an information processing device that, when the foreground is extracted, determines whether or not the object exists based on the image cropped from the later captured images in the time series according to the predetermined region and the detection result of the object detection means, and determines that the object does not exist when the foreground is not extracted.

Images