Image processing device, and image processing method

Abstract

To provide a technique for acquiring information relating to a parking frame in a picked-up image with more simplified user operation.SOLUTION: Based on a part of a rectangular area which is set in response to a user operation to a picked-up image of a parking lot, the number of divisions of the rectangular area is determined. Information relating to each of divided rectangular areas obtained by dividing the rectangular area by the number of divisions is acquired as information relating to a parking frame.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to image processing technology.

Background Art

[0002] Conventionally, for the purpose of knowing the congestion level and the vacant vehicle situation in a parking lot, there has been a need to detect whether a vehicle (for example, an automobile) is parked in each parking area of the parking lot. As a method for detecting the presence or absence of a vehicle in each parking area, there is a method of photographing the parking area with a camera and performing image processing on the photographed image to detect whether a vehicle is parked. In this method, by photographing a plurality of parking areas, there is an advantage that the presence or absence of a vehicle in a plurality of parking areas for a plurality of vehicles can be detected with an image taken by one camera. At this time, as a method for setting the position of the parking area, a method as shown in Patent Document 1 can be cited.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the prior art methods, after designating at least four vertices of a quadrilateral including a plurality of parking areas, it is necessary to specify the number of divisions of the quadrilateral, but no mention is made of the necessary user operations or the necessary user interface for that purpose. The present invention provides a technique for acquiring information related to a parking frame in a captured image with a simpler user operation.

Means for Solving the Problem

[0006] One aspect of the present invention includes a determination means for determining the number of divisions of a quadrilateral region based on a part of the quadrilateral region set according to a user operation on a captured image of a parking lot, and an acquisition means for acquiring information related to each divided quadrilateral region obtained by dividing the quadrilateral region by the number of divisions as information related to a parking frame.

Effects of the Invention

[0007] According to the present invention, it is possible to provide a technique for acquiring information related to a parking frame in a captured image with a simpler user operation.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0010] [First Embodiment] First, a configuration example of the image analysis system 101 according to this embodiment will be described with reference to FIG. 1. In this embodiment, as an example, a case where the image analysis system 101 is applied to a parking situation determination system that is a system for determining the parking situation of vehicles in a parking lot will be described. However, the application of the image analysis system 101 is not limited to the parking situation determination system, and it can be applied to any system that analyzes images and outputs predetermined information.

[0011] As shown in FIG. 1, the image analysis system 101 includes imaging devices 110a to 110d, a network 120, and a server 130. FIG. 1 shows a case where four imaging devices (imaging devices 110a to 110d) are illustrated for the network 120, but the number of imaging devices connected to the network 120 is not limited to four. Hereinafter, the imaging devices 110a to 110d will be referred to as "imaging device 110".

[0012] First, the imaging device 110 will be described. The imaging device 110 is an imaging device such as a network camera, and transmits the captured moving images and still images to the server 130 via the network 120. For example, the imaging device 110 captures a moving image and transmits the image of each frame in the captured moving image as a captured image to the server 130 via the network 120. Also, for example, the imaging device 110 transmits the still image captured periodically or irregularly as a captured image to the server 130 via the network 120. In the present embodiment, it is assumed that the imaging device 110 is installed to image a parking lot. Further, in the present embodiment, the imaging device 110 is assumed to incorporate an arithmetic device capable of processing the captured moving images and still images, but is not limited thereto. For example, an external computer such as a PC (personal computer) connected to the imaging device 110 may exist, and these combinations may be treated as the imaging device 110.

[0013] Next, the server 130 will be described. The server 130 is a computer device such as a PC, and has a function of performing image analysis on the captured images received from the imaging device 110 via the network 120.

[0014] Next, the network 120 will be described. In the image analysis system 101, the imaging device 110 and the server 130 are configured to be able to communicate with each other via the network 120. The network 120 includes, for example, a plurality of routers, switches, cables, etc. that satisfy a communication standard such as Ethernet (registered trademark). In the present embodiment, the network 120 is any network that enables communication between the imaging device 110 and the server 130, and can be constructed with any scale, configuration, and communication standard to which it conforms. For example, the network 120 can be the Internet, a wired LAN, a wireless LAN, a WAN, etc. Further, the network 120 can be configured to be able to communicate using a communication protocol compliant with, for example, the ONVIF (Open Network Video Interface Forum) standard. However, it is not limited to this, and the network 120 can be configured to be able to communicate using other communication protocols such as, for example, its own communication protocol.

[0015] Next, an example of the hardware configuration of the imaging device 110 will be described with reference to the block diagram of FIG. 2. The imaging unit 201 includes a lens unit for forming an image of light, an image sensor that outputs an analog signal corresponding to the light imaged by the lens unit, and an A / D converter that converts the analog signal into a digital signal. The lens unit has a zoom function for adjusting the angle of view and an aperture function for adjusting the amount of light. The image sensor has a gain function for adjusting the sensitivity when converting light into an analog signal. These functions are adjusted based on the set values notified from the image processing unit 202. The digital signal converted by the A / D converter is input to the image processing unit 202 as an image signal.

[0016] The image processing unit 202 is composed of an image processing engine and its peripheral devices and the like. The peripheral devices include, for example, RAM and drivers for each I / F. The image processing unit 202 performs image processing such as development processing, filter processing, sensor correction, noise removal, etc. on the image signal input from the imaging unit 201 to generate a captured image. Also, the image processing unit 202 can execute exposure adjustment by transmitting set values to the lens unit and the imaging device so as to obtain a captured image captured with appropriate exposure. The captured image generated in the image processing unit 202 is transferred to the arithmetic processing unit 203.

[0017] The arithmetic processing unit 203 is composed of one or more processors such as a CPU and an MPU, memories such as RAM and ROM, drivers for each I / F, etc. The processor in the arithmetic processing unit 203 executes various processes using the computer programs and data stored in the memory in the arithmetic processing unit 203. Thereby, the arithmetic processing unit 203 performs various processes using the captured image transferred from the image processing unit 202, and transfers various information including the captured image and data indicating the result of the process to the distribution unit 204.

[0018] The distribution unit 204 is composed of a network distribution engine and peripheral devices such as, for example, RAM and an ETH PHY module. The ETH PHY module is a module that executes the processing of the physical (PHY) layer of Ethernet. The distribution unit 204 converts the information transferred from the arithmetic processing unit 203 into information in a form that can be distributed to the network 120, and outputs the converted information to the network 120.

[0019] Next, an example of the functional configuration of the imaging device 110 will be described using the block diagram of FIG. 3. The functional units shown in FIG. 3 may be implemented by software (computer programs) or by hardware. The functional units implemented by software are executed by processors and the like included in the arithmetic processing unit 203.

[0020] The imaging control unit 301 executes control for imaging the surrounding environment by the imaging unit 201. The signal processing unit 302 performs predetermined processing on the captured image captured under the control of the imaging control unit 301. The signal processing unit 302 encodes the captured image, for example. The signal processing unit 302 encodes a still image using an encoding method such as JPEG (Joint Photographic Experts Group), for example. Also, the signal processing unit 302 encodes a moving image using an encoding method such as H.264 / MPEG-4 AVC (hereinafter referred to as "H.264"), HEVC (High Efficiency Video Coding), or the like. Further, the signal processing unit 302 may encode the captured image using an encoding method selected by the user via an operation unit (not shown) of the imaging device 110 from among a plurality of preset encoding methods.

[0021] The control unit 304 controls the imaging control unit 301, the signal processing unit 302, the analysis unit 305, and the communication unit 306 so that each executes a predetermined process. The analysis unit 305 performs image analysis processing on the captured image. The communication unit 306 performs data communication with the server 130 via the network 120.

[0022] Next, a hardware configuration example of the server 130 will be described with reference to the block diagram of FIG. 4. The processor 401 is composed of a CPU, an MPU, or the like, and executes various processes using computer programs and data stored in the RAM 402. Thereby, the processor 401 controls the operation of the entire server 130 and executes or controls various processes described as processes performed by the server 130. Further, the processor 401 has a timer function and acquires the current time / date by measuring the time / date.

[0023] The RAM 402 has an area for storing computer programs and data loaded from the ROM 403 and the HDD 404, and an area for storing computer programs and data received from the outside via the communication I / F 405. Further, the RAM 402 has a work area used when the processor 401 executes various processes. Thus, the RAM 402 can appropriately provide various areas.

[0024] The ROM 403 stores setting data of the server 130, computer programs and data related to the startup of the server 130, computer programs and data related to the basic operations of the server 130, and the like.

[0025] The HDD 404 stores an OS, computer programs and data for causing the processor 401 to execute or control various processes described as processes performed by the server 130, and the like.

[0026] The communication I / F 405 is an interface for connecting to the network 120, and the server 130 performs data communication with devices connected to the network 120 via the communication I / F 405.

[0027] The operation unit 406 is a user interface such as a keyboard, a mouse, a touch panel screen, etc., and various instructions and information can be input to the server 130 by the user's operation.

[0028] The display unit 407 has a liquid crystal screen or a touch panel screen, and can display the processing result by the processor 401 as an image, characters, etc. Note that the display unit 407 may be a projection device such as a projector that projects images and characters.

[0029] The processor 401, the RAM 402, the ROM 403, the HDD 404, the communication I / F 405, the operation unit 406, and the display unit 407 are all connected to the system bus 408. Note that the configuration shown in FIG. 4 is only an example, and can be appropriately changed / modified.

[0030] Next, a functional configuration example of the server 130 will be described with reference to the block diagram of FIG. 5. Hereinafter, a case where the functional units shown in FIG. 5 are implemented by software (computer program) will be described. Hereinafter, when the functional units shown in FIG. 5 are described as the main body of processing, actually, the functions of the functional units are realized by the processor 401 executing the computer program corresponding to the functional units. Note that one or more of the functional units shown in FIG. 5 may be implemented by hardware.

[0031] The setting processing unit 507 executes various setting processes described in the present embodiment. The analysis unit 505 executes a process for detecting a vehicle parked in the parking lot (parked vehicle) from the captured image of the parking lot. The control unit 502 controls the operations of various functional units including the setting processing unit 507 and the analysis unit 505.

[0032] Next, a GUI related to a process (parked vehicle detection process) for detecting a vehicle parked in the parking lot (parked vehicle) from the captured image of the parking lot will be described with reference to FIG. 6. In the present embodiment, unless otherwise specified, the GUI is displayed on the display unit 407 by the processor 401, and user operations (user operations) on the GUI are performed by the user using the operation unit 406. Also, all of the display control of the GUI and the processes performed according to the operation inputs to the GUI are performed by the processor 401. And the content set by the user operating the GUI using the operation unit 406 is saved in the HDD 404 by the processor 401.

[0033] A configuration example of a parked vehicle detection screen 600, which is the main screen of the GUI related to the parked vehicle detection process, is shown in FIG. 6(a). In the area 601, a captured image of the parking lot captured by the imaging device 110 is displayed.

[0034] The button 602 is a button for instructing a setting process for detecting a vehicle parked in the parking lot (parked vehicle) from the captured image of the parking lot, that is, a setting process for the position of the parking frame (parking frame position) in the captured image.

[0035] The button 603 is a button for determining whether a vehicle is parked within the parking frame at the set parking frame position and updating the parking vehicle information (parking lot information) of the parking lot. However, the button 603 is not essential, and it may be determined whether a vehicle is parked within the parking frame at the parking frame position triggered by the setting of the parking frame position. For example, when the button 603 is instructed, as illustrated in FIG. 6(b), a parking status display 611 is superimposed and displayed on the captured image displayed in the area 601. In the parking status display 611, the parking frame with a parked vehicle is displayed as a solid line, and the parking frame without a parked vehicle is displayed as a dotted line.

[0036] An example of the parking lot information is shown in FIG. 7. The parking lot information 700 has six records: a parking frame ID 701, a parking status 702, a parking start time 703, a vertex ID 704, an X coordinate 705, and a Y coordinate 706.

[0037] The parking frame ID 701 is a unique identifier for each parking frame. In the example of FIG. 6, since the number of parking frames is 16, 16 numerical values from 1 to 16 are set as the parking frame ID 701. However, the upper limit of the number of configurable parking frame IDs 701 is not limited to a specific number, and the rule for assigning the parking frame ID 701 is not limited to this.

[0038] The parking status 702 is a determination result of the presence or absence of a parked vehicle for each parking frame. In the example of FIG. 7, the parking status 702 corresponding to the parking frame ID = 1 is "Yes", which indicates that it has been determined that there is a parked vehicle in the parking frame corresponding to the parking frame ID = 1. The parking start time 703 is the time (date and time) when the parking status 702 changes from "No" to "Yes".

[0039] The vertex ID 704 is an identifier unique to each vertex that defines a parking space. In this embodiment, since the shape of the parking space is a quadrilateral, four numerical values from 1 to 4 are set for the vertex ID 704. The X coordinate 705 is the X coordinate of each vertex of the parking space in the captured image, and the Y coordinate 706 is the Y coordinate of each vertex of the parking space in the captured image. In FIG. 7, the vertex ID 704, X coordinate 705, and Y coordinate 706 corresponding to the parking space ID = 1 are shown, but the parking lot information 700 also includes the vertex ID 704, X coordinate 705, and Y coordinate 706 corresponding to other parking space IDs. Note that the information included in the parking lot information is not limited to the above information. Also, the information related to the parking space is not limited to the above information.

[0040] When the above button 602 is instructed, a parking space position setting screen 800 exemplified in FIG. 8(a) is displayed on the display unit 407 as a GUI for performing a setting process of the position of the parking space (parking space position) in the captured image.

[0041] The button 801 is a button for instructing to automatically set the parking space position (fully automatic setting of the parking space position). In this embodiment, in order to execute the fully automatic setting of the parking space position in a specified manner in advance and output or save the setting result, it is not necessarily accompanied by a screen transition. Here, as a method for automatically setting the parking space position, there are methods such as estimating the parking space position from a straight line extracted by Hough transform on the captured image, or estimating the parking space position by statistical processing from the accumulated parking lot images, but the method is not particularly specified. When accompanied by a screen transition, it is sufficient if it is possible to select a method for executing the fully automatic setting or select an area for which the parking space position is to be set by the fully automatic setting. However, the process of the fully automatic setting is not limited to the above method, and if the semi-automatic setting process described later can be executed, a button for executing the fully automatic setting such as the button 801 is not necessarily required. Also, when transitioning from the parked vehicle detection screen 600 to the parking space position setting screen 800, in particular, the automatic setting of the parking space position may be executed without instructing a button for automatically setting the parking space position such as the button 801, and the result may be displayed as shown in FIGS. 8(b) and 8(c).

[0042] FIG. 8(b) is a diagram showing an example of the display of the parking frame position setting screen 800 when the parking frame position is set in a part of the captured image displayed in the area 601 as a result of the button 801 being instructed. As shown in FIG. 8(b), 16 parking frames are shown in the captured image, and among them, the parking frame positions of 6 parking frames are set. One parking frame 811 has four vertices 812, 813, 814, and 815. In this case, in the parking lot information, the vertex IDs of the four vertices of the parking frame 811 and the X coordinates and Y coordinates of the four vertices are registered.

[0043] FIG. 8(c) is a diagram showing an example of the display of the parking frame position setting screen 800 when the parking frame positions are set for all 16 parking frames in the captured image displayed in the area 601 as a result of the button 801 or the button 802 being instructed.

[0044] The button 802 is a button for instructing to perform the setting of the parking frame position semi-automatically. When the button 802 is instructed, the semi-automatic setting screen 900 illustrated in FIG. 9(a) is displayed on the display unit 407 as a GUI.

[0045] The button 803 is a button for instructing the transition to the setting screen of each parking frame (not shown). In the setting screen of each parking frame, it is sufficient that the vertex positions (parking frame positions) of the four points of each parking frame can be confirmed / changed, and it is not necessarily accompanied by a screen transition.

[0046] The button 804 is a button for instructing the discard of the setting contents set by the above buttons 801 to 803. When the button 804 is instructed, the screen transitions to the parked vehicle detection screen 600 which is the source screen.

[0047] The button 805 is a button for instructing the saving of the setting contents set by the above buttons 801 to 803. When the button 805 is instructed, the parking lot information 700 with the setting contents added and registered is saved in the HDD 404, and the screen transitions to the parked vehicle detection screen 600 which is the source screen.

[0048] Figure 9(a) is a diagram showing a configuration example of the semi-automatic setting screen 900. Also, the parking frame 811 is shown in FIGS. 9(a) and 9(b). The parking frame 811 is displayed only when there is a parking frame set by executing full-automatic setting of the parking frame position or the like, and when the checkbox 903 is checked.

[0049] Button 901 is a button for instructing the start of setting the parking area rectangle 911. When button 901 is instructed, the parking area rectangle 911 can be set. When button 901 is instructed, the user can specify the four vertices of a rectangular area including one or more parking frames for which the user wants to set the parking frame position on the captured image using the operation unit 406. For example, as shown in FIG. 9(b), the user can operate the operation unit 406 to specify the four vertices 912 to 915 on the captured image. After that, when the user operates the operation unit 406 to instruct the end of setting the parking area rectangle 911, the rectangular area defined by the four vertices 912 to 915 is set as the parking area rectangle 911. Also, whether or not there is a screen transition is acceptable. FIGS. 9(b), (c), and (d) show the set parking area rectangle 911. Also, when the parking area rectangle 911 is set, the radio button 902, the checkbox 903, the radio button 904, and the radio button 905 become active (selectable state).

[0050] Radio button 902 is a button for setting the parking frame position by selecting a parking frame or a parked vehicle within the parking area rectangle 911 (within the rectangular area). Checkbox 903 can be instructed when one or more parking frame positions have been set by full-automatic setting of the parking frame position or the like. When checkbox 903 is instructed, the parking frames at the set parking frame positions are displayed.

[0051] The radio button 904 is a button for setting the parking frame position by selecting the intersection of the parking frames within the parking area quadrilateral 911. The radio button 905 is a button for setting the parking frame position by specifying the number of divisions within the parking area quadrilateral 911. When the radio button 905 is checked, the text boxes 906 and 907 become active (in a state where numerical values can be input).

[0052] The text boxes 906 and 907 are text boxes for displaying the number of vertical divisions and the number of horizontal divisions of the parking area quadrilateral 911, respectively. By changing the number of divisions displayed in the text box 906 or the text box 907, the number of vertical divisions and the number of horizontal divisions of the parking area quadrilateral 911 can be changed.

[0053] The button 908 is a button for discarding the settings made using the semi-automatic setting screen 900. When the button 908 is instructed, the system transitions to the parking frame position setting screen 800, which is the previous screen.

[0054] The button 909 is a button for instructing the execution of the division of the parking area quadrilateral 911. When the button 909 is instructed, the parking area quadrilateral 911 is divided vertically by the number of divisions displayed in the text box 906 and horizontally by the number of divisions displayed in the text box 907, thereby dividing the parking area quadrilateral 911 into multiple divided quadrilateral regions, and the system transitions to the parking frame position setting screen 800, which is the previous screen.

[0055] An example of the display of the semi-automatic setting screen 900 when the radio button 902 is selected and the checkbox 903 is checked is shown in FIG. 9(b). The arrow pointer 916 is a pointer that can be operated by an operation unit 504 such as a mouse. When the arrow pointer 916 is moved into the parking frame and then a selection instruction is input, the number of vertical divisions M and the number of horizontal divisions N of the parking area quadrilateral 911 are calculated. The calculated numbers of divisions M and N are displayed in the text boxes 906 and 907, respectively.

[0056] When the radio button 902 is selected and the checkbox 903 is not checked, and a selection instruction is input after moving the arrow pointer 916 to a position within the parking frame or a position within the parked vehicle, the size of the parking frame is estimated based on the pixel value of the pixel at that position, and the division numbers M and N for dividing the parking area rectangle 911 by that size are calculated. The calculated division numbers M and N are respectively displayed in the text boxes 906 and 907.

[0057] An example of the display of the semi-automatic setting screen 900 after selecting the radio button 904 and indicating the intersection of the parking frame with the cross pointer 921 is shown in FIG. 9(c). The cross pointer 921 is a pointer that can be operated by an operation unit 504 such as a mouse, for example. When a selection instruction is input after moving the cross pointer 921 to the intersection position of the parking frame, the division number M in the vertical direction and the division number N in the horizontal direction of the parking area rectangle 911 are calculated. The calculated division numbers M and N are respectively displayed in the text boxes 906 and 907. Also, when the parking area rectangle 911 is divided in the vertical direction by the division number displayed in the text box 906 and in the horizontal direction by the division number displayed in the text box 907, each division line 922 is displayed as a dotted line.

[0058] An example of the display of the semi-automatic setting screen 900 when the radio button 905 is selected and the division numbers of the parking frame counted in the text boxes 906 and 907 are input is shown in FIG. 9(d). When the parking area rectangle 911 is divided in the vertical direction by the division number displayed in the text box 906 and in the horizontal direction by the division number displayed in the text box 907, each division line 922 is displayed as a dotted line.

[0059] In the semi-automatic setting screen 900, when the button 909 is instructed, the parking area rectangle 911 is divided vertically by the number of divisions displayed in the text box 906, and the parking area rectangle 911 is divided horizontally by the number of divisions displayed in the text box 907. The coordinates (X coordinate and Y coordinate) of the four vertices of each of the obtained plurality of divided rectangle areas are stored in the HDD 404 as the parking frame position of the parking frame corresponding to the divided rectangle area, and the process proceeds to the parking frame position setting screen 800 of FIG. 8.

[0060] Next, the operation of the server 130 will be described according to the flowchart of FIG. 10. Note that one or more of the processing steps in the flowchart shown in FIG. 10 and the sub-processing steps in the processing steps may be performed by a device other than the server 130 (for example, the imaging device 110), or may be performed by dedicated hardware for image processing mounted on the server 130. Also, at the start of the processing according to the flowchart of FIG. 10, it is assumed that the semi-automatic setting screen 900 illustrated in FIG. 9(a) is displayed on the display unit 407.

[0061] In step S1001, when the setting processing unit 507 detects that the button 901 is instructed in response to a user operation, the setting processing unit 507 accepts a designation operation of the four vertices of the parking area rectangle 911. Then, when the user operates the operation unit 406 to designate the four vertices 912 to 915 on the captured image, the four vertices 912 to 915 are displayed. Also, the setting processing unit 507 accepts a user operation on the parking area rectangle 911 (for example, changing the vertex coordinates by a mouse operation on the operation unit 504 or designating the vertex coordinates by a keyboard input).

[0062] In step S1002, the setting processing unit 507 accepts a user operation for selecting a method for setting the parking frame position. In the present embodiment, the method for setting the parking frame position can be selected by selecting either the radio button 902 or the radio button 904 in response to a user operation. However, the method for selecting the method for setting the parking frame position is not limited to a specific method, and it is not necessarily required to select the method for setting the parking frame position in response to a user operation, such as designating one of the methods as a default method.

[0063] In step S1003, the setting processing unit 507 calculates the number of vertical divisions and the number of horizontal divisions of the parking area rectangle 911 based on the setting method received in step S1002. Then, the setting processing unit 507 calculates the coordinates of each vertex of each divided rectangle area obtained by dividing the parking area rectangle 911 by the calculated number of divisions. Details of the processing in step S1003 will be described later.

[0064] In step S1004, when the button 909 is instructed, the setting processing unit 507 sets the coordinates of each vertex of each divided rectangle area calculated in step S1003 as the "parking frame position of the parking frame to be newly added to the parking lot information 700", and associates the parking frame ID of the parking frame to be newly added to the parking lot information 700 with the parking frame position of the parking frame, and registers it in the parking lot information 700. Here, it may be possible to adjust the position of each parking lot frame, or the parking frame position of a parking frame that does not require detection processing of parked vehicles may be deleted.

[0065] Next, the calculation (determination) process of the number of divisions performed in step S1003 when the radio button 902 is selected will be described according to the flowchart of FIG. 11. In step S1101, the setting processing unit 507 determines whether the parking frame position calculated in advance by full automatic setting or the like is stored in the HDD 404. As a result of this determination, if the parking frame position calculated in advance by full automatic setting or the like is stored in the HDD 404, the process proceeds to step S1002. On the other hand, if the parking frame position calculated in advance by full automatic setting or the like is not stored in the HDD 404, the setting processing unit 507 makes the check box 903 non-selectable, and the process proceeds to step S1105.

[0066] In step S1102, the setting processing unit 507 determines whether the checkbox 903 is checked. As a result of this determination, if the checkbox 903 is checked, the process proceeds to step S1103, and if the checkbox 903 is not checked, the process proceeds to step S1105.

[0067] In step S1103, the setting processing unit 507 displays the parking frames at the parking frame positions within the parking area quadrilateral 911 among the parking frame positions calculated in advance by full-auto setting or the like. In step S1104, the setting processing unit 507 accepts a user operation for selecting one of the parking frames displayed in step S1103.

[0068] In step S1105, the setting processing unit 507 accepts a user operation for designating a position within the parking frame in the parking area quadrilateral 911 or a position within the parked vehicle. For example, the setting processing unit 507 accepts an operation in which the user operates the mouse on the operation unit 504 to click a position within the parking frame in the parking area quadrilateral 911 or a position within the parked vehicle.

[0069] In step S1106, the setting processing unit 507 designates the position (a position within the parking frame in the parking area quadrilateral 911 or a position within the parked vehicle) specified according to the user operation accepted in step S1105 as the designated position, and acquires the pixel value of the pixel at the designated position. Here, any color space of the pixel value may be used as long as it represents the characteristics of the pixel, such as RGB (red - green - blue), HSV (hue - saturation - value), or HLS (hue - luminance - saturation).

[0070] In step S1107, the setting processing unit 507 performs clustering processing on the captured image displayed in area 601 based on the pixel values acquired in step S1106. Then, when the position within the parking frame inside the parking area quadrilateral 911 is designated as the designated position, the setting processing unit 507 determines that a continuous area including the designated position is one parking frame. On the other hand, when the position within the parked vehicle inside the parking area quadrilateral 911 is designated as the designated position, the setting processing unit 507 determines that a continuous area including the designated position is one parked vehicle.

[0071] Here, the clustering process is a process of grouping data based on the similarity between data. In the present embodiment, the clustering process classifies the pixel group in the captured image into a group of pixels having pixel values similar to the pixel values acquired in step S1106 and a group of pixels having pixel values not similar to the pixel values acquired in step S1106.

[0072] In step S1108, the setting processing unit 507 calculates the circumscribed rectangle of the area (one parking frame or one parked vehicle) determined in step S1107. In the present embodiment, an example of extracting the parking frame and parked vehicle in the captured image by clustering processing is shown, but it is not limited to this method. For example, the circumscribed rectangle of the vehicle may be calculated by a detector capable of detecting the position and type of an object by deep learning such as YOLO (Non-Patent Document 1).

[0073] In step S1109, the setting processing unit 507 calculates the number of vertical and horizontal divisions of the parking area quadrilateral 911 based on the size of the parking frame received in step S1104 or the size of the circumscribed rectangle calculated in step S1108. Then, the setting processing unit 507 calculates the coordinates of each vertex of each divided rectangle area obtained by dividing the vertical and horizontal directions of the parking area quadrilateral 911 by the calculated number of vertical and horizontal divisions. Details of the processing in step S1109 will be described later.

[0074] In step S1110, the setting processing unit 507 displays the calculated number of vertical divisions and the number of horizontal divisions in text boxes 906 and 907, respectively. Further, the setting processing unit 507 displays the dividing lines in region 601 when dividing the vertical and horizontal sides of the parking area quadrilateral 911 by the number of vertical and horizontal divisions.

[0075] In step S1111, the setting processing unit 507 accepts a user operation on text boxes 906 and 907, and changes the number of divisions of the parking area quadrilateral 911 according to the accepted user operation. Note that the processing in step S1111 is not essential and does not need to be executed if the user determines it is unnecessary.

[0076] Next, the details of the processing in step S1109 above will be described according to the flowchart in FIG. 12. Hereinafter, an example of dividing the parking area quadrilateral ABCD (a trapezoid where side AB is parallel to side DC and side AD is not parallel to side BD) shown in FIG. 13 into quadrilaterals abcd will be used for the description. The quadrilateral abcd is the parking frame received in step S1104 or the circumscribed quadrilateral calculated in step S1108.

[0077] In step S1201, the setting processing unit 507 calculates the centroid g of the quadrilateral abcd. The centroid of a quadrilateral can be calculated, for example, as the intersection point of two straight lines connecting the centroids of two sets of triangles formed by the diagonals of the quadrilateral and the vertices of the quadrilateral.

[0078] In step S1202, the setting processing unit 507 determines whether the opposite sides to be divided in the parking area quadrilateral ABCD are parallel. As a result of this determination, if it is determined that the opposite sides to be divided in the parking area quadrilateral ABCD are parallel, the process proceeds to step S1203, and if it is determined that the opposite sides to be divided in the parking area quadrilateral ABCD are not parallel, the process proceeds to step S1204.

[0079] For example, the setting processing unit 507 uses the positions of the vertices of the parking area quadrilateral ABCD to obtain straight lines representing the respective sides of the parking area quadrilateral ABCD, and determines whether the difference in the slopes of the straight lines of the respective sides on the opposite sides to be divided is within a threshold value. Then, if the difference in the slopes of the straight lines of the respective sides on the opposite sides to be divided is within the threshold value, the setting processing unit 507 determines that the opposite sides are parallel, and if the difference in the slopes of the straight lines of the respective sides on the opposite sides to be divided is not within the threshold value, the setting processing unit 507 determines that the opposite sides are not parallel.

[0080] In the example of FIG. 13, when the opposite sides to be divided are side AB and side DC, since side AB and side DC are parallel, the process of step S1203 is executed for these opposite sides. On the other hand, when the opposite sides to be divided are side AD and side BC, since side AD and side BC are not parallel, the processes of step S1204 and step S1205 are executed for these opposite sides.

[0081] In step S1203, the setting processing unit 507 obtains a straight line passing through the centroid g of the quadrilateral abcd and parallel to the opposite side (side AB or side DC) to be divided. In step S1204, the setting processing unit 507 calculates the vanishing point S, which is the intersection point of the extension line of side AD and the extension line of side BC. In step S1205, the setting processing unit 507 obtains a straight line passing through the centroid g of the quadrilateral abcd and the vanishing point S.

[0082] In step S1206, the setting processing unit 507 calculates the intersection points of the straight line obtained in step S1203 or the straight line obtained in step S1205 and the parking area quadrilateral ABCD, and calculates the distance between the calculated intersection points as the parking area width.

[0083] When the process proceeds from step S1203 to step S1206, in step S1206, the setting processing unit 507 calculates the intersection points E and F of the straight line obtained in step S1203 and the parking area quadrilateral ABCD. Then, the setting processing unit 507 calculates the distance between the intersection point E and the intersection point F as the parking area width EF.

[0084] On the other hand, when the process proceeds from step S1205 to step S1206, in step S1206, the setting processing unit 507 calculates the intersection points E' and F' between the straight line obtained in step S1205 and the parking area quadrilateral ABCD. Then, the setting processing unit 507 calculates the distance between the intersection point E' and the intersection point F' as the parking area width E'F'.

[0085] In step S1207, the setting processing unit 507 calculates the intersection points between the straight line obtained in step S1203 or the straight line obtained in step S1205 and the quadrilateral abcd, and calculates the distance between the calculated intersection points as the parking frame width.

[0086] When the process proceeds from step S1203 to step S1206, in step S1207, the setting processing unit 507 calculates the intersection points e and f between the straight line obtained in step S1203 and the quadrilateral abcd. Then, the setting processing unit 507 calculates the distance between the intersection point e and the intersection point f as the parking frame width ef.

[0087] On the other hand, when the process proceeds from step S1205 to step S1206, in step S1207, the setting processing unit 507 calculates the intersection points e' and f' between the straight line obtained in step S1205 and the quadrilateral abcd. Then, the setting processing unit 507 calculates the distance between the intersection point e' and the intersection point f' as the parking frame width e'f'.

[0088] In step S1208, the setting processing unit 507 determines whether the opposite sides to be divided in the parking area quadrilateral ABCD are parallel or not in the same manner as in step S1202. As a result of this determination, if it is determined that the opposite sides to be divided in the parking area quadrilateral ABCD are parallel, the process proceeds to step S1209, and if it is determined that the opposite sides to be divided in the parking area quadrilateral ABCD are not parallel, the process proceeds to step S1210.

[0089] In the example of FIG. 13, when the opposite sides to be divided are side AB and side DC, since side AB and side DC are parallel, the process of step S1209 is executed for these opposite sides. On the other hand, when the opposite sides to be divided are side AD and side BC, since side AD and side BC are not parallel, the processes of step S1210 and step S1211 are executed for these opposite sides.

[0090] In step S1209, the setting processing unit 507 calculates, as the "number of horizontal divisions of the parking area quadrilateral ABCD", a value (division value) obtained by dividing the parking area width EF by the parking frame width ef.

[0091] Note that when the division value is not an integer, the setting processing unit 507 converts the division value to an integer value by rounding the first digit after the decimal point of the division value. Note that when the division value is less than 1, the setting processing unit 507 sets the division value to 1, which is the minimum value of the number of divisions.

[0092] In step S1210, the setting processing unit 507 calculates division candidate lines that divide the parking frame width E'F' by the number of divisions for each number of divisions. Details of the process in step S1210 will be described later.

[0093] In step S1211, the setting processing unit 507 calculates the number of divisions of the parking area quadrilateral 911 based on the division candidate lines for each number of divisions calculated in step S1210. Details of the process in step S1211 will be described later.

[0094] Details of the process in step S1210 will be described according to the flowchart of FIG. 14. Hereinafter, the parking area quadrilateral ABCD (a trapezoid in which side AB and side DC are parallel and side AD and side BD are not parallel) shown in FIGS. 15 and 16 will be described as an example. Here, FIG. 15 is a diagram showing an example of dividing the opposite sides, side AD and side BC, and FIG. 16 is a diagram showing an example of dividing the opposite sides, side AB and side DC.

[0095] In step S1401, the setting processing unit 507 selects a pair of opposite sides to be divided in the parking area quadrilateral ABCD. In FIG. 15, sides AD and BC are selected as a pair of opposite sides to be divided in the parking area quadrilateral ABCD, and in FIG. 16, sides AB and DC are selected as a pair of opposite sides to be divided in the parking area quadrilateral ABCD.

[0096] In step S1402, the setting processing unit 507 determines whether one side and the other side of the opposite side (the opposite side different from the opposite side selected in step S1401 among the two opposite sides in the parking area quadrilateral ABCD) paired with the opposite side selected in step S1401 are parallel. The determination of whether two sides are parallel can be performed, for example, in the same method as step S1202 described above.

[0097] As a result of this determination, if it is determined that one side and the other side of the opposite side paired with the opposite side selected in step S1401 are parallel, the process proceeds to step S1404. On the other hand, if it is determined that one side and the other side of the opposite side paired with the opposite side selected in step S1401 are not parallel, the process proceeds to step S1403.

[0098] In the example of FIG. 15, since it is determined that one side AB and the other side DC of the opposite side paired with the opposite side selected in step S1401 are parallel, in this case, the process proceeds to step S1404.

[0099] In the example of FIG. 16, since it is determined that one side AD and the other side BC of the opposite side paired with the opposite side selected in step S1401 are not parallel, in this case, the process proceeds to step S1403.

[0100] In step S1403, the setting processing unit 507 calculates the vanishing point S, which is the intersection point of the extension line of one side and the extension line of the other side of the opposite side paired with the opposite side selected in step S1401, in the same manner as step S1204 described above.

[0101] In step S1404, the setting processing unit 507 calculates a line segment passing through two opposite vertices in the parking area quadrilateral ABCD as the diagonal of the parking area quadrilateral ABCD. In step S1405, the setting processing unit 507 initializes the value of a variable I representing the current number of divisions of the parking area quadrilateral ABCD to 1. Thereafter, in step S1406, the setting processing unit 507 repeats loop 1 until one side of the (I + 1)-divided parking area quadrilateral ABCD is less than the specified size. However, the size of one side of the parking area quadrilateral ABCD, which is the condition for ending loop 1, may be specified as a constant in the program in consideration of the visibility of the quadrilateral, or may be specified via user operation on a setting screen or the like as the minimum value of the size of one side after division.

[0102] Here, FIGS. 15(a) and 16(a) show an example of calculating one dividing line E1F1 that divides the parking area quadrilateral ABCD into two when I = 1. Also, FIGS. 15(b) and 16(b) show an example of calculating two dividing lines E2F2 and E2'F2' that divide the parking area quadrilateral ABCD into three when I = 2. Also, FIGS. 15(c) and 16(c) show an example of calculating three dividing lines E3F3, E3'F3', and E3''F3'' that divide the parking area quadrilateral ABCD into four when I = 3. From here, each time loop 1 is repeated, the processing is performed in the order of (a)→(b)→(c)→... in FIGS. 15 and 16, showing an example of calculating a dividing line that divides the parking area quadrilateral ABCD into (I + 1) with an arbitrary variable I. In the following, the explanation will mainly be given using FIG. 15, but the same explanation is equally applicable to FIG. 16.

[0103] In step S1407, the setting processing unit 507 calculates one diagonal of the quadrilaterals obtained by dividing the parking area quadrilateral ABCD into I parts, using the positions of the vertices of the quadrilateral. In the example of Fig. 15(a), the setting processing unit 507 calculates the line segment AC in the parking area quadrilateral ABCD as the diagonal. In the example of Fig. 15(b), the setting processing unit 507 calculates the line segment AF1 in the quadrilateral ABF1E1 and the line segment E1C in the quadrilateral E1F1CD as the diagonals. In the example of Fig. 15(c), the setting processing unit 507 calculates the line segment AF2' in the quadrilateral ABF2'E2', the line segment E2'F2 in the quadrilateral E2'F2'F2E2, and the line segment E2C in the quadrilateral E2F2CD as the diagonals.

[0104] In step S1408, the setting processing unit 507 calculates the intersection point XI of the line segment BD, which is the diagonal of the parking area quadrilateral ABCD calculated in step S1404, and the "diagonal of the I-divided quadrilateral" calculated in step S1407.

[0105] In the example of Fig. 15(a), the setting processing unit 507 calculates the intersection point X1 of the line segment BD and the line segment AC. In the example of Fig. 15(b), the setting processing unit 507 calculates the intersection point X2 of the line segment BD and the line segment E1C, and the intersection point X2' of the line segment BD and the line segment AF1. In the example of Fig. 15(c), the setting processing unit 507 calculates the intersection point X3 of the line segment BD and the line segment E2C, the intersection point X3' of the line segment BD and the line segment E2'F2, and the intersection point X3" of the line segment BD and the line segment AF2'.

[0106] In step S1409, the setting processing unit 507 determines whether one side and the other side of the opposite side that is paired with the opposite side selected in step S1401 are parallel, in the same manner as in step S1402 above.

[0107] As a result of this determination, if it is determined that one side and the other side of the opposite side that is paired with the opposite side selected in step S1401 are parallel, the process proceeds to step S1410. On the other hand, if it is determined that one side and the other side of the opposite side that is paired with the opposite side selected in step S1401 are not parallel, the process proceeds to step S1411.

[0108] In step S1410, the setting processing unit 507 obtains, for each intersection point calculated in step S1408, a straight line that passes through the intersection point and is parallel to the opposite side that forms a pair with the opposite side selected in step S1401.

[0109] In step S1411, the setting processing unit 507 calculates, for each intersection point calculated in step S1408, a straight line that passes through the intersection point and the vanishing point S calculated in step S1403.

[0110] In step S1412, the setting processing unit 507 calculates the intersection points EI and FI between the I straight lines calculated in step S1410 or step S1411 and the parking area quadrilateral ABCD. Then, the setting processing unit 507 calculates the line segment EIFI as a division candidate line, and divides the parking area quadrilateral ABCD into (I + 1) quadrilaterals using the division candidate line.

[0111] In the example of FIG. 15(a), the parking area quadrilateral ABCD is divided into two quadrilaterals (quadrilateral ABF1E1 and quadrilateral E1F1CD) using the line segment E1F1 as the division line. In the example of FIG. 15(b), the parking area quadrilateral ABCD is divided into three quadrilaterals (quadrilateral ABF2’E2’, quadrilateral E2’F2’F2E2, and quadrilateral E2F2CD) using the line segments E2F2 and E2’F2’ as the division lines. In the example of FIG. 15(c), the parking area quadrilateral ABCD is divided into four quadrilaterals (quadrilateral ABF3”E3”, quadrilateral E3”F3”F3’E3’, quadrilateral E3’F3’F3E3, and quadrilateral E3F3CD) using the line segments E3F3, E3’F3’, and E3”F3” as the division lines.

[0112] In step S1413, the setting processing unit 507 increments the value of the variable I by one for the next loop process. When the processing according to the flowchart of FIG. 14 is completed, in the example of FIG. 15, the division candidate line group corresponding to I = 1, the division candidate line group corresponding to I = 2, and the division candidate line group corresponding to I = 3 are obtained.

[0113] In step S1211, the setting processing unit 507 obtains the intersection point of the line segment E'F' and the division candidate line E1F1 corresponding to I = 1 as the intersection point CT. When the centroid g is located between the intersection point E' and the intersection point CT on the line segment E'F', the setting processing unit 507 obtains the distance (division width) between the intersection point E' and the intersection point CT as the comparison distance DS1. On the other hand, when the centroid g is located between the intersection point F' and the intersection point CT on the line segment E'F', the setting processing unit 507 obtains the distance between the intersection point F' and the intersection point CT as the comparison distance DS1.

[0114] Further, the setting processing unit 507 obtains the intersection point of the line segment E'F' and the division candidate line E2F2 corresponding to I = 2 as the intersection point CT1, and obtains the intersection point of the line segment E'F' and the division candidate line E2'F2' corresponding to I = 2 as the intersection point CT2. When the centroid g is located between the intersection point E' and the intersection point CT2 on the line segment E'F', the setting processing unit 507 obtains the distance between the intersection point E' and the intersection point CT2 as the comparison distance DS2. When the centroid g is located between the intersection point CT2 and the intersection point CT1 on the line segment E'F', the setting processing unit 507 obtains the distance between the intersection point CT2 and the intersection point CT1 as the comparison distance DS2. When the centroid g is located between the intersection point CT2 and the intersection point F' on the line segment E'F', the setting processing unit 507 obtains the distance between the intersection point CT1 and the intersection point F' as the comparison distance DS2.

[0115] Further, the setting processing unit 507 obtains the intersection point of the line segment E'F' and the division candidate line E3F3 corresponding to I = 3 as the intersection point CT3, and obtains the intersection point of the line segment E'F' and the division candidate line E3'F3' corresponding to I = 3 as the intersection point CT4. Further, the setting processing unit 507 obtains the intersection point of the line segment E'F' and the division candidate line E3”F3” corresponding to I = 3 as the intersection point CT5. When the centroid g is located between the intersection point E' and the intersection point CT5 on the line segment E'F', the setting processing unit 507 obtains the distance between the intersection point E' and the intersection point CT5 as the comparison distance DS3. When the centroid g is located between the intersection point CT5 and the intersection point CT4 on the line segment E'F', the setting processing unit 507 obtains the distance between the intersection point CT5 and the intersection point CT4 as the comparison distance DS3. When the centroid g is located between the intersection point CT4 and the intersection point CT3 on the line segment E'F', the setting processing unit 507 obtains the distance between the intersection point CT4 and the intersection point CT3 as the comparison distance DS3. When the centroid g is located between the intersection point CT3 and the intersection point F' on the line segment E'F', the setting processing unit 507 obtains the distance between the intersection point CT3 and the intersection point F' as the comparison distance DS3.

[0116] Then, the setting processing unit 507 identifies the comparison distance DSm that is closest to the parking frame width e'f' among the comparison distances DSx (x = 1, 2, 3), and determines (m + 1) as the number of divisions in the vertical direction of the parking area quadrilateral 911 (the number of divisions of the sides AD and BC).

[0117] For example, when the comparison distance closest to the parking frame width e'f' among the comparison distances DS1, DS2, and DS3 is the comparison distance DS1, the setting processing unit 507 determines the number of divisions in the vertical direction of the parking area quadrilateral 911 to be "2".

[0118] For another example, when the comparison distance closest to the parking frame width e'f' among the comparison distances DS1, DS2, and DS3 is the comparison distance DS2, the setting processing unit 507 determines the number of divisions in the vertical direction of the parking area quadrilateral 911 to be "3".

[0119] For example, when the comparison distance DS3 is the closest to the parking frame width e'f' among the comparison distances DS1, DS2, and DS3, the setting processing unit 507 determines the number of vertical divisions of the parking area quadrilateral 911 to be "4".

[0120] Next, the calculation (determination) process of the number of divisions performed in step S1003 when the radio button 904 is selected will be described according to the flowchart of FIG. 17. In step S1701, the setting processing unit 507 calculates the division candidate lines by the same process as in step S1210 above. However, in step S1701, the setting processing unit 507 calculates the division candidate lines for each of the two pairs of opposite sides of the parking area quadrilateral ABCD. That is, the same process as in step S1210 is performed for each of one pair of opposite sides (side AB and side DC) and the other pair of opposite sides (side AD and side BC). As a result, the setting processing unit 507 can calculate the division candidate lines that divide the parking frame width E'F' and the division candidate lines that divide the parking frame width EF.

[0121] In step S1702, the setting processing unit 507 calculates the intersections of the division candidate lines on each of the opposite sides calculated in step S1701 as division candidate points. As a result, each division candidate point in each combination (M, N) of integers M and N that divide the parking area quadrilateral ABCD into M×N is calculated. Here, due to the condition that at least one side of the parking frame is in contact with the passage area, each combination (M, N) of integers M and N is one of (M, 1), (M, 2), (1, N), and (2, N). Therefore, in this embodiment, only the division candidate points where each combination (M, N) of integers M and N is one of (M, 1), (M, 2), (1, N), and (2, N) will be calculated. However, this is not always the case when there is no condition that at least one side of the parking frame is in contact with the passage area.

[0122] The loop 1 in step S1703 repeats while the user drags the cross pointer 921 using the operation unit 504 (for example, a mouse). In step S1704, the setting processing unit 507 causes the display unit 407 to display some or all of the division candidate points within a specified distance from the cross pointer 921 among the division candidate points calculated in step S1702. An example of the display in step S1704 is shown in FIG. 18.

[0123] In FIG. 18, when the position of the cross pointer 921 is defined as the position of the point that divides the parking area quadrilateral ABCD into m × n using real numbers m and n (for example, m = 2.5, n = 2.3), the integers obtained by rounding up the decimal parts of the real numbers m and n, "m" and "n", and the integers obtained by rounding down the decimal parts of the real numbers m and n, m" and n", are used to represent the points at (m", n"), (m", "n), ("m, n") respectively, and examples of displaying the points as division candidate points 1801 to 1803 are shown.

[0124] For example, when m = 2.5 and n = 2.3, the division candidate points 1801 to 1803 are the points (2, 2), (2, 3), and (3, 2) respectively. That is, the division candidate point 1801 is the division candidate point closest to the position of the cross pointer 921 among the division candidate points in the 2x2 division. Also, the division candidate point 1802 is the division candidate point closest to the position of the cross pointer 921 among the division candidate points in the 2x3 division. Also, the division candidate point 1803 is the division candidate point closest to the position of the cross pointer 921 among the division candidate points in the 3x2 division.

[0125] At this time, by displaying both the displayed division candidate points 1801 to 1803 and the division candidate line 1804 passing through these points, the user can confirm whether the division candidate line 1804 matches the parking frame on the captured image, and the user can select the intersection points of the parking frame with high accuracy. However, it is not limited to this. For example, a plurality of division candidate points may be displayed in descending order of the distance from the cross pointer 921 on the captured image, or the division candidate point closest to each intersection point of the line of the parking frame extracted by Hough transform or the like for the captured image may be displayed. Also, here, an example of displaying the division candidate points 1801 to 1803 and the division candidate line 1804 during the drag operation of the cross pointer 921 is shown. However, it is not limited to this. For example, the division candidate points 1801 to 1803 and the division candidate line 1804 may be displayed during the pointer operation, and the end condition of the loop may be when the pointer is clicked. Furthermore, either one of the division candidate points 1801 to 1803 or the division candidate line 1804 may be displayed.

[0126] In step S1705, the setting processing unit 507 selects the division candidate point closest to the position of the cross pointer at the time when loop 1 in step S1703 ends as the selected division candidate point, and moves the cross pointer to the position of the selected division candidate point. Then, the setting processing unit 507 sets the division numbers M and N corresponding to the selected division candidate point as the division number in the vertical direction and the division number in the horizontal direction of the parking area rectangle 911, respectively, and displays each division number in the text boxes 906 and 907.

[0127] In step S1706, the setting processing unit 507 causes the display unit 407 to display a division line 922 passing through the position of the cross pointer. In step S1707, when the user operates the operation unit 504 to change the division numbers displayed in the text boxes 906 and 907, the setting processing unit 507 changes the division number of the parking area rectangle 911. Note that the processing in step S1707 is not essential and does not need to be executed if the user determines it is unnecessary.

[0128] In this way, for each number of divisions of the parking area quadrilateral ABCD, the setting processing unit 507 obtains the intersection points of the dividing lines that divide the parking area quadrilateral ABCD, and determines the number of divisions corresponding to the intersection points selected based on the positions specified according to the user operation among these intersection points as the number of divisions of the parking area quadrilateral ABCD. Through such processing, it is possible to set the parking frame positions that divide the parking area quadrilateral 911 with the optimal numbers of divisions M and N.

[0129] Next, the parking presence / absence determination process performed by the server 130 when the button 603 is instructed according to the user operation will be described according to the flowchart of FIG. 19. Loop 1 in step S1901 repeats until an end instruction for the detection process is received.

[0130] In step S1902, the control unit 502 acquires the captured image transmitted from the imaging device 110. In step S1903, the control unit 502 crops the image of the parking frame at each parking frame position included in the parking lot information 700 from the captured image acquired in step S1902 as a crop image.

[0131] In step S1904, the analysis unit 505 inputs the crop image acquired in step S1903 to a two-class classifier that uses deep learning to determine whether a vehicle exists in the image, and performs the operation of the two-class classifier to determine whether a parked vehicle is shown in the crop image.

[0132] In step S1905, the control unit 502 updates the parking status 702 and the parking start time 703 included in the parking lot information 700 based on the result of the determination in step S1904. For example, the control unit 502 updates the parking status 702 corresponding to the parking frame ID 701 of the parking frame corresponding to the crop image in which a vehicle is determined to be shown to "yes", and registers the current time in the parking start time 703 corresponding to the parking frame ID 701. Note that the control unit 502 may display the parking status and the parking time for each parking frame on the display unit 503. In this way, it is possible to determine the presence or absence of parked vehicles in each parking frame of the parking lot.

[0133] In this embodiment, the centroid g of the quadrilateral abcd is calculated in step S1201, and the number of divisions is calculated by dividing the line segment passing through the centroid g. In this method, even if the quadrilateral abcd is distorted, since the line segment passing through the centroid g bisects the area of the quadrilateral abcd, the average width of the quadrilateral abcd can be used as the parking frame width. However, it is not necessarily required to calculate the parking frame width from the line segment passing through the centroid. The parking frame width may be obtained from a line segment passing through an arbitrary point specified inside the quadrilateral abcd, or the length of any side of the quadrilateral abcd may be used as the parking frame width.

[0134] Also, in this embodiment, an example is shown in which the number of divisions for dividing two pairs of opposite sides of the quadrilateral ABCD is calculated by dividing two line segments EF and E'F' passing through the centroid g of a single point. However, the present invention is not limited to this, and the number of divisions may be calculated by dividing each pair of opposite sides of the quadrilateral ABCD.

[0135] Also, in this embodiment, regarding the process of calculating the number of divisions for dividing the parking area quadrilateral 911, a trapezoid in which one pair of opposite sides is a set of parallel sides and the other pair of opposite sides is a set of non-parallel sides is taken as an example for explanation. However, this process is not limited to this, and it may be a parallelogram in which both pairs of opposite sides are parallel, or a parking area quadrilateral having no parallel opposite sides. However, in the case of a quadrilateral in which the sides of the quadrilateral intersect at points other than the four vertices, it is not the target of this process.

[0136] As described above, according to this embodiment, the positions of the respective parking frames can be set by a simple operation of specifying the setting of the parking area quadrilateral and the information for estimating the size of the parking frames within the parking area quadrilateral without counting the number of parking frames in the captured image.

[0137] [Second Embodiment] In this embodiment, differences from the first embodiment will be described. Unless otherwise specified below, it is assumed to be the same as the first embodiment. In the first embodiment, an example of semi-automatically setting the parking frame position by selecting from a plurality of setting methods was described. In this embodiment, an example of a setting screen for directly specifying the number of divisions of the parking frame will be described in particular. Also, in this embodiment, a method for calculating the division candidate lines of the parking area rectangle will be described, particularly a method for calculating the division candidate lines using a projective transformation matrix (known).

[0138] FIG. 20(a) is a diagram showing an example of a semi-automatic setting screen 2000 for the parking frame position for directly specifying the number of divisions of the parking frame. In FIG. 20(a), a semi-automatic setting screen 2000 is shown in which the number of vertical divisions is 2, the number of horizontal divisions is 8, and division lines 922 for dividing the parking area rectangle 911 are displayed. The semi-automatic setting screen 2000 is displayed on the display unit 407 when the button 802 is instructed.

[0139] Button 2001 is a button for changing the division direction of the parking area rectangle 911 by swapping the number of vertical divisions displayed in the text box 906 and the number of horizontal divisions displayed in the text box 907. In FIG. 20(a), the parking area rectangle 911 is divided into 2 rows and 8 columns, but when the button 2001 is instructed once, the parking area rectangle 911 is divided into 8 rows and 2 columns.

[0140] Text 2002 is the number of parking frames registered in the parking frame information 700 stored in the HDD 404 (the number of set parking frame positions). Here, since the number of parking frames registered in the parking frame information 700 is 0, it is displayed as 0.

[0141] Text 2003 is the value obtained by subtracting the number of set parking frames from the upper limit number of parking frames that can be set in this system, that is, the number of parking frames that can be set currently. Assuming the upper limit number of parking frames that can be set is 30, when the number of set parking frames is 0, it is displayed as 30.

[0142] Text 2004 is the number of parking frames newly created (set) by operating the semi-automatic setting screen 2000. Here, 16, which is the product of 2 rows and 8 columns, is displayed.

[0143] The numerical values displayed as Text 2002 to 2004 are not limited to these, and the numerical values of the text may be integrated, divided, or hidden. For example, Text 2002 and Text 2003 may be integrated and the number of set parking frames may be displayed in the form of a fraction, with the numerator being the number of set parking frames and the denominator being the upper limit of the currently settable number of parking frames. Also, for example, Text 2002 may be divided and displayed in operation units. Also, for example, one or more of Text 2002 to 2004 may be hidden. Various information set on the semi-automatic setting screen 2000 is saved in the parking lot information 700 when the button 909 is instructed.

[0144] Figure 20(b) is a diagram showing an example of the semi-automatic setting screen 2000 in the process of creating (setting) 16 more parking frames by semi-automatic setting on the semi-automatic setting screen 2000 with 16 parking frames 2005 already set. In Figure 20(b), the upper limit of the number of parking frames that can be set in this system is set to 30, and since the numerical value of Text 2004 exceeds the numerical value of Text 2003, an error message text 2006 is displayed. Also, while the error message text 2006 is displayed, the button 909 is grayed out so that the user cannot give an instruction.

[0145] Figure 21 is a flowchart of the process for calculating the candidate lines for dividing the parking area quadrilateral when the shape or the number of divisions of the parking area quadrilateral is changed. Below, an example will be used to explain the case where the parking area quadrilateral ABCD shown in Figure 22 is divided into 2 equal parts for the opposite sides (side AD and side BC) and 4 equal parts for the opposite sides (side AB and side DC) using the number of vertical divisions "2" and the number of horizontal divisions "4".

[0146] In step S2101, the setting processing unit 507 acquires the number of divisions of the parking area quadrilateral ABCD. Here, a case will be described where the number of divisions M = 2 for the opposite sides (side AD and side BC) and the number of divisions N = 4 for the opposite sides (side AB and side DC) are acquired. Also, if the correspondence between the number of divisions and the opposite sides is different from the user's intention, for example, the user can switch the correspondence by instructing the above button 2001.

[0147] In step S2102, the setting processing unit 507 calculates a projective transformation matrix H from the reference quadrilateral A'B'C'D' illustrated in FIG. 22(b) to the parking area quadrilateral ABCD illustrated in FIG. 22(a) according to the following equations (1) to (9).

[0148]

Equation

[0149] Here, x and y are pixel coordinates in a coordinate system that defines the coordinates of the parking area quadrilateral ABCD, and x' and y' are coordinates in a reference coordinate system that defines the coordinates of the reference quadrilateral A'B'C'D'. Also, h11, h12, h13, h21, h22, h23, h31, and h32 are the respective elements within the matrix of the projective transformation matrix H. Hereinafter, the coordinates of each vertex of the parking area quadrilateral ABCD are defined as A(x1, y1), B(x2, y2), C(x3, y3), and D(x4, y4). Also, the reference quadrilateral A'B'C'D' is defined as a square with a side length of 1, where the coordinates of each vertex are A'(0, 0), B'(1, 0), C'(1, 1), and D'(0, 1). And for the parking area quadrilateral ABCD and the reference quadrilateral A'B'C'D', vertex A and vertex A', vertex B and vertex B', vertex C and vertex C', and vertex D and vertex D' respectively correspond. However, this is not limited to this. For example, the shape and coordinates of the reference quadrilateral A'B'C'D' are not limited, and the corresponding vertices of the two quadrilaterals are not limited to this either.

[0150] Under the above definition, each element (h11, h12, h13, h21, h22, h23, h31, h32) within the matrix of the projective transformation matrix H can be calculated based on the following equations (2) to (9).

[0151]

Number

[0152] Here, the calculation results of equation (8) are required for the calculations of equations (2) and (5), and the calculation result of equation (9) is required for the calculations of equations (3), (6), and (8). Therefore, it is necessary to calculate them in order from equations that do not depend on other values.

[0153] In step S2103, as shown in FIG. 22(d), the setting processing unit 507 calculates division candidate lines E'F', G1'H1', G2'H2', G3'H3' that divide the reference quadrilateral A'B'C'D' into the number of divisions in the vertical direction "2" and the number of divisions in the horizontal direction "4".

[0154] Here, the reference quadrilateral A'B'C'D' is defined as a square with a side length of 1. Therefore, the coordinates of the endpoints of the division candidate lines are E'(0, 0.5), F'(1, 0.5), G1'(0.25, 0), H1'(0.25, 1), G2'(0.5, 0), H2'(0.5, 1), G3'(0.75, 0), H3'(0.75, 1), respectively.

[0155] In step S2104, the setting processing unit 507 calculates division candidate lines EF, G1H1, G2H2, G3H3 that are projected from the division candidate lines E'F', G1'H1', G2'H2', G3'H3' calculated in step S2103 onto the parking area quadrilateral ABCD using the projective transformation matrix H calculated in step S2102.

[0156] In this way, in the present embodiment, it is possible to directly specify the number of divisions for dividing the parking area quadrilateral 911, and the division candidate lines can also be obtained by a projective transformation matrix (known). Note that the method of recalculating the division candidate lines due to a change in the number of divisions is not limited to a specific method.

[0157] [Embodiment 3] In the above embodiment, a case has been described in which the captured image captured by the imaging device 110 is analyzed and processed by the server 130 to obtain information related to each parking space. However, various processes described as processes performed by the server 130 may be performed by the imaging device 110. In this case, the imaging device 110 may analyze and process the captured image captured by itself or the captured image captured by another imaging device 110 to obtain information related to each parking space.

[0158] Also, the imaging device 110 and the server 130 may be directly connected without going through the network 120 or other devices.

[0159] In the above embodiment, information is presented to the user by display. However, part of the information to be presented may be presented to the user using sound, vibration, etc. instead of or in addition to display.

[0160] In addition to operations on a keyboard or mouse, the operation input to the GUI may be performed, for example, by voice input by the user. In that case, the operation unit 406 will include a device (e.g., a microphone) for voice input. Also, information input may be performed by the user's gesture. In that case, the operation unit 406 will include a device for recognizing gestures. Also, each of the above-described GUI configurations, each of the above-described GUI operation methods, the transition pattern between GUIs, etc. are examples and can be appropriately modified / changed.

[0161] The numerical values, processing timings, processing order, processing entity, data (information) acquisition method / transmission destination / transmission source / storage location, etc. used in each of the above embodiments are given as examples for the purpose of specific explanation and are not intended to be limited to such examples.

[0162] Also, some or all of the above-described embodiments may be appropriately combined and used. Also, some or all of the above-described embodiments may be selectively used.

[0163] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0164] The invention described in this specification includes the following image processing apparatus, image processing method, and computer program. (Item 1) Determining means for determining the number of divisions of the rectangular area based on a part of the rectangular area set according to a user operation on the captured image of the parking lot, Acquiring means for acquiring information related to each divided rectangular area obtained by dividing the rectangular area by the number of divisions as information related to a parking frame An image processing apparatus comprising the same. (Item 2) The image processing apparatus according to Item 1, wherein the determining means determines the number of divisions of the rectangular area based on the size of the parking frame selected according to a user operation among the parking frames at the set parking frame positions for the rectangular area. (Item 3) The image processing apparatus according to Item 1, wherein the determining means determines the number of divisions of the rectangular area based on the size of the parking frame determined from the captured image based on the pixel selected according to a user operation as a pixel of the parking frame within the rectangular area. (Item 4) The image processing apparatus according to Item 1, wherein the determining means determines the number of divisions of the rectangular area based on the size of the parked vehicle determined from the captured image based on the pixel selected according to a user operation as a pixel of the parked vehicle within the rectangular area. (Item 5) The determination means determines the number of divisions of the rectangular region in the direction of a straight line passing through a position within the rectangular region, based on the width of the rectangular region and the width of the part in the direction of the straight line. The image processing apparatus according to item 1, characterized in that. (Item 6) The determination means obtains a dividing line for dividing the rectangular region for each number of divisions of the rectangular region, and determines the number of divisions of the rectangular region in the direction based on the width of the part and the division width by the dividing line in the direction of a straight line passing through a position within the rectangular region. The image processing apparatus according to item 1, characterized in that. (Item 7) The determination means obtains the intersection points of the dividing lines for dividing the rectangular region for each number of divisions of the rectangular region, and determines, as the number of divisions of the rectangular region, the number of divisions corresponding to the intersection points selected based on the positions specified according to the user operation among the intersection points. The image processing apparatus according to item 1, characterized in that. (Item 8) Furthermore, The image processing apparatus according to any one of items 1 to 7, further comprising adjustment means for adjusting the number of divisions according to a user operation. (Item 9) Furthermore, The image processing apparatus according to any one of items 1 to 8, further comprising display control means for superimposing and displaying a dividing line when the rectangular region is divided according to the number of divisions on the captured image. (Item 10) The user operation is input via a GUI displayed by the display control means. The image processing apparatus according to item 9, characterized in that. (Item 11) The display control means displays the number of set parking frames. The image processing apparatus according to item 9, characterized in that. (Item 12) The display control means displays the number of parking frames that can be set. The image processing apparatus according to item 9, characterized in that. (Item 13) The image processing apparatus according to item 9, wherein the display control means displays an error message when the number of parking frames being set exceeds the number of parking frames that can be set. (Item 14) The image processing apparatus according to item 1, wherein the determination means exchanges the number of vertical divisions and the number of horizontal divisions of the rectangular area according to a user operation. (Item 15) The image processing apparatus according to item 6, wherein the determination means recalculates the dividing lines according to the change in the number of divisions. (Item 16) Furthermore, The image processing apparatus according to any one of items 1 to 15, further comprising determination means for cropping, as a crop image, an image at a position defined by the information related to the parking frame from a captured image of a parking lot, and determining whether or not a parked vehicle is shown in the crop image. (Item 17) The image processing apparatus according to any one of items 1 to 16, wherein the determination means acquires a captured image of a parking lot captured by an imaging device via a network. (Item 18) Furthermore, it comprises imaging means, The image processing apparatus according to any one of items 1 to 16, wherein the determination means acquires a captured image of a parking lot captured by the imaging means. (Item 19) An image processing method performed by an image processing apparatus, A determination step in which a determination means of the image processing apparatus determines the number of divisions of the rectangular area based on a part of the rectangular area set according to a user operation on a captured image of a parking lot, An acquisition step in which an acquisition means of the image processing apparatus acquires information related to each divided rectangular area obtained by dividing the rectangular area by the number of divisions as information related to a parking frame An image processing method characterized by comprising the above. (Item 20) A computer program for causing a computer to function as each means of the image processing apparatus according to any one of items 1 to 17.

[0165] The invention is not limited to the above embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the claims are appended to disclose the scope of the invention.

Explanation of Reference Numerals

[0166] 502: Control Unit 505: Analysis Unit 507: Setting Processing Unit

Claims

1. Determination means for determining the number of divisions of the rectangular area based on a part of the rectangular area set according to a user operation on the captured image of the parking lot, and acquisition means for acquiring information related to each divided rectangular area obtained by dividing the rectangular area by the number of divisions as information related to a parking frame An image processing apparatus characterized by comprising:

2. The determination means determines the number of divisions of the rectangular area based on the size of the parking frame selected according to a user operation among the parking frames at the set parking frame positions for the rectangular area. The image processing apparatus according to claim 1.

3. The determination means determines the number of divisions of the rectangular area based on the size of the parking frame determined from the captured image based on the pixels selected according to a user operation as the pixels of the parking frame within the rectangular area. The image processing apparatus according to claim 1.

4. The determination means determines the number of divisions of the rectangular area based on the size of the parked vehicle determined from the captured image based on the pixels selected according to a user operation as the pixels of the parked vehicle within the rectangular area. The image processing apparatus according to claim 1.

5. The determination means determines the number of divisions of the rectangular area in the direction of the straight line passing through the position within the rectangular area based on the width of the rectangular area and the width of the part in the direction of the straight line. The image processing apparatus according to claim 1.

6. The determination means obtains a dividing line for dividing the rectangular area for each number of divisions of the rectangular area, and based on the width of the part and the dividing width by the dividing line in the direction of the straight line passing through the position within the rectangular area, determines the number of divisions of the rectangular area in the direction. The image processing apparatus according to claim 1.

7. The determination means obtains the intersection points of the dividing lines for dividing the rectangular area for each number of divisions of the rectangular area, and determines the number of divisions corresponding to the intersection points selected based on the positions specified according to a user operation among the intersection points as the number of divisions of the rectangular area. The image processing apparatus according to claim 1.

8. Furthermore, The image processing apparatus according to claim 1, further comprising adjustment means for adjusting the number of divisions according to a user operation.

9. Furthermore, The image processing apparatus according to claim 1, further comprising display control means for superimposing and displaying on the captured image a dividing line when the quadrilateral area is divided according to the number of divisions.

10. The image processing apparatus according to claim 9, wherein the user operation is input via a GUI displayed by the display control means.

11. The image processing apparatus according to claim 9, wherein the display control means displays the number of set parking frames.

12. The image processing apparatus according to claim 9, wherein the display control means displays the number of parking frames that can be set.

13. The image processing apparatus according to claim 9, wherein the display control means displays an error message when the number of parking frames being set exceeds the number of parking frames that can be set.

14. The image processing apparatus according to claim 1, wherein the determining means exchanges the number of divisions in the vertical direction and the number of divisions in the horizontal direction of the quadrilateral area according to a user operation.

15. The image processing apparatus according to claim 6, wherein the determining means recalculates the dividing line according to a change in the number of divisions.

16. Furthermore, The image processing apparatus according to claim 1, further comprising determination means for cropping, as a crop image, an image at a position defined by information related to the parking frame from a captured image of a parking lot, and determining whether or not a parked vehicle is shown in the crop image.

17. The image processing apparatus according to claim 1, wherein the determining means acquires a captured image of a parking lot captured by an imaging device via a network.

18. Furthermore, comprising imaging means, The image processing apparatus according to any one of claims 1 to 16, wherein the determining means acquires a captured image of a parking lot captured by the imaging means.

19. An image processing method performed by an image processing apparatus, a determining step in which a determining means of the image processing apparatus determines the number of divisions of a quadrilateral area based on a part of the quadrilateral area set according to a user operation on a captured image of a parking lot; an acquiring step in which an acquiring means of the image processing apparatus acquires, as information related to a parking frame, information related to each divided quadrilateral area obtained by dividing the quadrilateral area by the number of divisions An image processing method characterized by comprising:

20. A computer program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 17.

Images