Person detection method, vehicle control method, person detection program, person detection device, and vehicle control system

Abstract

The present invention provides a person detection method, a vehicle control method, a person detection program, a person detection device, and a vehicle control system that can improve the stability of person segmentation in time-series image data. [Solution] The person detection method includes: obtaining the difference value of time-series data of multiple captured images; executing a first detection process to detect a person based on the later image in the time series among the multiple captured images used to obtain the difference value if the difference value is greater than or equal to a predetermined value; and executing a second detection process to detect a person based on the earlier image in the time series among the multiple captured images used to obtain the difference value if the difference value is less than the predetermined value.

Description

Technical Field

[0001] The present disclosure relates to a person detection method, a vehicle control method, a person detection program, a person detection device, and a vehicle control system.

Background Art

[0002] As described in Patent Document 1, there is known a person tracking method including steps of recognizing and detecting a person to be tracked from a captured image obtained by capturing a monitoring area from above by an imaging unit, and recognizing and tracking the detected person from captured images after the next frame.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Regardless of the presence or absence of a person's movement, the output of person segmentation may decrease due to changes in the way a person appears in the time-series data of images showing the person, resulting in the loss of person segmentation. There is a need to improve the stability of segmentation for time-series data of images.

[0005] In view of such circumstances, an object of the present disclosure is to enhance the stability of person segmentation for time-series data of images.

Means for Solving the Problems

[0006] A person detection method according to one embodiment of the present disclosure includes: obtaining a difference value of time-series data of a plurality of captured images; executing a first detection process to detect a person based on a later image in the time series among the plurality of captured images used to obtain the difference value if the difference value is greater than or equal to a predetermined value; and executing a second detection process to detect a person based on an earlier image in the time series among the plurality of captured images used to obtain the difference value if the difference value is less than the predetermined value.

[0007] A vehicle control method according to one embodiment of the present disclosure includes prohibiting the execution of a predetermined operation of the vehicle when a person detected by the person detection method is present in a predetermined space provided in the vehicle.

[0008] A person detection program according to one embodiment of the present disclosure causes a processor to perform the following actions: obtain a difference value of time-series data of a plurality of captured images; execute a first detection process to detect a person based on the later image in the time series among the plurality of captured images used to obtain the difference value if the difference value is greater than or equal to a predetermined value; and execute a second detection process to detect a person based on the earlier image in the time series among the plurality of captured images used to obtain the difference value if the difference value is less than the predetermined value.

[0009] A person detection device according to one embodiment of the present disclosure includes a control unit. The control unit acquires the difference value of time-series data of a plurality of captured images, and if the difference value is greater than or equal to a predetermined value, it executes a first detection process to detect a person based on the later image in the time series among the plurality of captured images used to acquire the difference value, and if the difference value is less than the predetermined value, it executes a second detection process to detect a person based on the earlier image in the time series among the plurality of captured images used to acquire the difference value.

[0010] A vehicle control system according to one embodiment of the present disclosure comprises a person detection device and a vehicle control device. The vehicle control device prohibits the execution of a predetermined operation of the vehicle when a person detected by the person detection device is present in a predetermined space provided in the vehicle. [Effects of the Invention]

[0011] According to a person detection method, vehicle control method, person detection program, person detection device, and vehicle control system according to one embodiment of the present disclosure, the stability of person segmentation with respect to time-series image data is improved. [Brief explanation of the drawing]

[0012] [Figure 1] This block diagram shows an example configuration of the detection system related to this disclosure. [Figure 2] This is a schematic diagram showing an example of a person detection model configuration. [Figure 3] This flowchart shows an example of the procedure for the person detection method related to this disclosure. [Figure 4] This is a block diagram showing an example configuration of a vehicle control system related to this disclosure. [Figure 5] This flowchart shows an example procedure for the vehicle control method related to this disclosure. [Figure 6] This is a schematic diagram illustrating intrusion detection using a human detection rectangle. [Figure 7] This is a schematic diagram explaining intrusion detection using human-controlled areas. [Modes for carrying out the invention]

[0013] (Example configuration of detection system 1) As shown in Figure 1, the detection system 1 according to this disclosure comprises a camera 10 and a detection device 20. The detection system 1 captures an image of the area to be detected by the camera 10 and analyzes the captured image by the detection device 20 to detect a person present in the area to be detected. The area to be detected is, for example, the interior of a vehicle or the area around a vehicle, but is not limited to these.

[0014] Camera 10 is installed to image the area to be detected. Camera 10 may be installed to image the area to be detected from above. Camera 10 may be configured to image the area to be detected using a fisheye lens. Camera 10 may be configured to image light of various wavelengths, such as visible light or infrared light. Camera 10 may be replaced by radar or a finder, etc.

[0015] The detection device 20 comprises a detection unit 22, a tracking unit 24, and a segmentation unit 26. The detection device 20 may include one or more processors or dedicated circuits to realize the functions of the detection unit 22, the tracking unit 24, and the segmentation unit 26. In this embodiment, the processor is a general-purpose processor or a dedicated processor specialized for a specific process, but is not limited to these. The dedicated circuit may include, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). Each of the detection unit 22, the tracking unit 24, and the segmentation unit 26 may be composed of a separate processor or dedicated circuit. At least two of the detection unit 22, the tracking unit 24, and the segmentation unit 26 may be composed of a single processor or dedicated circuit. The detection unit 22, the tracking unit 24, and the segmentation unit 26 are also collectively referred to as the control unit.

[0016] The detection device 20 may be configured to include a storage unit. The storage unit may be configured to include, for example, a semiconductor memory, a magnetic memory, an optical memory, etc., but is not limited thereto. The storage unit may function as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit may be configured to include an electromagnetic storage medium such as a magnetic disk. The storage unit may be configured to include a non-temporary computer-readable medium. The storage unit stores any information or programs used for the operations of the detection unit 22, the tracking unit 24, and the segmentation unit 26 respectively. The storage unit may store, for example, a system program or an application program, etc. The storage unit may be included in a processor or a dedicated circuit, etc.

[0017] The detection device 20 may be configured to include an interface for communicating information, data, etc. with other components of the detection system 1 such as the camera 10, or an external device. The interface may include a communication module configured to be communicable with other components or an external device via a network. The communication module may be compatible with, for example, mobile communication standards such as 4G (4th Generation) or 5G (5th Generation). The communication module may also be compatible with communication standards such as LAN (Local Area Network). The communication module may be compatible with wired or wireless communication standards. The communication module is not limited thereto and may be compatible with various communication standards. The interface may be configured to be connectable to the communication module. The interface may be provided with terminals compatible with, for example, the RS-232C or RS-485 standards so as to be directly connected to other components of the detection system 1 such as the camera 10 or an external device.

[0018] The detection device 20 may be configured to include an input device that receives input of information, data, etc. from a user of the detection system 1. The input device may be configured to include, for example, a touch panel or touch sensor, or a pointing device such as a mouse. The input device may be configured to include physical keys. The input device may be configured to include a voice input device such as a microphone. The detection device 20 may be configured to be connectable to an external input device. The detection device 20 may be configured to be able to acquire information or data input to the external input device from the external input device.

[0019] The detection device 20 may be configured to include an output device that outputs information, data, etc. to the user. The output device may include, for example, a display device that outputs visual information such as an image, character, or figure. The display device may be configured to include, for example, an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display or an inorganic EL display, or a PDP (Plasma Display Panel), etc. The display device is not limited to these displays and may be configured to include various other types of displays. The display device may be configured to include a light-emitting device such as an LED (Light Emitting Diode) or LD (Laser Diode). The display device may be configured to include various other devices. The output device may include, for example, a voice output device such as a speaker that outputs auditory information such as voice. The output device is not limited to these examples and may include various other devices. The detection device 20 may be configured to be connectable to an external output device. The detection device 20 may be configured to be able to output information or data to the external output device.

[0020] The detection device 20 may be configured to include one or a plurality of server devices that can communicate with each other. The detection device 20 may be realized as a cloud server.

[0021] (Operation example of the detection system 1) In this embodiment, the detection system 1 captures an image of the target area using the camera 10, and the detection device 20 analyzes the captured image to detect a person present in the target area. The detection device 20 may use the detection model 80 illustrated in Figure 2 for detecting a person. The detection model 80 illustrated in Figure 2 is a Mask R-CNN (Region based Convolutional Neural Networks) model.

[0022] The detection device 20 inputs the captured image obtained by the camera 10 from the area to be detected as the input image 70 to the detection model 80. The input image 70 includes pixels in which the person to be detected 71 is captured. The detection model 80 detects the person to be detected 71 in the input image 70 and identifies the person detection rectangle 72. The detection model 80 may include a layer that identifies the person detection rectangle 72 in which the person to be detected 71 is captured. The detection model 80 may also perform RoIAlign to identify the person detection rectangle 72. The detection unit 22 of the detection device 20 may correspond to the layer that identifies the person detection rectangle 72.

[0023] The detection device 20 acquires time-series data of captured images, inputs captured images from multiple time points into the detection model 80, and identifies a human detection rectangle 72 in the captured image at each time point.

[0024] The tracking unit 24 of the detection device 20 associates the person detection rectangles 72 that detect the same person among the person detection rectangles 72 identified in each of the captured images taken at multiple different time points. The tracking unit 24 tracks the person detection rectangles 72 of the same person over time.

[0025] The segmentation unit 26 of the detection device 20 performs segmentation on the image of the person to be detected 71 that is captured within the person detection rectangle 72, and outputs the result as the person region 74 of the person to be detected 71. The detection model 80 may include a layer that performs segmentation. The detection model 80 may also include a classifier.

[0026] Through the above operations, the detection device 20 can detect a person in the captured image, perform segmentation, and obtain the result of identifying the area in which the person is located. The detection model 80 may generate an output image 73 in which the area of ​​the person 74 is superimposed on the input image 70. The detection device 20 may also obtain an image in which the result of identifying the area in which the person is located is superimposed on the captured image.

[0027] Here, the segmentation unit 26 may not be able to perform segmentation stably due to changes in how the person to be detected 71 appears at each time point in the time-series data of the captured image. For example, if the way light hits the person to be detected 71 changes in the captured image at each time point, the segmentation output for the person to be detected 71 may decrease in the captured image at a certain time point, and a segmentation result may not be obtained. Also, if the way the person to be detected 71 appears changes due to the movement of the person to be detected 71 in the captured image at each time point, the segmentation output for the person to be detected 71 may decrease in the captured image at a certain time point, and a segmentation result may not be obtained.

[0028] Therefore, the detection device 20 according to this disclosure differs in how it adopts the results of segmentation of the person to be detected 71 depending on the movement or appearance of the person to be detected 71. The detection device 20 may execute a person detection method including the steps of the flowchart illustrated in Figure 3 in order to realize the above-described process. The person detection method may be implemented as a person detection program to be executed by the processor of the detection device 20. The person detection program may be stored on a non-temporary computer-readable medium.

[0029] The detection device 20 acquires an image captured at time t from the camera 10 (S1).

[0030] The detection unit 22 of the detection device 20 detects a person in the image (S2). The detection unit 22 generates a person detection rectangle surrounding the detected person.

[0031] The tracking unit 24 of the detection device 20 tracks the person in the image (S3). The tracking unit 24 identifies whether the person in the images taken at each of multiple time points is the same person in the time-series data of the captured images, including images taken at a time before time t, and tracks the movement of the person identified as the same person.

[0032] The segmentation unit 26 of the detection device 20 performs segmentation of the detected person (S4).

[0033] The detection device 20 calculates a difference value for each person who appears in both the image captured at time t-1 and the image captured at time t (S5). In other words, the detection device 20 obtains the difference value of time-series data from multiple captured images.

[0034] The detection device 20 may, for example, calculate parameters related to the human detection rectangle for the image captured at time t-1 and the image captured at time t, and calculate the absolute value of the difference between the parameters related to the human detection rectangle between the image captured at time t-1 and the image captured at time t as the difference value. The parameters related to the human detection rectangle may include the width or height of the human detection rectangle, the X coordinate of the left or right end of the human detection rectangle, the Y coordinate of the top or bottom end of the human detection rectangle, the XY coordinates of each of the four corners of the human detection rectangle, or the center coordinate of the human detection rectangle. In other words, the detection device 20 may calculate a difference value based on the coordinates of the captured images.

[0035] The detection device 20 may calculate a difference value based on the pattern of the area enclosed by the human detection rectangle between the image captured at time t-1 and the image captured at time t. The detection device 20 may calculate the average value of the brightness, etc., of the pixels in the area enclosed by the human detection rectangle in both the image captured at time t-1 and the image captured at time t, and calculate the absolute value of the difference between the calculated average values ​​as the difference value based on the pattern. The detection device 20 may calculate the standard deviation of the brightness, etc., of the pixels in the area enclosed by the human detection rectangle in both the image captured at time t-1 and the image captured at time t, and calculate the absolute value of the difference between the calculated standard deviations as the difference value based on the pattern. The detection device 20 may calculate the optical flow of the area enclosed by the human detection rectangle in both the image captured at time t-1 and the image captured at time t, and calculate the absolute value of the difference between the calculated optical flows as the difference value based on the pattern.

[0036] The detection device 20 determines whether the difference value is greater than or equal to a predetermined value (S6). A difference value greater than or equal to a predetermined value means that the way the person appears in the captured image changed as time progressed from t-1 to t. A change in the way the person appears includes a movement in the person's position or a change in the person's posture. A change in the way the person appears includes how light hits the person. The predetermined value may be set as appropriate.

[0037] If the detection device 20 determines that the difference value is greater than or equal to a predetermined value (S6: YES), it performs a first detection process on the segmentation result for the person targeted for difference value determination (S7). The first detection process is a process that uses only the segmentation result of the image captured at the most recent time, i.e., the image captured at time t, as the result of the segmentation. In the first detection process, the results of images captured at previous times, i.e., images captured before time t-1, are not used. The first detection process corresponds to the process of detecting a person based on the image that appears later in the time series among the multiple images used to obtain the difference value.

[0038] If the detection device 20 determines that the difference value is less than a predetermined value, that is, that the difference value is not greater than or equal to a predetermined value (S6:NO), it performs a second detection process on the segmentation result for the person whose difference value is to be determined (S8). The second detection process is a process that adopts the average result of the segmentation execution of images captured up to this point, that is, images captured before time t, as the result of the segmentation execution. The second detection process may be a process that averages the segmentation execution result of the image captured at time t-1 and the segmentation execution result of the image captured at time t. In other words, the second detection process corresponds to a process of detecting a person based on the earlier and later images in the time series from among the multiple images captured to obtain the difference value.

[0039] The captured images used for accumulating the segmentation results do not include images captured at times prior to when the difference value was determined to be greater than or equal to a predetermined value. For example, if the difference value is determined to be greater than or equal to a predetermined value at time tx, the captured images used for accumulating the segmentation results are those captured at time t-x+1 and later. In other words, if the difference value between an image captured at a certain time and an image captured at the time immediately preceding it is determined to be greater than or equal to a predetermined value, the accumulation of segmentation results for images captured before the time when the difference value was determined to be greater than or equal to the predetermined value is reset. In this case, the second detection process corresponds to averaging the segmentation results for the period during which the difference value remains below the predetermined value.

[0040] The second detection process is not limited to averaging the segmentation results of images captured at multiple past time points, as described above. The second detection process may, for example, be a process that accumulates the segmentation results of images captured at past time points. The second detection process may also be a process that assigns weights to each of the segmentation results of images captured at past time points and then averages or accumulates them.

[0041] The detection device 20 terminates the execution of the steps in the flowchart of Figure 3 after executing the steps in S7 or S8.

[0042] As described above, the detection device 20 according to this disclosure adopts the average of past segmentation results when the change in parameters related to the human detection rectangle in captured images at two different time points is small. In this way, even if the detection device 20 loses segmentation results despite small parameter changes, it can obtain segmentation results by using each of the multiple captured images at which the parameter changes become small. As a result, the stability of segmentation is improved.

[0043] In the operation examples described above, the detection device 20 obtained the difference value of the parameters related to the human detection rectangle from the time-series data of multiple captured images. However, it may decide whether to execute the first detection process or the second detection process based on the difference value of the time-series data of multiple captured images.

[0044] The detection device 20 may perform a second detection process after performing a first detection process to detect a person.

[0045] (Example configuration of vehicle control system 2) As shown in Figure 4, the vehicle control system 2 comprises a camera 10, a detection device 20, and a vehicle control device 30.

[0046] The vehicle control device 30 controls the vehicle based on the results of person detection by the camera 10 and the detection device 20. The vehicle control device 30 may include a processor, a memory unit, and an interface, similar to the detection device 20 of the detection system 1 described above. The vehicle is equipped with doors. The vehicle control device 30 controls the opening and closing of the doors.

[0047] Camera 10 and detection device 20 may be configured similarly to Camera 10 and detection device 20 of detection system 1. Camera 10 may be installed inside the vehicle's passenger compartment and image the interior of the passenger compartment as the detection target area. When Camera 10 is installed inside the passenger compartment, it may photograph the passengers of the vehicle. Camera 10 may be installed outside the vehicle and image the area around the vehicle as the detection target area. Detection device 20 may be mounted on the vehicle controlled by the vehicle control device 30. Detection device 20 does not have to be mounted on the vehicle. At least a part of the components of detection device 20 may be mounted on the vehicle. At least a part of the components of detection device 20 does not have to be mounted on the vehicle.

[0048] The vehicle control system 2 may perform a vehicle control method, including the steps of the flowchart illustrated in Figure 5, to detect passengers in the vehicle using the detection device 20 and to control the opening and closing of the vehicle doors using the vehicle control device 30 based on the passenger detection results. The vehicle control method may be implemented as a vehicle control program to be executed by the processor of the vehicle control device 30. The vehicle control program may be stored on a non-temporary computer-readable medium.

[0049] The detection device 20 detects a person from the captured image (S11). The detection device 20 may, for example, perform the steps in the flowchart of Figure 3 to detect a person from the captured image and calculate the person detection rectangle and person region of the detected person. The detection device 20 outputs the calculation results of the person detection rectangle and person region to the vehicle control device 30.

[0050] The vehicle control device 30 calculates the degree of overlap between the person detection rectangle and the first intrusion determination area in the captured image (S12). The first intrusion determination area is an area appropriately set as the area around the vehicle door. The degree of overlap may be the area in the captured image where the person detection rectangle and the first intrusion determination area overlap. The degree of overlap may be the ratio of the area where the person detection rectangle and the first intrusion determination area overlap to the area of ​​the person detection rectangle. The degree of overlap between the person detection rectangle and the first intrusion determination area is also called the first overlap degree.

[0051] Specifically, as shown in Figure 6, the vehicle control device 30 may obtain the calculation result of a person detection rectangle 72 corresponding to the person detected from an image captured showing the person 71, who is a passenger in the vehicle, standing near the vehicle door 3. The vehicle control device 30 may calculate the overlapping area 76 between the first intrusion determination area 75 and the person detection rectangle 72. The vehicle control device 30 may calculate the ratio of the area of ​​the overlapping area 76 to the area of ​​the person detection rectangle 72 as the degree of overlap.

[0052] The vehicle control device 30 determines whether the overlap rate calculated in S12 is less than a threshold (S13). The threshold in S13 is a value that can be set as appropriate and is also called the first overlap threshold. As shown in Figure 6, the vehicle control device 30 may obtain the result of detecting the person to be detected 71 from an image captured showing the person to be detected 71, who is a passenger in the vehicle, standing near the vehicle door 3. The person to be detected 71 has only their arms extended to grasp the handrail 4 located next to the door 3. Therefore, the person detection rectangle 72 is calculated to be wider than the area in which the person to be detected 71 is actually captured. If the vehicle control device 30 calculates the overlap rate as the ratio of the area of ​​the overlapping region 76 to the area of ​​the person detection rectangle 72, the first overlap threshold may be set to, for example, 1 / 9. The value of the first overlap threshold is not limited to the example value and may be set to other values ​​as appropriate.

[0053] If the overlap rate is greater than or equal to a threshold, i.e., if the overlap rate is not less than a threshold (S13: NO), the vehicle control device 30 proceeds to step S20. If the overlap rate is less than a threshold (S13: YES), the vehicle control device 30 extracts the human region of one of the one or more people captured in the image (S14). The vehicle control device 30 calculates the overlap rate between the human region of the one person extracted in S14 and the second intrusion detection area (S15). The second intrusion detection area is an area appropriately set as the area around the vehicle door. The second intrusion detection area may be set to the same area as the first intrusion detection area. The second intrusion detection area may be set to an area that overlaps with the first intrusion detection area in at least part. The second intrusion detection area may be set to an area larger than the first intrusion detection area. The second intrusion detection area may be set to an area smaller than the first intrusion detection area. The overlap rate may be the area in the image where the human region and the second intrusion detection area overlap. The degree of overlap may be the ratio of the area where the human area and the second intrusion detection area overlap to the area of ​​the human area. The degree of overlap between the human area and the second intrusion detection area is also called the second degree of overlap.

[0054] Specifically, as shown in Figure 7, the vehicle control device 30 may obtain a calculation result for the human area 74 corresponding to the detected person 71 as a result of detecting the detected person 71 from an image captured showing the person 71, who is a passenger in the vehicle, standing near the vehicle door 3. The vehicle control device 30 may calculate the area where the second intrusion determination area 77 and the human area 74 overlap. The vehicle control device 30 may calculate the degree of overlap as the ratio of the area of ​​the overlapping area of ​​the second intrusion determination area 77 and the human area 74 to the area of ​​the human area 74.

[0055] The vehicle control device 30 determines whether the overlap rate calculated in S15 is less than a threshold (S16). The threshold in S16 is a value that is set as appropriate and is also called the second overlap threshold. If the overlap rate is greater than or equal to the threshold, i.e., if the overlap rate is not less than the threshold (S16: NO), the vehicle control device 30 proceeds to step S20. If the overlap rate is less than the threshold (S16: YES), the vehicle control device 30 determines that there is no area intrusion by one person corresponding to the extracted person area (S17). The vehicle control device 30 determines whether the person areas 74 of all people in the captured image have been extracted (S18). If the person areas 74 of all people have not been extracted (S18: NO), the vehicle control device 30 returns to step S14 and extracts the person areas 74 of the people that have not yet been extracted. If the vehicle control device 30 has extracted the area 74 for all persons (S18: YES), it determines that there is no area intrusion for any person captured in the image and permits door opening and closing control (S19). After executing the procedure in S19, the vehicle control device 30 terminates the execution of the procedure in the flowchart in Figure 5.

[0056] If the vehicle control device 30 determines in the procedure of S13 or S16 that the degree of overlap is greater than or equal to a threshold, that is, if it determines that the degree of overlap is not less than a threshold (S13 or S16: NO), it determines that at least one person in the captured image has entered the area (S20). If at least one person has entered the area around door 3, opening or closing the vehicle door 3 may cause the person in the area to collide with or be caught in the opening or closing door 3. Therefore, if the vehicle control device 30 determines that at least one person has entered the area, it prohibits the opening and closing of the door (S21). The vehicle control device 30 may also display a message on the screen or output an audio message such as "It is dangerous, please move away from the door" to get the person to move away from the area around door 3 in order to control the opening and closing of the door. After executing the procedure of S21, the vehicle control device 30 finishes executing the procedure in the flowchart of Figure 5.

[0057] As described above, the vehicle control system 2 according to this disclosure can control the opening and closing of a door by determining whether a person has entered a predetermined space, such as the area around the door, based on the result of detecting a person from an captured image by the detection device 20. In this case, the accuracy of determining whether a person has entered a human area based on the human area is improved by enhancing the stability of segmentation by the detection device 20.

[0058] The vehicle control system 2 may prohibit the departure of the vehicle if a person detected by the detection device 20 is present in a predetermined space around the vehicle, and may permit the departure of the vehicle if the detected person is not present in the predetermined space around the vehicle. In other words, the vehicle control system 2 may prohibit the execution of predetermined vehicle operations, such as opening and closing the vehicle doors or departing the vehicle, if a person detected by the detection device 20 is present in a predetermined space provided in the vehicle.

[0059] While embodiments relating to this disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of this disclosure. For example, the functions, etc., included in each means or each step, etc., can be rearranged in a logically consistent manner, and multiple means or steps, etc., can be combined into one or divided.

[0060] In the embodiments described above, the detection device 20 detects a person based on the earlier and later images in the time series from among the multiple captured images used to acquire the difference value. If there is no change in the captured images in the time series, the detection device 20 may detect a person based only on the earlier image and not on the later image. In other words, the detection device 20 may detect a person based on the earlier image in the time series from among the multiple captured images used to acquire the difference value, or it may detect a person based on both the earlier and later images.

[0061] Some embodiments of the present disclosure are described below. However, it should be noted that the embodiments of the present disclosure are not limited to these. [Note 1] Obtaining the difference value of time-series data from multiple captured images, If the difference value is greater than or equal to a predetermined value, a first detection process is performed to detect a person based on the later image in the time series among the multiple captured images used to obtain the difference value. If the difference value is less than the predetermined value, a second detection process is performed to detect a person based on the earlier image in the time series among the multiple captured images used to obtain the difference value. A person detection method, including the following. [Note 2] The person detection method according to [Appendix 1], wherein the second detection process includes detecting a person based on the earlier and later images in the time series from among the plurality of captured images used to obtain the difference value when the difference value is less than the predetermined value, and detecting a person by averaging the person detection result based on the earlier image in the time series and the person detection result based on the later image in the time series from among the plurality of captured images used to obtain the difference value. [Note 3] The person detection method according to [Appendix 2], which includes detecting a person by averaging the person detection results during the period in which the difference value remains below a predetermined value. [Note 4] The person detection method described in any one of [Appendix 1] to [Appendix 3], wherein the difference value is a value based on the coordinates of the captured image. [Note 5] The person detection method described in any one of [Appendix 1] to [Appendix 3], wherein the difference value is a value based on the pattern of the captured image. [Note 6] The second detection process is performed after the first detection process has been performed to detect a person, and is the person detection method described in any one of the [Appendix 1] to [Appendix 5]. [Note 7] A vehicle control method that includes prohibiting the execution of a predetermined operation of the vehicle when a person detected by performing the person detection method described in any one of the persons described in [Appendix 1] to [Appendix 6] is present in a predetermined space provided in the vehicle. [Note 8] Obtaining the difference value of time-series data from multiple captured images, If the difference value is greater than or equal to a predetermined value, a first detection process is performed to detect a person based on the later image in the time series among the multiple captured images used to obtain the difference value. If the difference value is less than the predetermined value, a second detection process is performed to detect a person based on the earlier image in the time series among the multiple captured images used to obtain the difference value. A person detection program that causes the processor to execute. [Note 9] As the second detection process, the person detection program described in [Appendix 8] causes the processor to detect a person based on the earlier and later images in the time series from among the plurality of captured images used to obtain the difference value when the difference value is less than the predetermined value, and to detect a person by averaging the person detection result based on the earlier image in the time series and the person detection result based on the later image in the time series from among the plurality of captured images used to obtain the difference value. [Note 10] The person detection program described in [Appendix 9], which, if the state in which the difference value remains below a predetermined value continues, causes the processor to perform the average of the person detection results during that period and then detect a person. [Note 11] The person detection program described in any one of [Appendix 8] to [Appendix 10], wherein the difference value is a value based on the coordinates of the captured image. [Note 12] The person detection program described in any one of [Appendix 8] to [Appendix 10], wherein the difference value is a value based on the pattern of the captured image. [Note 13] The second detection process is a person detection program described in any one of the [Appendix 8] to [Appendix 12], which is executed after the first detection process has been performed to detect a person. [Note 14] Equipped with a control unit, The control unit, Obtain the difference value of time-series data from multiple captured images, If the difference value is greater than or equal to a predetermined value, a first detection process is executed to detect a person based on the later image in the time series among the multiple captured images used to obtain the difference value. A person detection device that, when the difference value is less than the predetermined value, performs a second detection process to detect a person based on the earlier image in the time series among the plurality of captured images used to obtain the difference value. [Note 15] The control unit, as the second detection process, detects a person based on the earlier and later images in time series from among the plurality of captured images used to obtain the difference value when the difference value is less than the predetermined value, and detects a person by averaging the person detection result based on the earlier image in time series and the person detection result based on the later image in time series from among the plurality of captured images used to obtain the difference value. Person detection device as described in [Appendix 14]. [Note 16] The control unit detects a person by averaging the person detection results during the period in which the difference value remains below a predetermined value. [Appendix 15] [Note 17] The person detection device described in any one of [Appendix 14] to [Appendix 16], wherein the difference value is a value based on the coordinates of the captured image. [Note 18] The person detection device described in any one of [Appendix 14] to [Appendix 16], wherein the difference value is a value based on the pattern of the captured image. [Note 19] The person detection device described in any one of [Appendix 14] to [Appendix 18] is executed after the first detection process has been performed to detect a person. [Note 20] The vehicle comprises a person detection device as described in any one of [Appendix 14] to [Appendix 19] and a vehicle control device, The vehicle control device prohibits the execution of a predetermined operation of the vehicle when a person detected by the person detection device is present in a predetermined space provided in the vehicle. Vehicle control system. [Explanation of Symbols]

[0062] 1. Detection System 2. Vehicle control system 3 doors 4 Handrails 10 Cameras 20 Detection device (22: detection unit, 24: tracking unit, 26: segmentation unit) 30 Vehicle control device 70 Input Images 71 Detection targets 72 people detected in a rectangular area 73 Output image 74 people area 75. First Intrusion Detection Area 76 Overlapping area 77 Second Intrusion Detection Area 80 detection models

Claims

1. Obtaining the difference value of time-series data from multiple captured images, If the difference value is greater than or equal to a predetermined value, a first detection process is performed to detect a person based on the later image in the time series among the multiple captured images used to obtain the difference value. If the difference value is less than the predetermined value, a second detection process is performed to detect a person based on the earlier image in the time series among the multiple captured images used to obtain the difference value. A person detection method, including the following.

2. The person detection method according to claim 1, wherein the second detection process includes, when the difference value is less than the predetermined value, detecting a person based on the earlier and later images in time series from among the plurality of captured images used to obtain the difference value, and detecting a person by averaging the person detection result based on the earlier image in time series and the person detection result based on the later image in time series from among the plurality of captured images used to obtain the difference value.

3. The person detection method according to claim 2, further comprising detecting a person by averaging the person detection results during the period in which the difference value remains below a predetermined value.

4. The person detection method according to any one of claims 1 to 3, wherein the difference value is a value based on the coordinates of the captured image.

5. The person detection method according to any one of claims 1 to 3, wherein the difference value is a value based on the pattern of the captured image.

6. The person detection method according to any one of claims 1 to 3, wherein the second detection process is performed after the first detection process is performed to detect a person.

7. A vehicle control method that includes prohibiting the execution of a predetermined operation of a vehicle when a person detected by performing the person detection method described in any one of claims 1 to 3 is present in a predetermined space provided in the vehicle.

8. Obtaining the difference value of time-series data from multiple captured images, If the difference value is greater than or equal to a predetermined value, a first detection process is performed to detect a person based on the later image in the time series among the multiple captured images used to obtain the difference value. If the difference value is less than the predetermined value, a second detection process is performed to detect a person based on the earlier image in the time series among the multiple captured images used to obtain the difference value. A person detection program that causes the processor to execute.

9. The person detection program according to claim 8, wherein, as the second detection process, if the difference value is less than the predetermined value, the processor is instructed to detect a person based on the earlier and later images in time series from among the plurality of captured images used to obtain the difference value, and to detect a person by averaging the person detection result based on the earlier image in time series and the person detection result based on the later image in time series from among the plurality of captured images used to obtain the difference value.

10. The person detection program according to claim 9, wherein if the state in which the difference value remains below a predetermined value continues, the processor is instructed to detect a person by averaging the person detection results during that period.

11. The person detection program according to any one of claims 8 to 10, wherein the difference value is a value based on the coordinates of the captured image.

12. The person detection program according to any one of claims 8 to 10, wherein the difference value is a value based on the pattern of the captured image.

13. The person detection program according to any one of claims 8 to 10, wherein the second detection process is performed after the first detection process is performed to detect a person.

14. Equipped with a control unit, The control unit, Obtain the difference value of time-series data from multiple captured images, If the difference value is greater than or equal to a predetermined value, a first detection process is executed to detect a person based on the later image in the time series among the multiple captured images used to obtain the difference value. A person detection device that, when the difference value is less than the predetermined value, performs a second detection process to detect a person based on the earlier image in the time series among the plurality of captured images used to obtain the difference value.

15. The person detection device according to claim 14, wherein the control unit, as the second detection process, detects a person based on the earlier and later images in time series from among the plurality of captured images used to obtain the difference value when the difference value is less than the predetermined value, and detects a person by averaging the person detection result based on the earlier image in time series and the person detection result based on the later image in time series from among the plurality of captured images used to obtain the difference value.

16. The person detection device according to claim 15, wherein the control unit detects a person by averaging the person detection results during the period in which the difference value remains below a predetermined value.

17. The person detection device according to any one of claims 14 to 16, wherein the difference value is a value based on the coordinates of the captured image.

18. The person detection device according to any one of claims 14 to 16, wherein the difference value is a value based on the pattern of the captured image.

19. The person detection device according to any one of claims 14 to 16, wherein the second detection process is performed after the first detection process is performed to detect a person.

20. A person detection device according to any one of claims 14 to 16 and a vehicle control device, The vehicle control device prohibits the execution of a predetermined operation of the vehicle when a person detected by the person detection device is present in a predetermined space provided in the vehicle. Vehicle control system.

Images