Person detection method, vehicle control method, non-transitory computer readable medium, person detection device, and vehicle control system

Abstract

This invention relates to a person detection method, a vehicle control method, a non-transitory computer-readable medium, a person detection device, and a vehicle control system, which can improve the stability of person segmentation relative to time-series image data. The person detection method includes the following steps: obtaining difference values ​​of time-series data from multiple captured images; if the difference value is above a predetermined value, performing a first detection process to detect a person based on a later image in the time series from the multiple captured images used to obtain the difference value; and if the difference value is below a predetermined value, performing a second detection process to detect a person based on an earlier image in the time series from the multiple captured images used to obtain the difference value.

Description

Technical Field

[0001] This disclosure relates to a person detection method, a vehicle control method, a non-transitory computer-readable medium, a person detection device, and a vehicle control system. Background Technology

[0002] As described in Patent Document 1, a person tracking method is known, which includes the steps of identifying and detecting a person to be tracked from a captured image obtained by a camera capturing a monitored area from above; and the steps of identifying and tracking the detected person from captured images in subsequent frames.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2024-57695 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] Regardless of whether the person is moving, due to variations in how the person is represented in the time-series data of the image, the segmentation output of the person may sometimes decrease, resulting in the loss of the person segment. Therefore, it is necessary to improve the stability of the segmentation relative to the time-series data of the image.

[0008] The purpose of this disclosure, made in view of the above circumstances, is to improve the stability of person segmentation relative to time-series image data.

[0009] Technical solutions for solving the problem

[0010] One embodiment of the person detection method disclosed herein includes the following steps: obtaining a difference value of time series data of a plurality of captured images; if the difference value is above a predetermined value, performing a first detection process to detect a person based on an image in the time series among the plurality of captured images used to obtain the difference value; and if the difference value is below the predetermined value, performing a second detection process to detect a person based on an image in the time series among the plurality of captured images used to obtain the difference value.

[0011] One embodiment of the vehicle control method disclosed herein includes the following steps: if a person detected by executing the person detection method exists in a predetermined space set in the vehicle, the predetermined operation of the vehicle is prohibited.

[0012] One embodiment of this disclosure provides a non-transitory computer-readable medium storing a person detection program. The person detection program causes a processor to perform the following steps: acquiring difference values ​​of time-series data of a plurality of captured images; if the difference value is above a predetermined value, performing a first detection process to detect a person based on a later image in the time series among the plurality of captured images used to acquire the difference value; and if the difference value is below the predetermined value, performing a second detection process to detect a person based on an earlier image in the time series among the plurality of captured images used to acquire the difference value.

[0013] One embodiment of the person detection apparatus disclosed herein includes a control unit. The control unit performs the following steps: acquiring difference values ​​of time-series data of a plurality of captured images; if the difference value is above a predetermined value, performing a first detection process to detect a person based on a later image in the time series among the plurality of captured images used to acquire the difference value; if the difference value is below the predetermined value, performing a second detection process to detect a person based on an earlier image in the time series among the plurality of captured images used to acquire the difference value.

[0014] One embodiment of the vehicle control system disclosed herein includes the aforementioned person detection device and vehicle control device. If the person detected by the person detection device is present in a predetermined space within the vehicle, the vehicle control device prohibits the execution of predetermined operations of the vehicle.

[0015] Invention Effects

[0016] According to one embodiment of the present disclosure, a person detection method, a vehicle control method, a non-transitory computer-readable medium, a person detection device, and a vehicle control system can improve the stability of person segmentation relative to the time-series data of images. Attached Figure Description

[0017] Figure 1 This is a block diagram illustrating an example of the structure of the detection system of this disclosure.

[0018] Figure 2 This is a schematic diagram illustrating the structure of a person detection model.

[0019] Figure 3 This is a flowchart illustrating an example of the person detection method disclosed herein.

[0020] Figure 4 This is a block diagram illustrating an example of the structure of the vehicle control system of this disclosure.

[0021] Figure 5 This is a flowchart illustrating an example of the vehicle control method of this disclosure.

[0022] Figure 6 This is a schematic diagram illustrating the intrusion determination of a human-detected rectangle.

[0023] Figure 7 This is a schematic diagram illustrating the intrusion determination process for a user's area.

[0024] (Symbol Explanation)

[0025] 1: Detection system; 2: Vehicle control system; 3: Door; 4: Handrail; 10: Camera; 20: Detection device (22: Detection unit; 24: Tracking unit; 26: Segmentation unit); 30: Vehicle control device; 70: Input image; 71: Detection object; 72: Person detection rectangle; 73: Output image; 74: Person area; 75: First intrusion determination area; 76: Overlapping area; 77: Second intrusion determination area; 80: Detection model. Detailed Implementation

[0026] (Example of the structure of detection system 1)

[0027] like Figure 1 As shown, the detection system 1 of this disclosure includes a camera 10 and a detection device 20. The detection system 1 uses the camera 10 to capture images of the target area and uses the detection device 20 to analyze the captured images to detect people present in the target area. The target area may be, for example, the interior area of ​​a vehicle or the area surrounding the vehicle, but is not limited thereto.

[0028] Camera 10 is configured to capture images of the target area. Camera 10 can be configured to capture images of the target area from above. Camera 10 can also be configured to use a fisheye lens to capture images of the target area. Camera 10 can be configured to capture light of various wavelengths, such as visible light or infrared light. Camera 10 can also be replaced by radar or a viewfinder.

[0029] The detection device 20 includes a detection unit 22, a tracking unit 24, and a segmentation unit 26. To realize the functions of the detection unit 22, tracking unit 24, and segmentation unit 26, the detection device 20 can be configured to include one or more processors or dedicated circuits. In this embodiment, the processor can be a general-purpose processor or a dedicated processor specifically designed for a particular process, but is not limited to these. Dedicated circuits can include, for example, FPGAs (Field-Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits). The detection unit 22, tracking unit 24, and segmentation unit 26 can each be composed of different processors or dedicated circuits. Alternatively, at least two of the detection unit 22, tracking unit 24, and segmentation unit 26 can be composed of a single processor or dedicated circuit. The detection unit 22, tracking unit 24, and segmentation unit 26 are collectively referred to as the control unit.

[0030] The detection device 20 may be configured to include a storage unit. The storage unit may include, for example, semiconductor memory, magnetic memory, or optical memory, but is not limited to these. The storage unit may also function as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit may be configured to include an electromagnetic storage medium such as a magnetic disk. The storage unit may be configured to include a non-transitory computer-readable medium. The storage unit stores any information or programs used in the operation of each of the detection unit 22, tracking unit 24, and segmentation unit 26. The storage unit may store, for example, system programs or application programs. The storage unit may also be included in a processor or dedicated circuitry.

[0031] The detection device 20 may be configured to include an interface for communicating information or data with other components of the detection system 1, such as the camera 10, or external devices. The interface may include a communication module configured to communicate with other components or external devices via a network. The communication module may correspond to mobile communication standards such as 4G (4th Generation) or 5G (5th Generation). The communication module may also correspond to communication standards such as LAN (Local Area Network). The communication module may also correspond to wired or wireless communication standards. The communication module is not limited to these and may correspond to various communication standards. The interface may also be configured to connect to the communication module. In order to directly connect with other components of the detection system 1, such as the camera 10, or external devices, the interface may also have terminals corresponding to standards such as RS-232C or RS-485.

[0032] The detection device 20 can be configured as an input device that includes input such as user-received information or data from the detection system 1. The input device can be configured as, for example, a touch panel, a contact sensor, or a pointing device such as a mouse. The input device can also be configured to include physical keys. The input device can also be configured to include a voice input device such as a microphone. The detection device 20 can be configured to connect to an external input device. The detection device 20 can be configured to obtain information or data input to an external input device.

[0033] The detection device 20 can be configured as an output device that outputs information or data to a user. The output device may include, for example, a display device that outputs visual information such as images, text, or graphics. The display device may include, for example, an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display, an inorganic EL display, or a PDP (Plasma Display Panel). The display device is not limited to these displays and can be configured as a display of various other types. The display device may be configured using light-emitting devices such as LEDs (Light Emitting Diodes) or LDs (Laser Diodes). The display device may include various other devices. The output device may include, for example, a sound output device such as a speaker that outputs auditory information such as sound. The output device is not limited to these examples and may include various other devices. The detection device 20 can be configured to connect to an external output device. The detection device 20 can be configured to output information or data to an external output device.

[0034] The detection device 20 can be configured to include one or more server devices capable of communicating with each other. The detection device 20 can also be implemented as a cloud server.

[0035] (Example of the action of detection system 1)

[0036] The detection system 1 of this embodiment uses a camera 10 to capture images of the target area and a detection device 20 to analyze the captured images to detect people present in the target area. The detection device 20, for detecting people, can be used in… Figure 2 The example shown is detection model 80. In... Figure 2 The detection model 80 shown in the example is a Mask R-CNN (Region based Convolutional Neural Networks) model.

[0037] The detection device 20 inputs an image 70, obtained by capturing the target area using the camera 10, to the detection model 80. The input image 70 includes pixels reflecting the target person 71. The detection model 80 detects the target person 71 reflected in the input image 70 and determines a person detection rectangle 72. The detection model 80 may include a layer that determines the person detection rectangle 72 reflecting the target person 71. The detection model 80 may also perform RoIAlign (region alignment algorithm) to determine the person detection rectangle 72. The detection unit 22 of the detection device 20 may correspond to the layer that determines the person detection rectangle 72.

[0038] The detection device 20 acquires time-series data of the captured images, inputs the captured images at multiple times into the detection model 80, and determines the human detection rectangle 72 in the captured images at each time.

[0039] The tracking unit 24 of the detection device 20 matches the person detection rectangles 72 that detect the same person in images captured at different times. The tracking unit 24 tracks the person detection rectangles 72 of the same person as time passes.

[0040] The segmentation unit 26 of the detection device 20 performs segmentation on the image of the person to be detected 71, which is now in the person detection rectangle 72, and outputs the result as the person region 74 of the person to be detected 71. The detection model 80 may include a layer that performs segmentation. The detection model 80 may also include a category classifier.

[0041] Through the above actions, the detection device 20 detects people from the captured image and performs segmentation, obtaining a result that identifies the region containing the person. The detection model 80 can also generate an output image 73 that overlays the person region 74 onto the input image 70. The detection device 20 can also obtain an image obtained by overlaying the result of identifying the region containing the person in the captured image onto the captured image.

[0042] Here, due to changes in the appearance of the detected person 71 at different times in the time-series data of the captured images, the segmentation unit 26 sometimes cannot perform segmentation stably. For example, when the illumination pattern of the detected person 71 changes in the captured images at different times, the segmentation output of the detected person 71 may decrease in a certain captured image, resulting in no segmentation result. In addition, when the manifestation pattern of the detected person 71 changes due to its movement in the captured images at different times, the segmentation output of the detected person 71 may decrease in a certain captured image, resulting in no segmentation result.

[0043] Therefore, the detection device 20 of this disclosure uses the segmentation results of the detection object 71 in different ways according to the movement or appearance changes of the detection object 71. To achieve the above processing, the detection device 20 can perform actions including... Figure 3 The flowchart illustrated herein illustrates a person detection method. The person detection method can be implemented as a person detection program executed by the processor of the detection device 20. The person detection program can be stored on a non-transitory computer-readable medium.

[0044] The detection device 20 acquires the image captured at time t from the camera 10 (S1).

[0045] The detection unit 22 of the detection device 20 detects people projected onto the image (S2). The detection unit 22 generates a person detection rectangle surrounding the detected person.

[0046] The tracking unit 24 of the detection device 20 tracks the person reflected in the image (S3). The tracking unit 24 determines whether the person reflected in the images taken at multiple times is the same person in the time series data of the captured images, which includes images taken at times before time t, and tracks the movement of the person determined to be the same person.

[0047] The segmentation unit 26 of the detection device 20 performs segmentation of the detected person (S4).

[0048] The detection device 20 calculates a difference value (S5) for each person reflected in the images captured at time t-1 and time t, based on the images captured at time t-1 and time t. That is, the detection device 20 obtains the difference values ​​of the time series data of multiple captured images.

[0049] The detection device 20 can, for example, calculate parameters related to the human detection rectangle for the images captured at time t-1 and time t respectively, and calculate the absolute value of the difference between the parameters related to the human detection rectangle between the images captured at time t-1 and time t as a difference value. Parameters related to the human detection rectangle may include the width or height of the human detection rectangle, the X-coordinate of the left or right end of the human detection rectangle, the Y-coordinate of the top or bottom end of the human detection rectangle, the XY coordinates of each of the four corners of the human detection rectangle, or the center coordinates of the human detection rectangle, etc. That is, the detection device 20 can calculate the difference value based on the coordinates of the captured images.

[0050] The detection device 20 can also calculate the difference value between the image captured at time t-1 and the image captured at time t, based on the pattern of the region enclosed by the human detection rectangle. The detection device 20 can calculate the average value of the brightness, etc., of the pixels in the region enclosed by the human detection rectangle in each of the images captured at time t-1 and time t, and calculate the absolute value of the difference between the calculated average values ​​as a pattern-based difference value. The detection device 20 can calculate the standard deviation of the brightness, etc., of the pixels in the region enclosed by the human detection rectangle in each of the images captured at time t-1 and time t, and calculate the absolute value of the difference between the calculated standard deviations as a pattern-based difference value. The detection device 20 can calculate the optical flow of the region enclosed by the human detection rectangle in each of the images captured at time t-1 and time t, and calculate the absolute value of the difference between the calculated optical flow as a pattern-based difference value.

[0051] The detection device 20 determines whether the difference value is greater than or equal to a predetermined value (S6). A difference value greater than or equal to a predetermined value means that the way the person in the captured image is projected has changed as time progresses from t-1 to t. The change in the way the person is projected includes a change in the person's position or a change in the person's posture. The change in the way the person is projected includes the way light illuminates the person. The predetermined value can be set appropriately.

[0052] If the detection device 20 determines that the difference value is above a predetermined value (S6: Yes), it performs a first detection process (S7) regarding the segmentation result for the person being judged based on the difference value. The first detection process uses only the segmentation result from the image captured at the most recent time, i.e., the image captured at time t, as the segmentation result. In the first detection process, the results from images captured at previous times, i.e., images captured before time t-1, are not used. The first detection process is equivalent to detecting a person based on the later image in the time series among the multiple images used to obtain the difference value.

[0053] If the detection device 20 determines that the difference value is lower than a predetermined value, that is, if it determines that the difference value is not higher than the predetermined value (S6: No), it performs a second detection process (S8) regarding the segmentation execution result for the person being judged as the difference value. The second detection process uses the result obtained by averaging the segmentation execution results from previously captured images, i.e., images captured before time t, as the segmentation execution result. The second detection process can be an averaging process between the segmentation execution result in the image captured at time t-1 and the segmentation execution result in the image captured at time t. In other words, the second detection process is equivalent to detecting a person based on the earlier and later images in the time sequence among the multiple captured images used to obtain the difference value.

[0054] The images included in the accumulation of segmentation execution results do not include images captured at times earlier than when the difference value was determined to be above a predetermined value. For example, if the difference value is determined to be above a predetermined value at time t-x, the images included in the accumulation of segmentation execution results are images captured at time t-x+1 onwards. In other words, if the difference between an image captured at a certain time and an image captured at a previous time is determined to be above a predetermined value, the accumulation of segmentation execution results for images captured before the time when the difference value was determined to be above the predetermined value is reset. In this case, the second detection process is equivalent to averaging the segmentation execution results during the period when the difference value is below the predetermined value.

[0055] The second detection process is not limited to averaging the segmentation results from images captured at multiple past times, as described above. Alternatively, the second detection process could also involve accumulating the segmentation results from images captured at past times. Furthermore, the second detection process could involve weighted averaging or accumulating the segmentation results from images captured at past times.

[0056] After executing process S7 or S8, the detection device 20 terminates. Figure 3 The execution of the flowchart process.

[0057] As described above, when the changes in parameters related to the human detection rectangle in the images captured at two different times are small, the detection device 20 of this disclosure uses the average of past segmentation results. By doing so, even if the segmentation result is lost despite small parameter changes, the detection device 20 can obtain a segmentation result by using multiple captured images where the parameter changes are small. As a result, the stability of the segmentation can be improved.

[0058] In the above example of operation, the detection device 20 obtains the difference values ​​of parameters related to the human detection rectangle from the time series data of multiple captured images. However, it can also determine whether to perform the first detection process or the second detection process based on the difference values ​​of the time series data of multiple captured images.

[0059] The detection device 20 can also perform the second detection process after performing the first detection process and detecting a person.

[0060] (Example of the structure of vehicle control system 2)

[0061] like Figure 4 As shown, the vehicle control system 2 includes a camera 10, a detection device 20, and a vehicle control device 30.

[0062] The vehicle control unit 30 controls the vehicle based on the detection results of people detected by the camera 10 and the detection device 20. Like the detection device 20 of the detection system 1 described above, the vehicle control unit 30 may include a processor, storage unit, and interface. The vehicle has doors. The vehicle control unit 30 controls the opening and closing of the doors.

[0063] The camera 10 and detection device 20 can be configured similarly to those of the detection system 1. The camera 10 can be installed inside the vehicle's passenger compartment, capturing images of the interior as the detection target area. When installed inside the passenger compartment, the camera 10 can photograph the vehicle's passengers. The camera 10 can also be installed outside the vehicle, capturing images of the vehicle's perimeter as the detection target area. The detection device 20 can also be mounted on the vehicle that is controlled by the vehicle control device 30. Alternatively, the detection device 20 may not be mounted on the vehicle. At least a portion of the structure of the detection device 20 may be mounted on the vehicle. Alternatively, at least a portion of the structure of the detection device 20 may not be mounted on the vehicle.

[0064] In order to detect passengers in the vehicle via the detection device 20 and control the opening and closing of the vehicle doors via the vehicle control device 30 based on the passenger detection results, the vehicle control system 2 may perform actions including... Figure 5 The flowchart illustrated herein illustrates a vehicle control method. The vehicle control method can be implemented as a vehicle control program executed by the processor of the vehicle control device 30. The vehicle control program can be stored on a non-transitory computer-readable medium.

[0065] The detection device 20 detects people from the captured image (S11). The detection device 20 may, for example, perform... Figure 3 The flowchart describes a process for detecting people from captured images, calculating the person detection rectangle and person region for each detected person. The detection device 20 outputs the calculated results of the person detection rectangle and person region to the vehicle control device 30.

[0066] The vehicle control device 30 calculates the overlap degree between the person detection rectangle and the first intrusion determination region in the captured image (S12). The first intrusion determination region is an area appropriately set as the area surrounding the vehicle's door. The overlap degree can be the area where the person detection rectangle and the first intrusion determination region overlap in the captured image. The overlap degree can be the ratio of the area where the person detection rectangle and the first intrusion determination region overlap to the area of ​​the person detection rectangle. The overlap degree between the person detection rectangle and the first intrusion determination region is also called the first overlap degree.

[0067] Specifically, such as Figure 6As shown, based on the result of detecting the passenger 71 (i.e., the detection target person 71) from an image captured in which the passenger of the vehicle is standing near the vehicle door 3, the vehicle control device 30 can obtain the calculation result of the person detection rectangle 72 corresponding to the detection target person 71. The vehicle control device 30 can calculate the overlapping area 76 between the first intrusion determination area 75 and the person detection rectangle 72. The vehicle control device 30 can calculate the overlap degree as the ratio of the area of ​​the overlapping area 76 to the area of ​​the person detection rectangle 72.

[0068] The vehicle control unit 30 determines whether the overlap calculated in S12 is lower than a threshold (S13). The threshold in S13 is an appropriately set value, also known as the first overlap threshold. Figure 6 As shown, the vehicle control device 30 can obtain the result of detecting the subject 71 from an image captured of the current vehicle passenger, i.e., the subject 71, standing near the vehicle door 3. The subject 71 extends only his arm to grasp the handrail 4 located next to the door 3. Therefore, the person detection rectangle 72 is calculated to be wider than the actual area of ​​the current subject 71. When calculating the overlap degree as the ratio of the area of ​​the overlapping region 76 to the area of ​​the person detection rectangle 72, the vehicle control device 30 can set the first overlap threshold to, for example, 1 / 9. The value of the first overlap threshold is not limited to the illustrated value and can be appropriately set to other values.

[0069] If the overlap is above a threshold, i.e., if the overlap is not below a threshold (S13: No), the vehicle control device 30 proceeds to process S20. If the overlap is below a threshold (S13: Yes), the vehicle control device 30 extracts the human region of one person from one or more people appearing in the captured image (S14). The vehicle control device 30 calculates the overlap between the human region of the person extracted in S14 and the second intrusion determination region (S15). The second intrusion determination region is an area appropriately set as the area surrounding the vehicle's door. The second intrusion determination region can be set to be the same area as the first intrusion determination region. The second intrusion determination region can be set to be an area that overlaps with the first intrusion determination region in at least a portion. The second intrusion determination region can be set to be an area larger than the first intrusion determination region. The second intrusion determination region can be set to be an area smaller than the first intrusion determination region. The overlap can be the area in the captured image where the human region overlaps with the second intrusion determination region. The overlap ratio can be the ratio of the area of ​​overlap between the human region and the second intrusion detection region to the area of ​​the human region. The overlap ratio between the human region and the second intrusion detection region is also called the second overlap ratio.

[0070] Specifically, such as Figure 7As shown, based on the result of detecting the passenger 71 (i.e., the detection target 71) from an image captured of the vehicle standing near the door 3 of the vehicle, the vehicle control device 30 can obtain the calculation result of the person region 74 corresponding to the detection target 71. The vehicle control device 30 can calculate the area where the second intrusion determination area 77 overlaps with the person region 74. The vehicle control device 30 can calculate the overlap degree as the ratio of the area of ​​the area where the second intrusion determination area 77 overlaps with the person region 74 to the area of ​​the person region 74.

[0071] The vehicle control device 30 determines whether the overlap calculated in S15 is lower than a threshold (S16). The threshold in S16 is an appropriately set value, also known as the second overlap threshold. If the overlap is above the threshold, that is, if the overlap is not lower than the threshold (S16: No), the vehicle control device 30 proceeds to process S20. If the overlap is lower than the threshold (S16: Yes), the vehicle control device 30 determines that one person corresponding to the extracted person area has no intrusion area (S17). The vehicle control device 30 determines whether the extraction of the person areas 74 of all people appearing in the captured image is complete (S18). If the extraction of the person areas 74 of all people is not complete (S18: No), the vehicle control device 30 returns to process S14 and extracts the person areas 74 of the unextracted persons. If the extraction of the person areas 74 of all people is complete (S18: Yes), the vehicle control device 30 determines that none of the people appearing in the captured image have intrusion areas, and permits the opening and closing control of the door (S19). After executing S19, the vehicle control device 30 ends. Figure 5 The execution of the flowchart process.

[0072] If, during process S13 or S16, the overlap is determined to be above a threshold, i.e., if the overlap is determined to be not below a threshold (S13 or S16: No), the vehicle control device 30 determines that at least one person has intruded into the area shown in the captured image (S20). If at least one person has intruded into the area surrounding the door 3, there is a risk that the person intruding may collide with or be caught in the door 3 due to its opening or closing. Therefore, the vehicle control device 30 prohibits door opening and closing control when it determines that at least one person has intruded into the area (S21). The vehicle control device 30 may also display a message such as "Danger, please stay away from the door" on the screen or output it as an audio signal to allow people to leave the area surrounding the door 3. After executing process S21, the vehicle control device 30 terminates the process. Figure 5 The execution of the flowchart process.

[0073] As described above, the vehicle control system 2 of this disclosure can determine whether a person has intruded into a predetermined space, such as the area surrounding a door, based on the result of the detection device 20 detecting a person in an image captured by the detection device 20, and control the opening and closing of the door. In this case, by improving the stability of the segmentation performed by the detection device 20, the accuracy of determining the intrusion area based on the person's region is improved.

[0074] The vehicle control system 2 can also prevent the vehicle from departing if the detection device 20 detects that a person is present in a predetermined space around the vehicle, and allow the vehicle to depart if the detected person is not present in the predetermined space around the vehicle. That is, the vehicle control system 2 can prevent the execution of predetermined vehicle operations such as opening and closing vehicle doors or departing the vehicle if the detection device 20 detects that a person is present in a predetermined space set in the vehicle.

[0075] The embodiments of this disclosure have been described based on the accompanying drawings and examples. However, it should be noted that those skilled in the art can make various modifications and changes based on this disclosure. Therefore, it should be understood that these modifications and changes are included within the scope of this disclosure. For example, the functions included in each unit or step can be reconfigured in a logically consistent manner, and multiple units or steps can be combined into one or divided.

[0076] In the above embodiment, the detection device 20 detects people based on the earlier and later images in the time series among the multiple captured images used to obtain the difference value. When the captured images do not change in the time series, the detection device 20 can detect people based solely on the earlier images without relying on the later images. That is, the detection device 20 can detect people either based on the earlier image in the time series among the multiple captured images used to obtain the difference value, or based on both the earlier and later images.

[0077] The following examples illustrate some embodiments of the present disclosure. However, it should be noted that the embodiments of the present disclosure are not limited to these.

[0078] [Appendix 1]

[0079] A method for detecting people includes the following steps:

[0080] Obtain the difference values ​​of time-series data from multiple captured images;

[0081] If the difference value is above a predetermined value, a first detection process is performed to detect a person based on the later image in the time series among the plurality of captured images used to obtain the difference value; and

[0082] If the difference value is lower than the predetermined value, a second detection process is performed to detect a person based on the image that is earlier in the time series among the plurality of captured images used to obtain the difference value.

[0083] [Appendix 2]

[0084] According to the character detection method described in [Appendix 1], wherein,

[0085] The second detection process includes the following steps: when the difference value is lower than the predetermined value, detecting a person based on the earlier and later images in the time series among the plurality of captured images used to obtain the difference value, and averaging the detection results of the person based on the earlier image in the time series among the plurality of captured images used to obtain the difference value with the detection results of the person based on the later image in the time series to detect a person.

[0086] [Appendix 3]

[0087] The character detection method described in [Appendix 2] includes the following steps:

[0088] If the difference value remains below a predetermined value, the detection results of the person during that period are averaged to detect the person.

[0089] [Appendix 4]

[0090] According to any one of [Appendix 1] to [Appendix 3], the person detection method, wherein,

[0091] The difference value is based on the coordinates of the captured image.

[0092] [Appendix 5]

[0093] According to any one of [Appendix 1] to [Appendix 3], the person detection method, wherein,

[0094] The difference value is a value based on the style of the captured image.

[0095] [Appendix 6]

[0096] According to any one of [Appendix 1] to [Appendix 5], the person detection method, wherein,

[0097] After the first detection process is performed and a person is detected, the second detection process is performed.

[0098] [Appendix 7]

[0099] A vehicle control method includes the following steps:

[0100] If a person is detected by executing any of the person detection methods described in [Appendix 1] to [Appendix 6] and is present in a predetermined space set up in the vehicle, the predetermined operation of the vehicle shall be prohibited.

[0101] [Appendix 8]

[0102] A non-transitory computer-readable medium storing a person detection program, the person detection program causing a processor to perform the following steps:

[0103] Obtain the difference values ​​of time-series data from multiple captured images;

[0104] If the difference value is above a predetermined value, a first detection process is performed to detect a person based on the later image in the time series among the plurality of captured images used to obtain the difference value; and

[0105] If the difference value is lower than the predetermined value, a second detection process is performed to detect a person based on the image that is earlier in the time series among the plurality of captured images used to obtain the difference value.

[0106] [Appendix 9]

[0107] According to the non-transitory computer-readable medium described in [Appendix 8], wherein,

[0108] As part of the second detection process, the person detection program causes the processor to perform the following steps: when the difference value is lower than the predetermined value, detecting a person based on the earlier and later images in the time series among the plurality of captured images used to obtain the difference value, and averaging the detection results of the person based on the earlier image in the time series among the plurality of captured images used to obtain the difference value with the detection results of the person based on the later image in the time series to detect a person.

[0109] [Postscript 10]

[0110] According to the non-transitory computer-readable medium described in [Appendix 9], wherein,

[0111] The character detection program causes the processor to perform the following steps: while the state where the difference value is lower than a predetermined value continues, the detection results of characters during the continuous period are averaged to detect characters.

[0112] [Postscript 11]

[0113] The non-transitory computer-readable medium according to any one of [Appendix 8] to [Appendix 10], wherein,

[0114] The difference value is based on the coordinates of the captured image.

[0115] [Postscript 12]

[0116] The non-transitory computer-readable medium according to any one of [Appendix 8] to [Appendix 10], wherein,

[0117] The difference value is a value based on the style of the captured image.

[0118] [Postscript 13]

[0119] The non-transitory computer-readable medium according to any one of [Appendix 8] to [Appendix 12], wherein,

[0120] After the first detection process is performed and a person is detected, the second detection process is performed.

[0121] [Postscript 14]

[0122] A person detection device, comprising a control unit,

[0123] The control unit performs the following steps:

[0124] Obtain the difference values ​​of time-series data from multiple captured images.

[0125] If the difference value is above a predetermined value, a first detection process is performed to detect a person based on the later image in the time series among the plurality of captured images used to obtain the difference value.

[0126] If the difference value is lower than the predetermined value, a second detection process is performed to detect a person based on the image that is earlier in the time series among the plurality of captured images used to obtain the difference value.

[0127] [Postscript 15]

[0128] According to the person detection device described in [Appendix 14], wherein,

[0129] As part of the second detection process, the control unit performs the following steps: when the difference value is lower than the predetermined value, it detects people based on the earlier and later images in the time series among the plurality of captured images used to obtain the difference value, and detects people by averaging the detection results of people based on the earlier images in the time series among the plurality of captured images used to obtain the difference value and the detection results of people based on the later images in the time series.

[0130] [Postscript 16]

[0131] According to the person detection device described in [Appendix 15], wherein,

[0132] If the difference value remains below a predetermined value, the control unit averages the detection results of the person during that period to detect the person.

[0133] [Postscript 17]

[0134] The person detection device according to any one of [Appendix 14] to [Appendix 16], wherein,

[0135] The difference value is based on the coordinates of the captured image.

[0136] [Postscript 18]

[0137] The person detection device according to any one of [Appendix 14] to [Appendix 16], wherein,

[0138] The difference value is a value based on the style of the captured image.

[0139] [Postscript 19]

[0140] The person detection device according to any one of [Appendix 14] to [Appendix 18], wherein,

[0141] After the first detection process is performed and a person is detected, the second detection process is performed.

[0142] [Postscript 20]

[0143] A vehicle control system,

[0144] Equipped with a person detection device and a vehicle control device as described in any one of [Appendix 14] to [Appendix 19],

[0145] If a person detected by the person detection device is present in a predetermined space set in the vehicle, the vehicle control device prohibits the execution of the vehicle's predetermined operation.

Claims

1. A method for detecting people, comprising the following steps: Obtain the difference values ​​of time-series data from multiple captured images; If the difference value is above a predetermined value, a first detection process is performed to detect a person based on the later image in the time series among the plurality of captured images used to obtain the difference value; as well as If the difference value is lower than the predetermined value, a second detection process is performed to detect a person based on the image that is earlier in the time series among the plurality of captured images used to obtain the difference value.

2. The person detection method according to claim 1, wherein, The second detection process includes the following steps: when the difference value is lower than the predetermined value, detecting a person based on the earlier and later images in the time series among the plurality of captured images used to obtain the difference value, and averaging the detection results of the person based on the earlier image in the time series among the plurality of captured images used to obtain the difference value with the detection results of the person based on the later image in the time series to detect a person.

3. The person detection method according to claim 2, wherein, Includes the following steps: If the difference value remains below a predetermined value, the detection results of the person during that period are averaged to detect the person.

4. The person detection method according to any one of claims 1 to 3, wherein, The difference value is based on the coordinates of the captured image.

5. The person detection method according to any one of claims 1 to 3, wherein, The difference value is a value based on the style of the captured image.

6. The person detection method according to any one of claims 1 to 3, wherein, After the first detection process is performed and a person is detected, the second detection process is performed.

7. A vehicle control method, comprising the following steps: If a person is detected by performing the person detection method according to any one of claims 1 to 3 and is present in a predetermined space set in the vehicle, the predetermined operation of the vehicle shall be prohibited.

8. A non-transitory computer-readable medium storing a person detection program, the person detection program causing a processor to perform the following steps: Obtain the difference values ​​of time-series data from multiple captured images; If the difference value is above a predetermined value, a first detection process is performed to detect a person based on the later image in the time series among the plurality of captured images used to obtain the difference value; as well as If the difference value is lower than the predetermined value, a second detection process is performed to detect a person based on the image that is earlier in the time series among the plurality of captured images used to obtain the difference value.

9. The non-transitory computer-readable medium according to claim 8, wherein, As part of the second detection process, the person detection program causes the processor to perform the following steps: when the difference value is lower than the predetermined value, detecting a person based on the earlier and later images in the time series among the plurality of captured images used to obtain the difference value, and averaging the detection results of the person based on the earlier image in the time series among the plurality of captured images used to obtain the difference value with the detection results of the person based on the later image in the time series to detect a person.

10. The non-transitory computer-readable medium according to claim 9, wherein, The character detection program causes the processor to perform the following steps: while the state where the difference value is lower than a predetermined value continues, the detection results of characters during the continuous period are averaged to detect characters.

11. The non-transitory computer-readable medium according to any one of claims 8 to 10, wherein, The difference value is based on the coordinates of the captured image.

12. The non-transitory computer-readable medium according to any one of claims 8 to 10, wherein, The difference value is a value based on the style of the captured image.

13. The non-transitory computer-readable medium according to any one of claims 8 to 10, wherein, After the first detection process is performed and a person is detected, the second detection process is performed.

14. A person detection device, comprising a control unit, The control unit performs the following steps: Obtain the difference values ​​of time-series data from multiple captured images. If the difference value is above a predetermined value, a first detection process is performed to detect a person based on the later image in the time series among the plurality of captured images used to obtain the difference value. If the difference value is lower than the predetermined value, a second detection process is performed to detect a person based on the image that is earlier in the time series among the plurality of captured images used to obtain the difference value.

15. The person detection device according to claim 14, wherein, As part of the second detection process, the control unit performs the following steps: when the difference value is lower than the predetermined value, it detects people based on the earlier and later images in the time series among the plurality of captured images used to obtain the difference value, and detects people by averaging the detection results of people based on the earlier images in the time series among the plurality of captured images used to obtain the difference value and the detection results of people based on the later images in the time series.

16. The person detection device according to claim 15, wherein, If the difference value remains below a predetermined value, the control unit averages the detection results of the person during that period to detect the person.

17. The person detection device according to any one of claims 14 to 16, wherein, The difference value is based on the coordinates of the captured image.

18. The person detection device according to any one of claims 14 to 16, wherein, The difference value is a value based on the style of the captured image.

19. The person detection device according to any one of claims 14 to 16, wherein, After the first detection process is performed and a person is detected, the second detection process is performed.

20. A vehicle control system, Equipped with the person detection device and vehicle control device as described in any one of claims 14 to 16 If a person detected by the person detection device is present in a predetermined space set in the vehicle, the vehicle control device prohibits the execution of the vehicle's predetermined operation.

Images