Control device and program

Abstract

A control device (30), comprising:a detection unit (215) which detects a reference position indicating a position to be used as a reference based on an image recorded by an imaging device (10) recording an interior of a vehicle from predetermined portions of a passenger;anda control unit (216) provided in the vehicle and performing such control that, during a closing operation of an openable and closable opening and closing member, the closing operation of the opening and closing member is interrupted when a coordinate value of the reference position detected by the detection unit (215) exceeds a threshold value, characterized in that the threshold value comprises a first threshold value (yth), and the control unit (216) performs such control that the closing operation of the opening and closing member is interrupted when the coordinate value of the reference position in a vertical direction exceeds the first threshold value (yth) for a certain period of time;

Description

Technical area

[0001] The present invention relates to a control device and a program. Technical background

[0002] JP 2007-186915 A discloses an electric window lifting device that can detect the jamming of a foreign object in a window pane before it actually becomes jammed. Said window lifting device detects whether or not a foreign object, such as a human body, is jammed in the window by using images of the window captured by an image capture device, such as a camera, when the window is opened or closed.

[0003] JP 2017-182748 A discloses a method for simplifying the configuration of a terminal in a communications system in which a user terminal, a settings terminal, and a server device can be connected via a communications network. The terminal can utilize radio tags that enable mobile terminals such as smartphones to receive a variety of services by transmitting the signals received from the radio tags to a central system. Possible services include, for example, search assistance services for locating hikers, lost children, pets, and other targets, as well as search services for lost personal belongings or transported goods.

[0004] A method for detecting foreign object entrapment by an opening and closing component of a vehicle, such as a power window or the like, is known from the prior art. For example, JP 2007-186915 A (Document 1) discloses a method in which whether a foreign object is entrapped by a window is detected based on a feature value obtained from the brightness reflected in image data recorded by an imaging means.

[0005] In the above-described prior art, a contour of an object (such as a window) is extracted based on the brightness of the image data, and the occurrence of foreign object jamming is detected using a change amount and a range of the extracted contour as a feature value. However, when a large error occurs in the extracted contour, a problem arises that jamming is difficult to completely prevent because the occurrence of foreign object jamming in the opening and closing component of the vehicle cannot be detected with high accuracy. Summary of the invention

[0006] A control device according to one aspect of the invention comprises: a detection unit that detects a reference position indicating a position to be used as a reference from the predetermined portions of the passenger based on an image acquired by an imaging device recording an interior of the vehicle; and a control unit that is provided in the vehicle and performs control such that, during the closing operation of an openable and closable opening and closing member, the closing operation of the opening and closing member is interrupted when a coordinate value of a reference position detected by the detection unit exceeds a threshold value.According to this configuration, the occurrence of the pinching in the predetermined portion in the opening and closing member can be detected with high accuracy, and the pinching of the predetermined portion by the opening and closing member can be completely prevented.

[0007] In the control device according to this aspect, the threshold value includes a first threshold value, and the control unit performs control such that the closing operation of the opening and closing member is interrupted when the coordinate value of the reference position in a vertical direction exceeds the first threshold value for a certain period of time. According to this configuration, the occurrence of the foreign object being caught in the opening and closing member can be detected with high accuracy, and the occurrence of the predetermined portion being caught in the opening and closing member can be completely prevented.

[0008] In the control device according to this aspect, for example, the control unit may perform control such that, when the closing operation of the opening and closing member has been interrupted, the closing operation of the opening and closing member is resumed when a state in which the coordinate value of the reference position in the vertical direction is equal to or less than the first threshold value continues for a certain period of time. According to this configuration, the closing operation of the opening and closing member is resumed when the possibility of the predetermined portion in the opening and closing member being caught is eliminated, so that it is possible to complete the closing operation of the opening and closing member without requiring a new action input (command input) from the passenger.

[0009] In the control device according to this aspect, for example, the predetermined portion may be a passenger's hand, and the opening and closing member may be a sunroof. According to this configuration, it is possible to determine with high accuracy whether the passenger's hand is being trapped in the sunroof, and further, the passenger's hand being trapped in the sunroof can be completely prevented.

[0010] In the control device according to this aspect, the threshold value may include, for example, a second threshold value, and the control unit may perform control such that the closing operation of the opening and closing member is interrupted when a coordinate value of the reference position in a horizontal direction exceeds the second threshold value for a certain period of time. According to this configuration, the occurrence of foreign matter jamming in the opening and closing member can be detected with high accuracy, and jamming of the predetermined portion in the opening and closing member can be completely prevented.

[0011] In the control device according to this aspect, for example, the control unit may perform control such that the closing operation of the opening and closing member is resumed when, after the closing operation of the opening and closing member is interrupted, a state in which the coordinate value of the reference position in the horizontal direction is equal to or less than the second threshold value continues for a certain period of time. With this configuration, the closing operation of the opening and closing member is resumed when the possibility of the predetermined portion in the opening and closing member being caught is eliminated, so that it is possible to complete the closing operation of the opening and closing member without requiring a new action input (command input) from the passenger.

[0012] In the control device according to this aspect, for example, the predetermined portion may be a passenger's hand, and the opening and closing member may be a sliding door or a power window. According to this configuration, it is possible to determine with high accuracy whether a passenger's hand is being trapped in the sliding door or the power window, and the passenger's hand is being trapped in the sliding door or the power window can be completely prevented.

[0013] A program according to another aspect of this invention causes a computer to perform a detecting step of detecting, based on an image captured by an image recording device recording an interior of a vehicle, a reference position as a reference position from the predetermined portions of the passenger, and a controlling step of controlling such that, during a closing operation of an openable and closable opening and closing member, the closing operation of the opening and closing member is interrupted when a coordinate value of the reference position detected in the detecting step exceeds a threshold value, the threshold value including a first threshold value, and the closing operation of the opening and closing member is interrupted when the coordinate value of the reference position in a vertical direction exceeds the first threshold value for a predetermined period of time. Short description of the characters

[0014] The above and additional features and characteristics of the invention will become apparent from the following detailed description with reference to the accompanying figures, in which: Fig. 1 is a diagram showing a schematic configuration of an information processing system according to an embodiment; Fig. 2 is a diagram showing an example of a hardware configuration of an image processing apparatus; Fig. 3 is a view showing an example of a function included in an image processing apparatus; Fig. 4 is a view showing an example of skeleton information; Fig. 5 is a diagram showing an example of the information of a map; Fig. 6 is a view showing an example of an action corresponding to an “OPEN” command; Fig. 7 is a view showing an example of an action corresponding to a “CLOSE” command; Fig. 8 is a view showing an example of an action corresponding to a “PAUSE” command; Fig. Fig. 9 is a view showing the pinching of a passenger's hand by a sunroof; Fig. 10 is a view explaining control by a control unit according to a first embodiment; Fig. 11 is a flowchart showing an operation example of an image processing apparatus according to a first embodiment; Fig. 12 is a diagram explaining control by the control unit according to a second embodiment; and Fig. 13 is a flowchart showing an operation example of a processing apparatus according to a second embodiment. Detailed description

[0015] In the following, a control device and a program according to the invention are described in detail with reference to the associated figures. First embodiment

[0016] Fig. 1 is a diagram showing a schematic configuration of an information processing system 100 mounted in a vehicle, such as an automobile, having a power source such as an internal combustion engine or a motor. In the present embodiment, it is assumed that a configuration is provided in which a command is automatically executed when a passenger of the vehicle performs an action (a gesture) corresponding to any command. For example, when the passenger performs an action corresponding to a command to perform an opening / closing operation of an openable and closable sunroof (this is an example of an "openable and closable opening and closing component provided in the vehicle"), the command is automatically executed.However, the invention is not limited thereto, and the opening and closing control of the sunroof can also be performed without requiring gesture recognition. For example, an operating unit (such as a button) can be provided for inputting an opening / closing command of the sunroof, and the opening and closing operation of the sunroof can be performed automatically in accordance with the command input via the operating unit.

[0017] As in Fig. 1, the information processing system 100 includes an imaging device 10, an image processing device 20, and a vehicle control device 30.

[0018] The imaging device 10 is a device that records an interior of the vehicle. For example, the imaging device 10 includes a camera. In this example, the imaging device 10 continuously records images at a predetermined frame rate. An image recorded by the imaging device 10 (hereinafter, this may also be referred to as a "recorded image") is input to the image processing device 20.

[0019] Based on the images captured and input by the imaging device 10, the image processing device 20 determines whether an action corresponding to any command is being performed, and if the determination result is affirmative, outputs command-indicating information (command information) that allows output to the vehicle control device 30. In addition, the image processing device 20 detects whether pinching of a predetermined portion of the passenger occurs during the closing operation of a sunroof and performs control such that the closing operation of the sunroof is interrupted when it is determined that pinching of the predetermined portion of the passenger occurs. In this example, the image processing device 20 is an example of the "control device." A specific configuration of the image processing device 20 will be described below.

[0020] The vehicle control device 30 controls the respective components of the vehicle in accordance with the command indicated by the command information input from the image processing device 20.

[0021] A specific configuration of the image processing apparatus 20 of the present embodiment will be described below. Fig. 2 is a diagram showing an example of a hardware configuration of the image processing device 20. As shown in Fig. As shown in Figure 2, the image processing device 20 includes a CPU 201, a ROM 202, a RAM 203, and an external I / F 204. In this example, the image processing device 20 has the same hardware configuration as a normal computer. It should be noted that the hardware elements included in the image processing device 20 are not limited to those shown in Fig. 2 are limited and additional hardware elements may also be provided.

[0022] By executing the program, the CPU 201 comprehensively controls the operation of the image processing device 20 and implements various functions included in the image processing device 20. Various functions included in the image processing device 20 are described below.

[0023] The ROM 202 is a non-volatile memory and stores various data comprising a program for starting the image processing device 20. The RAM 203 is a volatile memory comprising a working area of the CPU 201.

[0024] The external I / F 204 is an interface for connecting to an external device. For example, an interface for connecting to the imaging device 10 and an interface for connecting to the vehicle control device 30 may be provided as the external I / F 204.

[0025] Fig. 3 is a diagram showing an example of functions provided in the image processing device 20. In the Fig. In the example shown in Fig. 3, only the functions related to the present invention are shown, but the functions provided in the image processing apparatus 20 are not limited thereto.

[0026] As in Fig. 3, the image processing device 20 includes a capture unit 211, a recognition unit 212, a determination unit 213, a command output unit 214, a detection unit 215, and a control unit 216. In this example, the CPU 201 executes a program stored in a storage device such as the ROM 202, so that the respective functions of the capture unit 211, the recognition unit 212, the determination unit 213, the command output unit 214, the detection unit 215, and the control unit 216 are implemented. However, the invention is not limited thereto, and for example, at least a part of the recording unit 211, the recognition unit 212, the determination unit 213, the command output unit 214, the detection unit 215 and the control unit 216 may each be configured as a designated hardware circuit.

[0027] The acquisition unit 211 acquires an image recorded by the imaging unit 10. Each time the imaging device 10 records (an image), the acquisition unit 211 acquires the recorded image obtained by the image recording.

[0028] The recognition unit 212 recognizes an action of the passenger based on the recorded image (the image recorded by the imaging device 10) acquired by the capture unit 211. In this example, the passenger's action is an action mode using hands, but is not limited to this.

[0029] Various known methods can be used as a method for recognizing the passenger's action based on the recorded image. For example, one embodiment may use the method disclosed in Japanese Patent Application JP 2017-182748A. In the present embodiment, the recognition unit 212 extracts joints (feature points) from each portion of the passenger's body (upper body) reflected in the recorded image and generates skeletal information (skeletal data) therefrom. The recognition unit 212 then recognizes the passenger's action based on the acquired skeletal information.

[0030] Fig. Fig. 4 is a view showing an example of skeleton information according to the present embodiment. Each feature point is represented by a combination (binary coordinate information) of a coordinate value in an x-direction (horizontal direction) and a coordinate value in a y-direction (vertical direction). In the example of Fig. 4 are a feature point P1 (x1, y1) corresponding to a head, a feature point P2 (x2, y2) corresponding to a neck, a feature point P3 (x3, y3) corresponding to a right shoulder, a feature point P4 (x4, y4) corresponding to a right elbow, a feature point P5 (x5, y5) corresponding to a right wrist, a feature point P6 (x6, y6) corresponding to a center point of a right hand, a feature point P7 (x7, y7) corresponding to a thumb of the right hand, a feature point P8 (x8, y8) corresponding to a middle finger of the right hand, a feature point P9 (x9, y9) corresponding to a right shoulder, a feature point P10 (x10, y10) corresponding to a left elbow, a feature point P11 (x11, y11) corresponding to a left wrist, a feature point P12 corresponding to a center point of the left hand (x12, y12), a feature point P13 (x13, y13) corresponding to a thumb of the left hand,a feature point P14 (x14, y14) corresponding to a middle finger of the left hand, a feature point P15 (x15, y15) corresponding to a right hip, and a feature point P16 (x16, y16) corresponding to a left hip are listed as feature points of the skeleton information, but the invention is not limited to the above.

[0031] Returning to Fig. 3. The explanation continues. The determination unit 213 determines whether an action corresponding to a command is executed based on the passenger's action detected by the detection unit 212. In the present embodiment, the determination unit 213 determines whether an action corresponding to any command is executed based on map information linking the commands to the actions. More specifically, the determination unit 213 identifies the commands in association with the actions and determines that an action corresponding to any command is executed when an action corresponding to the passenger's action detected by the detection unit 212 is included among the actions detected in the map information.

[0032] Fig. 5 is a diagram showing an example of the map information. The map information may be stored in a memory (e.g., ROM 202) in the image processing device 20, but may also be stored in a memory other than that of the image processing device 20. In the example of Fig. 5 describes a command for controlling the sunroof as an example, but the type of command is not limited to this.

[0033] In Fig. 5, the command designated "OPEN" is a command that instructs opening the sunroof, the command designated "CLOSE" is a command that instructs closing the sunroof, and the command designated "PAUSE" is a command that instructs pausing or stopping (interrupting) the opening and closing operation of the sunroof.

[0034] As in Fig. As shown in Figure 6, an action in which the hand is moved in the direction of opening the sunroof is linked to the "OPEN" command. In addition, as shown in Fig. 7, an action in which the hand is moved in the closing direction of the sunroof is linked to the command “CLOSE”. As shown in Fig. As shown in Figure 8, the "PAUSE" command is associated with an action that involves raising a fist. The actions corresponding to each command can be optional or arbitrary, and are not limited to this.

[0035] Thus, the determination unit 213 determines whether an action corresponding to a command is executed and outputs the determination result to the command output unit 214.

[0036] The Fig. The command output unit 214 shown in Fig. 3 outputs to the vehicle control device 30 command information indicating the command determined to correspond to the executed action.

[0037] In this example, when the command output unit 214 outputs the command information indicating the "CLOSE" command to the vehicle control device 30, the vehicle control device 30 closes the sunroof. At this time, as shown in Fig. 9, if a passenger on a rear seat of the vehicle reaches outward while the closing operation is continuing, the passenger's hand may become trapped by the sunroof.

[0038] The image processing device 20 according to the present embodiment detects a reference position indicating a position serving as a reference (in this example, the "passenger's hand") from the predetermined portions of the passenger based on the recorded image acquired by the vehicle interior imaging unit 10, and executes control such that the closing operation of the sunroof is interrupted when the coordinate value of the reference position detected based on the recorded image exceeds a threshold value during the closing operation of the sunroof. This corresponds to the functions of the detection unit 215 and the control unit 216 described in Fig. 3. The following describes the functions of the detection unit 215 and the control unit 216, which are shown in Fig. 3 are described.

[0039] The detection unit 215 detects, from the predetermined portions of the passenger, a reference position indicating a position serving as a reference based on an image recorded by the imaging device 10 (the recorded image acquired by the acquisition unit 211). In this example, the predetermined portion is the passenger's hand, but the invention is not limited to this, and therefore, it may also be, for example, the passenger's head. Additionally, in the above example, the reference position is a center point of the hand, but the invention is not limited to this, and therefore, for example, a middle finger or a similar portion of the passenger's hand may also be used.In addition, since in this example, all passengers in the vehicle (the driver, the passenger in the front passenger seat, and the passengers in the rear seat) are considered as passengers, it is assumed that the imaging device 10 is arranged (viewing angle and adjusted position) such that all passengers in the vehicle are recorded.

[0040] In this example, the detection unit 215 detects the reference position according to an instruction from the control unit 216, which will be described below, and notifies the control unit 216 of the detection result. While the method for detecting the reference position is optional, for example, the detection unit 215 generates the above-described skeleton information for each passenger included in the recorded image from the recorded image acquired by the recording unit 211, and detects the position of the center point of the passenger's hand as a reference position for each passenger based on the generated skeleton information. In this example, while the recognition unit 212 and the detection unit 215 are provided separately, the recognition unit 212 and the detection unit 215 may also be configured as a common module (e.g., a common module).in a mode in which the detection unit 212 also performs the function of the detection unit 215).

[0041] The control unit 216 performs such a control that the closing operation of the sunroof is interrupted when the coordinate value of the reference position detected by the detection unit 215 exceeds the threshold value during the closing operation of the sunroof. More specifically, as shown in Fig. 10, the control unit 216 performs such control that the closing operation of the sunroof is interrupted when the coordinate value y0 of the reference position (in this example, the center of the passenger's hand) in the vertical direction exceeds the first threshold value y for a predetermined period of time. th (as one aspect of the threshold). In this example, the first threshold is y tha coordinate value in the vertical direction on the basis of which the occurrence of the passenger's hand being pinched by the sunroof can be determined, and it can be determined that the passenger's hand being pinched by the sunroof occurs when the coordinate value y0 of the center of the passenger's hand in the vertical direction is greater than the first threshold value y th is.

[0042] As an example of the above-described control for interrupting the closing operation of the sunroof, the control unit 216 may, for example, output to the vehicle control device 30 command information indicating a command instructing interruption of the closing operation of the sunroof. The vehicle control device 30 may interrupt the closing operation of the sunroof by executing the command indicated by the command information received from the control device 216.

[0043] Additionally, in this example, when the command information indicating the "CLOSE" command is output to the vehicle control device 30, the command output unit 214 notifies the control unit 216. Upon receiving this notification, the control unit 216 recognizes that the sunroof is in the closing operation and requests that the detection unit 215 detect the reference position. Upon receiving this request, the detection unit 215 detects the reference position based on the recorded image acquired by the acquisition unit 211 and outputs the detection result to the control unit 216. Then, based on the recorded image, the control unit 216 determines from the relationship between the coordinate value y0 of the detected reference position in the vertical direction and the predetermined first threshold value y thWhether to execute control to interrupt the closing operation of the sunroof. In this example, the command output unit 214 and the control unit 216 are provided separately, but the invention is not limited thereto, and the command output unit 214 and the control unit 216 may also be configured as a common module (e.g., in a mode in which the command output unit 214 also performs the function of the control unit 216).

[0044] According to the present embodiment, only by extracting the reference position from the recorded image, it can be determined whether a passenger's hand is caught in the sunroof, based on the magnitude relationship between the coordinate value y0 of the extracted reference position in the vertical direction and the determined first threshold value y th, so that it is unnecessary to perform a process for extracting the contour of the entire object, such as a window frame or the like, included in the recorded image, as required in the prior art. In addition, compared with the case of extracting the contour of the entire object from the recorded image as provided in the prior art, the error occurring in the image recognition processing method is smaller in the case of extracting a reference position point from the recorded image according to the present embodiment, and therefore the accuracy of determining whether a passenger's hand is caught in the opening and closing members provided in the vertical direction of the vehicle, such as the sunroof and the like, can be improved.

[0045] In addition, the control unit 216 performs such a control that the closing operation of the sunroof is resumed when a state in which the coordinate value y0 of the reference position in the vertical direction is equal to or smaller than the first threshold value y this, continues for a certain period of time after the closing operation of the sunroof has been interrupted. For example, the control unit 216 may transmit to the vehicle control device 30 command information indicating a command instructing resumption of the closing operation of the sunroof. The vehicle control device 30 may resume the closing operation of the sunroof by executing the command indicated by the command information obtained from the control unit 216. In this example, when the possibility of a passenger's hand being caught by the sunroof is eliminated, the closing operation of the sunroof is resumed, whereby it is possible to complete the closing operation of the sunroof without requiring a new action input (command input) from the passenger.

[0046] Fig. 11 is a flowchart showing an example of the action of the image processing device 20 during the closing operation of the sunroof according to the present embodiment. Since the specific content of each step is as described above, a detailed description thereof will not be repeated where appropriate.

[0047] As in Fig. As shown in Figure 11, the detection unit 215 first detects the reference position (in this example, the center of the passenger's hand) based on the recorded image acquired by the recording unit 211 (step S1). Next, the control unit 216 determines whether the coordinate value y0 of the reference position detected in step S1 in the vertical direction is greater than the first threshold value y th is (step S2).

[0048] If the result of step S2 is affirmative (step S2: Yes), the control unit 216 determines whether a state in which the coordinate value y0 of the reference position in the vertical direction is greater than the first threshold value y th is continued for a certain period of time (step S3). If the result of step S3 is negative (step S3: No), the processing steps from step S1 are repeated. On the other hand, if the result of step S3 is positive (step S3: Yes), the control unit 216 performs control such that the closing operation of the sunroof is interrupted (step S4).

[0049] As described above, in the present embodiment, based on the image recorded by the imaging device 10 recording an interior of the vehicle during the closing operation of the sunroof, control is performed such that the closing operation of the sunroof is interrupted when the coordinate value y0 of the reference position of the predetermined portion of the passenger (in this example, the center of the passenger's hand) detected in the vertical direction exceeds the first threshold value y for a certain period of time. th As a result, it can be determined with high accuracy whether a passenger's hand is being trapped in the sunroof, and a passenger's hand being trapped in the sunroof can be completely prevented. Second embodiment

[0050] Next, a second embodiment will be described. Description of the portions common to the first embodiment described above will not be repeated where appropriate. In the first embodiment described above, it is assumed that a component provided in the vehicle in the vertical direction (e.g., a sunroof) is the opening and closing component. In the present embodiment, the difference lies in that a component provided in the vehicle in the horizontal direction, such as a sliding door or a power window, is assumed as the opening and closing component. In the following description, a sliding door, for example, will be described as an opening and closing component, but the invention is not limited thereto.In this example, since the configuration except for the functions included in the control unit 216 is the same as that of the first embodiment described above, a detailed description thereof will not be repeated.

[0051] As in the first embodiment described above, the basic function of the control unit 216 is to perform control such that, during the closing operation of an openable and closable opening and closing member, the closing operation of the opening and closing member is interrupted when a coordinate value of a reference position detected by the detection unit 215 exceeds a threshold value. In this example, the opening and closing member is the sliding door, the predetermined portion is the passenger's hand, and the reference position is the center of the passenger's hand, but the invention is not limited to this.

[0052] As in Fig. 12, the control unit 216 according to the present embodiment performs such control that the closing operation of the sliding door is interrupted when, during the closing operation of the sliding door, the coordinate value x0 of the center of the passenger's hand in the horizontal direction exceeds the second threshold value x th (one aspect of the threshold) for a certain period of time. In this example, the second threshold is x th a coordinate value in the horizontal direction on the basis of which the occurrence of the passenger's hand being caught by the sliding door can be determined, and it can be determined that the passenger's hand being caught by the sliding door occurs when the coordinate value x0 of the center of the passenger's hand in the horizontal direction is greater than the second threshold value x th is.

[0053] As an example of the control for interrupting the closing operation of the sliding door, the control unit 216 may output to the vehicle control device 30 command information indicating a command instructing the interruption of the closing operation of the sliding door. The vehicle control device 30 may interrupt the closing operation of the sliding door by executing the command indicated by the command information received from the control unit 216.

[0054] As in the first embodiment described above, the command output unit 214 notifies the control unit 216 when command information indicating a command instructing the closing operation of the sliding door is output to the vehicle control device 30. The action corresponding to the command instructing the closing operation of the sliding door or the command instructing the opening operation of the sliding door (an action defined in advance for each command) can be set optionally. Upon receiving the notification, the control unit 216 recognizes that the sliding door is in the closing operation and requests the detection unit 215 to detect the reference position.Upon receiving this request, the detection unit 215 detects the reference position based on the recorded image acquired by the recording unit 211 and transmits the detection result to the control unit 216. Then, based on the acquired image, the control unit 216 determines from the relationship between the coordinate value x0 of the detected reference position in the horizontal direction and the predetermined second threshold value x. th whether a control should be carried out so that the closing operation of the sliding door is interrupted.

[0055] According to the present embodiment, only by extracting the reference position from the recorded image and from the size relationship between the coordinate value x0 of the extracted reference position in the horizontal direction and the predetermined second threshold value x thIt can be determined whether a passenger's hand is being caught in the sliding door, so that it is unnecessary to perform processing of extracting the contour of the entire object, such as a window frame or the like, included in the recorded image, as required by the prior art. In addition, compared with the case of extracting the contour of the entire object from the recorded image according to the prior art, the error incurred in the image recognition processing method in the case of extracting only a reference position point from the recorded image according to the present embodiment is smaller, and therefore the accuracy of determining whether a passenger's hand is being caught in the opening and closing member provided in the horizontal direction of the vehicle, such as a sliding door or the like, can be improved.

[0056] In addition, the control unit 216 performs such control that the closing operation of the sliding door is resumed when, after the closing operation of the sliding door has been interrupted, a state in which the coordinate value x0 of the reference position in the horizontal direction is equal to or smaller than the second threshold value x this, stops for a certain period of time. For example, the control unit 216 may transmit to the vehicle control device 30 command information indicating a command instructing resumption of the closing operation of the sliding door. The vehicle control device 30 may resume the closing operation of the sliding door by executing the command indicated by the command information received from the control unit 216. In this example, when the possibility of a passenger's hand being caught by the sliding door is eliminated, the closing operation is resumed, whereby the closing operation of the sliding door can be completed without requiring a new action input (command input) from the driver.

[0057] Fig. 13 is a flowchart showing an example of operation of the image processing device 20 during the closing operation of the sliding door according to the present embodiment. Since the specific content of each step is as described above, a detailed explanation thereof will not be repeated where appropriate.

[0058] As in Fig. As shown in Figure 13, the detection unit 215 first detects the reference position (the center of the passenger's hand in this example) based on the recorded image acquired by the recording unit 211 (step S10). Next, the control unit 216 determines whether the horizontal coordinate value x0 of the reference position determined in step S10 is greater than the second threshold value x th is (step S11).

[0059] If the result of step S11 is affirmative (step S11: Yes), the control unit 216 determines whether a state in which the coordinate value x0 of the reference position in the horizontal direction is greater than the second threshold value x th has already been going on for a certain period of time (step S12). If the result of step S12 is negative (step S12: No), the processing steps from step S10 onwards are repeated. However, if the result of step S12 is positive (step S12: Yes), the control unit 216 performs control such that the closing operation of the sliding door is interrupted (step S13).

[0060] As described above, in the present embodiment, based on the image recorded by the imaging device 10 recording an interior of the vehicle during the closing operation of the sliding door, control is performed such that the closing operation of the sliding door is interrupted when the coordinate value x0 of the reference position of the predetermined portion of the passenger (in this example, the center of the passenger's hand) detected in the horizontal direction exceeds the second threshold value x for a certain period of time. th (as one aspect of the threshold). As a result, it can be determined with high accuracy whether a passenger's hand is being trapped in the sliding door, and the passenger's hand being trapped in the sliding door can be completely prevented.

[0061] Although embodiments of the invention have been described above, the invention is not limited to the above-described embodiments, but can be implemented by modifying the elements without departing from the spirit of the invention. In addition, various inventions can be formed by combining a plurality of elements disclosed in the above embodiments. For example, some elements may be omitted from the entirety of the elements shown in the embodiments. In addition, each embodiment and each modification example described above can be optionally combined.

[0062] The type of openable and closable opening and closing component provided in the vehicle is not limited to the types described above, but may also be, for example, a rear door / tailgate or the like.

[0063] Additionally, the control unit 216 described above may also be provided on the side of the vehicle control device 30, for example. In this case, the combination of the image processing device 20 and the vehicle control device 30 corresponds to the "control device." In short, the control device may be provided in a form including at least the detection unit 215 and the control device 216 described above, and may be formed by a single device or a plurality of devices (the detection unit 215 and the control device 216 are divided into a plurality of devices).

[0064] The principles, preferred embodiments, and mode of operation of the present invention have been described in the foregoing specification. However, the invention to be protected should not be construed as limited to the specific embodiments set forth. Rather, the above-described embodiments are for the purpose of illustration only, not limitation. Modifications and changes may be made, and equivalents may be employed, without departing from the spirit of the invention. Accordingly, all such modifications, changes, and equivalents that fall within the spirit and scope of the present invention as defined in the claims are expressly intended to be embraced herein.

[0065] A control device (20) comprises: a detection unit (215) that detects a reference position indicating a position serving as a reference from the predetermined portions of the passenger on the basis of an image recorded by an imaging device (10) recording an interior of a vehicle; and a control unit (216) that is provided in the vehicle and performs such control that, during a closing operation of an openable and closable opening and closing member, the closing operation of the opening and closing member is interrupted when a coordinate value of the reference position detected by the detection unit exceeds a threshold value.

Claims

[1] Control device (30), comprising: a detection unit (215) which, based on an image recorded by an imaging device (10) recording an interior of a vehicle, detects a reference position from predetermined portions of a passenger, which indicates a position serving as a reference; and a control unit (216) which is provided in the vehicle and carries out such a control that, during a closing operation of an openable and closable opening and closing component, the closing operation of the opening and closing component is interrupted when a coordinate value of the reference position which has been detected by the detection unit (215) exceeds a threshold value, characterized in that the threshold has a first threshold value (y th ), and the control unit (216) carries out such a control that the closing operation of the opening and closing member is interrupted when the coordinate value of the reference position in a vertical direction exceeds the first threshold value (y th ) goes beyond [2] The control device (30) according to claim 1, wherein the control unit (216) performs such control that the closing operation of the opening and closing member is resumed when, after the closing operation of the opening and closing member has been interrupted, a state in which the coordinate value of the reference position in the vertical direction is equal to or smaller than the first threshold value (y th ) lasts for a certain period of time. [3] The control device (30) according to claim 1 or 2, wherein the predetermined portion is a passenger's hand, and the opening and closing member is a sunroof. [4] Control device (30) according to claim 1, wherein the threshold value comprises a second threshold value (x th ), and the control unit (216) performs such control that the closing operation of the opening and closing member is interrupted when a coordinate value of the reference position in a horizontal direction exceeds the second threshold value (x th ) goes beyond. [5] The control device (30) according to claim 4, wherein the control unit (216) carries out such control that the closing operation of the opening and closing member is resumed when, after the closing operation of the opening and closing member has been interrupted, a state in which the coordinate value of the reference position in the horizontal direction is equal to or smaller than the second threshold value (x th ) lasts for a certain period of time. [6] The control device (30) according to claim 5, wherein the predetermined portion is a passenger's hand and the opening and closing member is a sliding door or an electric window. [7] A program product that causes a computer to perform the following steps: a detecting step (S1; S10) for detecting a reference position from the predetermined portions of a passenger, which indicates a position serving as a reference, based on an image recorded by an imaging device (10) recording an interior of a vehicle; and a control step (S4; S13) of controlling such that, during a closing operation of an openable and closable opening and closing member, the closing operation of the opening and closing member is interrupted when a coordinate value of the reference position detected in the detection step (S1; S10) exceeds a threshold value, the threshold value being a first threshold value (y th ) (S2; S11), and the closing operation of the opening and closing member is interrupted when the coordinate value of the reference position in a vertical direction exceeds the first threshold value (y th ) (S3; S12).

Images