Control system, control device, and computer program

Abstract

To allow a controlled device to perform a desired operation for a user without a need for the user to actively operate.SOLUTION: An imaging device 11 outputs image data ID corresponding to an image in which a subject appears. A control device 12 performs, on the image data, a process of applying to the subject a skeletal model in which a plurality of feature points each corresponding to a predetermined body part are connected, and identifies, within the image, a feature point corresponding to a specific body part. In addition, the control device 12 controls operation of an air conditioning device 13 for the user based on the position of the feature point.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a control system for controlling the operation of a controlled device based on image data output from an imaging device. The present disclosure also relates to a control device that may be included in the system, and a computer program executable by a processor installed in the device. [Background technology]

[0002] Patent Document 1 discloses an air conditioning control system that changes the direction of airflow sent out from an air conditioner in response to the movement of a user's hand captured in an image captured by an imaging device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-206308 A Summary of the Invention [Problem to be solved by the invention]

[0004] There is a demand for allowing a user to cause a controlled device to perform a desired action without requiring active action by the user. [Means for solving the problem]

[0005] One example aspect provided by the present disclosure is a control system, comprising: an imaging device that outputs image data corresponding to an image in which a subject is captured; A controlled device; a control device that applies a skeletal model to the subject by processing the image data, the skeletal model including a plurality of connected feature points, each of which corresponds to a specific body part, to the subject, thereby identifying feature points in the image that correspond to specific body parts, and controls the operation of the controlled device with respect to the user based on the positions of the feature points; Contains:

[0006] One example aspect provided by the present disclosure is a control device, comprising: an interface that receives image data corresponding to an image in which a subject is captured from an imaging device; a processor that applies a skeletal model to the subject, the skeletal model including a plurality of connected feature points, each of which corresponds to a specific body part, to the image data, thereby identifying feature points in the image that correspond to specific body parts, and controls an operation of a controlled device with respect to a user based on the positions of the feature points; It is equipped with:

[0007] One example of an aspect provided by the present disclosure is a computer program executable by a processor installed in a control device, the computer program comprising: By being executed, the control device receiving image data corresponding to an image in which a subject is captured from an imaging device; applying a skeletal model to the subject, the skeletal model including a plurality of connected feature points, each of which corresponds to a specific body part, to the image data to identify feature points in the image that correspond to specific body parts; The action of a controlled device relative to the user is controlled based on the location of the feature point.

[0008] According to the configurations of the above-mentioned embodiments, when a user is captured by an imaging device, a skeleton model is applied to the image in which the user is captured, and feature points corresponding to the body parts of the user required to make the controlled device perform a desired action are identified. Therefore, the user can make the controlled device perform a desired action without needing to actively move the imaging device. [Brief description of the drawings]

[0009] [Figure 1] 2 illustrates an example of a functional configuration of a control system according to an embodiment. [Diagram 2] 2 illustrates a vehicle in which the control system of FIG. 1 is installed. [Diagram 3] 2 shows an example of a flow of processing executed by the control system of FIG. 1. [Figure 4] 2 illustrates an example of an image that can be captured by the imaging device of FIG. 1. [Diagram 5] The image in Figure 4 shows an example of the skeleton model applied. [Figure 6] 2 shows an example of an operation executed by the air conditioner of FIG. [Figure 7] 4 illustrates another example of the operation performed by the air conditioner of FIG. [Figure 8] 4 illustrates another example of the operation performed by the air conditioner of FIG. [Figure 9] 4 illustrates another example of the operation performed by the air conditioner of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, examples of embodiments will be described in detail with reference to the accompanying drawings. In each drawing used in the following description, the scale is appropriately changed so that each element can be recognized.

[0011] 1 illustrates an example of a functional configuration of a control system 10 according to an embodiment. The control system 10 includes an imaging device 11, a control device 12, and an air conditioner 13. The control system 10 is configured to control the operation of the air conditioner 13 based on image data output from the imaging device 11. The air conditioner 13 is an example of a controlled device.

[0012] Specifically, the imaging device 11 is disposed at an appropriate location in a vehicle 20 illustrated in Fig. 2. A driver 30 seated in a seat 22 disposed in a vehicle interior 21 is imaged by the imaging device 11. The vehicle interior 21 in which the driver 30 sits is an example of a subject. The driver 30 is an example of a user.

[0013] In the accompanying drawings, arrow F represents the forward direction as seen by the driver 30. Arrow B represents the rearward direction as seen by the driver 30. Arrow L represents the leftward direction as seen by the driver 30. Arrow R represents the rightward direction as seen by the driver 30. Arrow U represents the upward direction as seen by the driver 30. Arrow D represents the downward direction as seen by the driver 30.

[0014] 1, the imaging device 11 is configured to output image data ID corresponding to an image that a subject is captured in. The image data ID may be in the form of analog data or digital data depending on the specifications of the imaging device 11.

[0015] The control device 12 includes an input interface 121. The input interface 121 is configured as a hardware interface that receives image data ID from the imaging device 11. When the image data ID is in the form of analog data, the input interface 121 includes an appropriate conversion circuit including an A / D converter.

[0016] The control device 12 includes a processor 122 and an output interface 123. The processor 122 is configured to execute image processing, which will be described later, on the received image data ID, and to output a control signal CS, which controls the operation of the air conditioner 13, from the output interface 123 based on the result of the processing.

[0017] The output interface 123 is configured as a hardware interface that outputs a control signal CS. The control signal CS may be an analog signal or a digital signal depending on the specifications of the air conditioner 13. When the control signal CS is an analog signal, the output interface 123 includes an appropriate conversion circuit including a D / A converter.

[0018] The image processing and the control processing of the air conditioner 13 executed by the control device 12 will be described in detail with reference to Fig. 3 to Fig. 7. Fig. 3 illustrates the flow of the processing.

[0019] The processor 122 receives image data ID from the imaging device 11 through the input interface 121 (STEP 1). Fig. 4 illustrates an image IM in which the vehicle interior 21 including the driver 30 is captured and is captured by the imaging device 11. The image data ID corresponds to the image IM.

[0020] Next, the processor 122 performs a process of applying the skeletal model to the subject (STEP 2). The term "process of applying the skeletal model to the subject" used in this specification means detecting a plurality of feature points defined in the skeletal model of the driver 30 captured in the image IM acquired by the imaging device 11, and connecting the plurality of feature points with a plurality of skeletal lines defined in the skeletal model.

[0021] 5 shows an example in which a skeletal model M is applied to a driver 30 captured in an image IM acquired by the imaging device 11. In this example, the skeletal model M includes a head feature point H, a neck feature point N, a waist feature point W, a left shoulder feature point LS, a right shoulder feature point RS, a left elbow feature point LE, a right elbow feature point RE, a left wrist feature point LW, and a right wrist feature point RW.

[0022] The head feature point H corresponds to the center of the head of the model human body. The neck feature point N corresponds to the neck of the model human body. The waist feature point W corresponds to the waist of the model human body. The head feature point H, the neck feature point N, and the waist feature point W are connected by a skeleton line.

[0023] The left shoulder feature point LS corresponds to the left shoulder of the model human body. The left elbow feature point LE corresponds to the left elbow of the model human body. The left wrist feature point LW corresponds to the left wrist of the model human body. The neck feature point N is connected to the left shoulder feature point LS by a skeleton line. The left elbow feature point LE is connected to each of the left shoulder feature point LS and left wrist feature point LW by skeleton lines.

[0024] The right shoulder feature point RS corresponds to the right shoulder of the model human body. The right elbow feature point RE corresponds to the right shoulder of the model human body. The right wrist feature point RW corresponds to the right wrist of the model human body. The neck feature point N is connected to the right shoulder feature point RS by a skeleton line. The right elbow feature point RE is connected to each of the right shoulder feature point RS and the right wrist feature point RW by skeleton lines.

[0025] The processor 122 identifies points of the driver 30 captured in the image IM that correspond to the respective feature points of the skeleton model M (STEP 3 in FIG. 3). In other words, feature points corresponding to specific body parts of the driver 30 are identified in the image IM. The processor 122 connects the identified multiple points with the above-mentioned skeleton lines. The algorithm for performing this process is well known, and therefore a detailed description thereof will be omitted.

[0026] The processor 122 determines the operation of the air conditioner 13 based on at least one of the identified feature points, and outputs a control signal CT corresponding to the operation to the output interface 123 (STEP 4). Specifically, the control signal CT is configured to determine at least one of the flow rate and the direction of the airflow sent by the air conditioner 13 into the vehicle compartment 21. After that, the process returns to STEP 1.

[0027] The operation of the air conditioner 13 controlled by the control device 12 can be specified by the driver 30. As an example, as illustrated in Fig. 6, a setting can be made to send out airflow AF so as to avoid the face of the driver 30. In this case, the processor 122 of the control device 12 identifies a neck feature point N corresponding to the neck of the driver 30 reflected in the image IM acquired by the imaging device 11, and outputs a control signal CT from the output interface 123 to cause the air conditioner 13 to send out airflow AF to the left and right of the neck feature point N. The symbol T represents the target position of the sent out airflow AF. This can provide an air-conditioned environment suitable for the driver 30 who dislikes airflow hitting his face.

[0028] 7, a setting may be made to send airflow AF toward both sides of the driver 30. In this case, the processor 122 of the control device 12 identifies a left shoulder characteristic point LS and a right shoulder characteristic point RS corresponding to the left and right shoulders of the driver 30 captured in the image IM acquired by the imaging device 11, and outputs a control signal CT to the air conditioner 13 to send airflow AF downward from the output interface 123. This makes it possible to provide an air-conditioned environment suitable for the driver 30 who wants to feel cool quickly.

[0029] According to the configuration of this embodiment, when the driver 30 is imaged by the imaging device 11, the skeleton model M is applied to the image IM in which the driver 30 appears, and feature points corresponding to the body parts of the driver 30 required to cause the air conditioner 13 to perform a desired operation are identified. Therefore, the driver 30 can cause the air conditioner 13 to perform a desired operation without needing to actively act on the imaging device 11.

[0030] In the process described with reference to Fig. 6, airflow is sent from the air conditioner 13 to the left and right of the neck characteristic point N corresponding to the neck of the driver 30. In this case, as illustrated in Fig. 8, the processor 122 of the control device 12 sets an area A that may include the neck characteristic point N in the image IM in which the driver 30 appears.

[0031] Area A may be determined in advance as a range in which the neck characteristic point N is likely to be located when the driver 30 seated in the seat 22 assumes a normal posture, or it may be dynamically determined so that the boundary is a specific distance from the identified neck characteristic point N.

[0032] In this example, the processor 122 is configured to monitor whether the neck feature point N is located inside the area A. When it is detected that the neck feature point N has moved outside the area A due to the driver 30 changing his / her posture as illustrated in FIG. 9, the processor 122 changes at least one of the amount and the direction of the airflow AF delivered from the air conditioner 13.

[0033] As an example, as shown by the dashed line in Fig. 9, the delivery of the airflow AF toward the left of the neck characteristic point N of the original driver 30 may be stopped. Alternatively, as shown by the arrow in Fig. 9, the delivery direction of the airflow AF may be changed so as to avoid the neck characteristic point N of the driver 30 after the movement.

[0034] With this configuration, an air-conditioning environment suitable for the driver 30 who dislikes airflow hitting his / her face can be maintained.

[0035] The processor 122 of the control device 12 having the various functions described above may be realized by a general-purpose microprocessor operating in cooperation with a general-purpose memory. Examples of the general-purpose microprocessor include a CPU, an MPU, and a GPU. Examples of the general-purpose memory include a ROM and a RAM. In this case, a computer program for realizing the function may be stored in the ROM. The ROM is an example of a non-transitory computer-readable medium storing a computer program. The general-purpose microprocessor specifies at least a part of the computer program stored in the ROM, expands it on the RAM, and executes the above-mentioned process in cooperation with the RAM. The above-mentioned computer program may be pre-installed in the general-purpose memory, or may be downloaded from an external server via a communication network and installed in the general-purpose memory. In this case, the external server is an example of a computer-readable medium storing a computer program.

[0036] The processor 122 may be realized by a dedicated integrated circuit capable of executing the above computer program, such as a microcontroller, an ASIC, or an FPGA. In this case, the above computer program is pre-installed in a storage element included in the dedicated integrated circuit. The storage element is an example of a computer-readable medium storing a computer program. The processor 122 may also be realized by a combination of a general-purpose microprocessor and a dedicated integrated circuit.

[0037] The configurations described above are merely examples for facilitating understanding of the present disclosure. Each configuration example may be appropriately modified and combined with other configuration examples without departing from the spirit of the present disclosure.

[0038] The control device 12 does not need to control the operation of the air conditioner 13 only for the driver 30 of the vehicle 20. The skeleton model M is also applied to a passenger of the vehicle 20 who may appear in the image IM acquired by the imaging device 11, and the operation of the air conditioner 13 can be controlled based on the identified feature points corresponding to the body parts of the passenger. In other words, the passenger of the vehicle 20 is an example of a user.

[0039] The object to be controlled based on the feature points corresponding to the body parts of the driver 30 identified by applying the skeletal model M to the subject captured in the image IM acquired by the imaging device 11 is not limited to the operation of the air conditioner 13.

[0040] An actuator that adjusts at least one of the position and posture of the seat 22 in which the driver 30 sits can be an example of a controlled device. For example, the processor 122 of the control device 12 can monitor whether the posture of the driver 30 is normal based on a plurality of feature points and a plurality of skeletal lines exemplified in FIG. 5. If it is determined that the posture is not normal, the processor 122 can output a control signal CT from the output interface 123 that causes the actuator to adjust at least one of the position and posture of the seat 22. For example, at least one of the position and posture of the seat 22 can be changed to prompt the driver 30 to pay attention to abnormal posture or correct the posture.

[0041] An actuator that adjusts the tension of the seat belt 23 (see FIG. 4) worn by the driver 30 can also be an example of a controlled device. In this example, the processor 122 of the control device 12 can monitor whether the posture of the driver 30 is normal based on a plurality of feature points and a plurality of skeletal lines exemplified in FIG. 5. If it is determined that the posture is not normal, the processor 122 can output a control signal CT from the output interface 123 to cause the actuator to adjust the tension of the seat belt 23. For example, the tension of the seat belt 23 can be changed to prompt the driver 30 to pay attention to abnormal posture or to correct the posture.

[0042] An actuator for adjusting the positional relationship between the seat 22 and the steering device 24 illustrated in Fig. 1 may also be an example of a controlled device. In this example, the processor 122 of the control device 12 may monitor whether the posture of the driver 30 holding the steering device 24 is normal based on a plurality of feature points and a plurality of skeleton lines illustrated in Fig. 5. If it is determined that the posture is not normal, the processor 122 may output a control signal CT from the output interface 123 to cause the actuator to adjust the positional relationship between the seat 22 and the steering device 24. For example, the positional relationship between the seat 22 and the steering device 24 may be changed so that the posture of the driver 30 can be corrected.

[0043] An actuator for adjusting the attitude of a rearview mirror arranged in the vehicle interior 21, a display arranged in a center cluster, or the like, can also be an example of a controlled device. In this example, the processor 122 of the control device 12 can output a control signal CT from the output interface 123 to cause the actuator to adjust the attitude of the rearview mirror, display, or the like according to the position of the neck characteristic point N illustrated in Fig. 4. For example, the rearview mirror, display, or the like can be changed to assume an easy-to-view position according to the driver 30's posture.

[0044] The control system 10 may be mounted on a moving object other than the vehicle 20. Examples of such moving objects include trains, airplanes, ships, etc. The moving object may not require a driver.

[0045] The control system 10 does not necessarily need to be mounted on a moving body. As long as the operation of the controlled device is controlled based on feature points corresponding to the user's body parts identified by applying the skeleton model M to a subject captured in an image IM acquired by the imaging device 11, the control system 10 can be mounted on an appropriate facility used by the user. [Explanation of symbols]

[0046] 10: control system, 11: imaging device, 12: control device, 121: input interface, 122: processor, 13: air conditioner, 21: vehicle interior, 22: seat, 23: seat belt, 30: driver, A: area, AF: air flow, ID: image data, IM: image, LS: left shoulder feature point, M: skeletal model, N: neck feature point, RS: right shoulder feature point

Claims

1. an imaging device that outputs image data corresponding to an image in which a subject is captured; A controlled device; a control device that applies a skeletal model to the subject by processing the image data, the skeletal model including a plurality of connected feature points, each of which corresponds to a specific body part, to the subject, thereby identifying feature points in the image that correspond to specific body parts, and controls the operation of the controlled device with respect to the user based on the positions of the feature points; Contains Control system.

2. the controlled device is an air conditioner, The control device controls at least one of a volume and a direction of an airflow delivered to the user by the air conditioning device. The control system of claim 1 .

3. The control device includes: A region including the feature point is set in the image; when it is detected based on the image data that the feature point has moved outside the region, changing at least one of a volumetric flow rate and a direction of the airflow. The control system of claim 2.

4. The controlled device is an actuator that adjusts at least one of the position and posture of a seat on which the user sits. The control system of claim 1 .

5. The controlled device is an actuator that adjusts the tension of a seat belt worn by the user. The control system of claim 1 .

6. an interface that receives image data corresponding to an image in which a subject is captured from an imaging device; a processor that applies a skeletal model to the subject, the skeletal model including a plurality of connected feature points, each of which corresponds to a specific body part, to the image data, thereby identifying feature points in the image that correspond to specific body parts, and controls an operation of a controlled device with respect to a user based on the positions of the feature points; Equipped with Control device.

7. A computer program executable by a processor mounted on the control device, By being executed, the control device receiving image data corresponding to an image in which a subject is captured from an imaging device; applying a skeletal model to the subject, the skeletal model including a plurality of connected feature points, each of the feature points corresponding to a specific body part, to the image data to identify the feature points in the image that correspond to a specific body part; controlling an action of a controlled device relative to a user based on the location of the feature point; Computer program.

Images