Control system and control method for controlling equipment in a vehicle

Abstract

This invention provides a novel control system and control method capable of expressing diverse functions in a complex manner through changes in surface shape. [Solution] The control system comprises at least one control unit (processor of the information processing device 8), the control unit comprising a contact detection unit 93 that detects the user's contact state with the operating surface 92a that the user can touch, and a shape changing unit 10 that changes the surface shape of the operating surface in a predetermined operating mode according to the detection result of the contact state. The operating modes include a mode in which the surface shape is changed according to the user's behavior and a mode in which the surface shape is changed regardless of the user's behavior.

Description

【Technical Field】 【0001】 The present invention relates to a control system and a control method for controlling devices in a vehicle. 【Background Art】 【0002】 Patent Document 1 discloses a human-machine interface technology capable of appropriately notifying a user of information without occupying the user's vision and hearing. The technology is provided with a tactile notification device having a plurality of protruding members arranged in a predetermined rule along a reference plane and having a tip portion provided so as to be able to protrude from the reference plane. A vehicle driving support system includes a target operation amount setting unit that sets a target operation amount, which is an operation amount of a driving device with respect to a driving support target, an actual operation amount acquisition unit that acquires an actual operation amount, which is an actual operation amount of the driving device by a vehicle occupant, and a protrusion control unit that controls each protrusion height of the protruding members with respect to the reference plane. The protrusion control unit controls so as to make the protrusion forms of the plurality of protruding members different according to the difference between the target operation amount and the actual operation amount. 【Prior Art Documents】 【Patent Documents】 【0003】 【Patent Document 1】 Japanese Unexamined Patent Application Publication No. 2013-184612 【Summary of the Invention】 【Problems to be Solved by the Invention】 【0004】 By the way, a human-machine interface device using an actuator for controlling the protrusion height, like the technology described in Patent Document 1, tends to be more expensive than a digital display device such as a display. Therefore, it is preferable that the human-machine interface device can comprehensively express various functions such as a function of presenting information for a specific situation, a function of presenting information in a normal state, a function of operating a device, and a tactile sensation due to deformation when touched, through a change in the surface shape. [Means for solving the problem] 【0005】 According to one aspect of the present invention, a control system is provided comprising at least one control unit, the control unit being configured to change the surface shape of an operating surface using a shape changing unit that changes the surface shape of the operating surface in a predetermined operating mode, in accordance with the results of a contact state detection unit that detects the user's contact state with an operating surface that the user can touch, the operating mode including a mode in which the surface shape is changed according to the user's behavior and a mode in which the surface shape is changed regardless of the user's behavior. 【0006】 This configuration provides a novel control system capable of comprehensively representing diverse functions through changes in surface shape. [Brief explanation of the drawing] 【0007】 [Figure 1] This is a diagram showing the configuration of control system 1. [Figure 2] This is a block diagram showing the hardware configuration of the information processing device 8. [Figure 3] This figure shows an example of the peripheral configuration of the HMI system 9 within the control system 1. [Figure 4] This is an exploded perspective view showing an example configuration of the HMI device 92. [Figure 5] This is an exploded view showing an example of the configuration of the shape-changing section 10. [Figure 6] This figure shows an example of an HMI device 92 on which an image is projected. [Figure 7] This figure shows 92 specific examples of HMI devices as shown in Figure 6. [Figure 8] This figure illustrates an example of the behavior of the HMI device 92 in operation mode M1. [Figure 9] This figure shows the behavior of a specific example of the HMI device 92 shown in Figure 7 when a hand is placed over its operating surface. [Figure 10]This flowchart shows an example of the information processing flow performed in control system 1. [Figure 11] This diagram summarizes the information regarding each operating mode M in this embodiment. [Figure 12] This photograph shows a specific example of an HMI device 92 in which the surface shape of the operating surface 92a changes so that a recess Rc along the vertical direction moves parallel to the horizontal direction. [Figure 13] This figure shows an example of an HMI device 92 that changes the display mode of the image projected onto the operating surface 92a in accordance with changes in the surface shape. [Figure 14] This figure shows an example of an HMI device 92 that changes the display mode of the image projected onto the operating surface 92a in accordance with changes in the surface shape. [Modes for carrying out the invention] 【0008】 Embodiments of the present invention will be described below with reference to the drawings. The various features shown in the embodiments below can be combined with each other. 【0009】 Incidentally, the program for implementing the software appearing in one embodiment may be provided as a non-transitory computer-readable medium, or it may be provided as a downloadable medium from an external server, or it may be provided so that the program is launched on an external computer and its functions are realized on a client terminal (so-called cloud computing). 【0010】 Furthermore, in various information processing according to one embodiment, an input and an output corresponding to the input can be realized. Here, as long as an output is obtained as a result of the input, the form of the information referenced in such information processing (hereinafter referred to as "reference information") is not limited. The reference information may be, for example, rule-based information such as a database, a lookup table, or a predetermined function (including a decision formula such as a regression equation constructed by a statistical method), or a pre-trained model that has learned the correlation between input and output in advance, or a large-scale language model that can output a desired result by inputting a prompt. 【0011】 Furthermore, in one embodiment, "part" may include, for example, hardware resources implemented by a circuit in a broad sense, and the information processing of software that can be specifically realized by these hardware resources. Also, in one embodiment, various types of information are handled, and this information can be represented, for example, by the physical values ​​of signal values ​​representing voltage and current, the high or low values ​​of signal values ​​as a set of binary bits composed of 0s or 1s, or by quantum superposition (so-called qubits), and communication and calculations can be performed on a circuit in a broad sense. 【0012】 Furthermore, a circuit in a broad sense is a circuit realized by combining at least a suitable combination of circuits, circuits, processors, and memory. The processor may be a general-purpose processor or a dedicated circuit. In other words, it includes application-specific integrated circuits (ASICs), programmable logic devices (for example, simple programmable logic devices (SPLDs), complex programmable logic devices (CPLDs), and field programmable gate arrays (FPGAs)), etc. 【0013】 1. Hardware Configuration In this section, the hardware configuration will be described. 【0014】 <Control System 1> FIG. 1 is a configuration diagram showing Control System 1. Control System 1 includes, for example, Vehicle 2. 【0015】 Vehicle 2 is configured to be operable by a passenger U, which is an example of a user. Vehicle 2 may be a general passenger car or an industrial vehicle such as a forklift or a towing tractor. Vehicle 2 includes a seat 3, a steering wheel H, an in-vehicle camera 4 which is an example of an imaging device, a biometric sensor 5 as an example of a biometric signal detection unit, a car audio 6, a display unit 7, an information processing device 8, and an HMI system 9. The car audio 6 is an example of a device in the vehicle in the present embodiment (in other words, a control target in the present Control System 1). 【0016】 Seat 3 is configured such that a passenger U, which is an example of a user, can sit on it. In the present embodiment, the passenger U is the driver of Vehicle 2, but it is not limited to the driver and may be a passenger. 【0017】 Steering wheel H is configured to be able to change the traveling direction of Vehicle 2 by rotating the wheels of Vehicle 2. The rotation angle of the steering wheel H can be associated with the direction of the curve of Vehicle 2. 【0018】 In-vehicle camera 4 is configured to generate an in-vehicle image IM1, which is an example of a target image, by imaging at least a part of the surroundings of Vehicle 2. The in-vehicle camera 4 in the present embodiment is a so-called front camera configured to image the front of Vehicle 2. Note that the in-vehicle camera 4 may be a so-called back camera configured to image the rear of Vehicle 2 when traveling backward, or a combination of these. 【0019】 The in-vehicle image IM1 can change over time. The temporal change in the in-vehicle image IM1 can be caused by the relative movement of objects within the imaging range of the in-vehicle camera 4 relative to the vehicle 2. For example, the temporal change in the in-vehicle image IM1 tends to increase as the speed of the vehicle 2 increases. Also, the temporal change in the in-vehicle image IM1 tends to increase in proportion to the amount of change in the direction of travel of the vehicle 2, such as due to sharp turns. Furthermore, the temporal change in the in-vehicle image IM1 tends to increase in the area corresponding to the wall, as the proximity of objects around the vehicle 2, such as a wall, increases. Therefore, the temporal change in the in-vehicle image IM1 tends to increase when the occupant U is required to pay a relatively large amount of attention when operating the vehicle 2. 【0020】 The biosensor 5 is configured to detect the biosignals of the passenger U. The biosignals represent the physical state of the passenger U. The biosignals may be characterized, for example, by a specific frequency that indicates the physical state. The biosignals may include information about the user's breathing or pulse. More specifically, such biosignals include signals that can suggest the unconscious actions of the passenger U, such as pulse waves, pulse rate, and breathing patterns. For the sake of explanation, the biosignals below will be assumed to represent the pulse waves of the passenger U. In this case, the specific frequency may be defined, for example, by the pulse rate. The biosensor 5 is preferably a so-called wearable device configured to detect biosignals by being attached to the body of the passenger U. With such a configuration, weaker biosignals can be detected with greater accuracy. The biosensor 5 is not limited to this, and may also be, for example, a portable terminal such as a smartphone with an application for detecting biosignals installed, or a camera that estimates biosignals by photographing the user's body. 【0021】 The car audio system 6 is configured to output auditory information that can be perceived by the occupant U through their hearing. This auditory information may include information about various sounds that the user can hear, such as sound effects, warning sounds, and music. 【0022】 The display unit 7 is configured to display a graphical user interface (GUI) screen that can be operated by the user. The display unit 7 may be implemented using a display device such as a CRT display, liquid crystal display, organic EL display, or plasma display. The display unit 7 may be integrated with the vehicle 2, for example, as an instrument panel, or it may be an external device, for example, a display for car navigation or a smartphone that can be attached to the vehicle 2. The display unit 7 may be configured to display information that can be seen by the passenger U, for example, information regarding the volume of the car audio 6 or the music being played. 【0023】 The information processing device 8 is configured to communicate via a telecommunications line with various sensors and electrical components installed in the vehicle 2, such as the in-vehicle camera 4, biosensor 5, car audio system 6, display unit 7, etc. In one embodiment, the control system 1 consists of one or more devices or components. For example, if it consists only of the information processing device 8, then the control system 1 can be the information processing device 8. The specific configuration of the information processing device 8 will be described later. 【0024】 The HMI system 9 is a human-machine interface device that can be operated by the occupant U. The HMI system 9 is a device for controlling equipment in the vehicle 2 (in this embodiment, the car audio 6 in particular). The HMI system 9 can constitute the control system 1 on its own. The specific configuration of the HMI system 9 will be described later. 【0025】 2. An example of hardware configuration around the information processing device 8 Figure 2 is a block diagram showing the hardware configuration of the information processing device 8. The information processing device 8 comprises a communication unit 81, a storage unit 82, a processor 83 as at least one control unit, and an input unit 84. These components are electrically connected within the information processing device 8 via a communication bus 80. 【0026】 The communication unit 81 preferably uses wired communication methods such as USB, IEEE1394, Thunderbolt®, and wired LAN network communication, but may also include wireless LAN network communication, mobile communication such as 3G / LTE / 5G, and Bluetooth® communication as needed. In other words, it is more preferable to implement it as a collection of these multiple communication methods. That is, the information processing device 8 may communicate various information from the outside via the communication unit 81 and the network. 【0027】 The storage unit 82 stores various types of information as defined above. This can be done, for example, as a storage device such as a solid-state drive (SSD) that stores various programs related to the information processing device 8 executed by the processor 83, or as memory such as random access memory (RAM) that stores temporarily necessary information (arguments, arrays, etc.) related to program calculations. The storage unit 82 stores various programs and variables related to the information processing device 8 executed by the processor 83. 【0028】 The processor 83 performs processing and control of the overall operation related to the information processing device 8. The processor 83 is, for example, a central processing unit (CPU) not shown. The processor 83 realizes various functions related to the information processing device 8 by reading predetermined programs stored in the memory unit 82. That is, information processing by software stored in the memory unit 82 is concretely realized by the processor 83, which is an example of hardware, and can be executed as each functional unit included in the processor 83. These will be described in more detail in the next section. Note that the processor 83 is not limited to being a single unit, and may be implemented with multiple processors 83 for each function, or a combination thereof. 【0029】 The processor 83 is configured to acquire information from devices such as the in-vehicle camera 4. The processor 83 is configured to acquire various information by reading various information stored in the storage area, which is at least a part of the memory unit 82, and writing the read information to the work area, which is at least a part of the memory unit 82. The storage area is, for example, the area of ​​the memory unit 82 that is implemented as a storage device such as an SSD. The work area is, for example, the area that is implemented as memory such as RAM. The acquisition by the processor 83 includes acquiring the output results of each functional unit included in the processor 83. 【0030】 The processor 83 is configured as a display processing unit to display various types of information. This information can be presented to the user via the display unit 7 or other devices. In such cases, for example, the processor 83 controls the display unit 7 to display visual information such as screens, images including still images or moving images, icons, and messages. The processor 83 may generate only rendering information for displaying the visual information on the display unit 7. The processor 83 may also present the outputted information to the user without going through the display unit 7 or other devices. 【0031】 The input unit 84 is configured to accept input from the user. The input unit 84 may be included in the housing of the vehicle 2 or it may be externally mounted. For example, the input unit 84 may be implemented as a touch panel integrated with the display unit 7. If it is a touch panel, the user can input tap operations, swipe operations, etc. Of course, instead of a touch panel, a switch button, mouse, QWERTY keyboard, voice recognition device, gesture detection device, gaze detection device, biosignal detection device, imaging device, etc. may be used. In other words, the input unit 84 accepts operation input made by the user. In response, the input unit 84 transmits a signal corresponding to the operation input to the processor 83 via the communication bus 80. The processor 83 can perform predetermined controls and calculations as needed. 【0032】 3. Example configurations for the HMI system 9 Next, the peripheral configuration of the HMI system 9 within the control system 1 will be described. Figure 3 is a diagram showing an example of the peripheral configuration of the HMI system 9 within the control system 1. As shown in Figure 3, the HMI system 9 includes a projector 91 as an example of an image display unit, an HMI device 92, and a contact detection unit 93. 【0033】 The projector 91 is configured to display a predetermined image on the operating surface 92a, which will be described later. 【0034】 The HMI device 92 functions as a controller that changes the operating mode of equipment in the vehicle 2, such as the car audio 6, by, for example, making contact with the passenger U's hand UH. Here, an example of the configuration of the HMI device 92 will be described with reference to Figures 4 and 5. Figure 4 is an exploded perspective view showing an example of the configuration of the HMI device 92. The HMI device 92 comprises a housing 921, an expandable sheet 922, and a shape-changing part 10. 【0035】 The enclosure 921 is a rectangular box-shaped body having a bottom wall and side walls, and has an opening at a position opposite the bottom wall. 【0036】 The expandable sheet 922 is formed to cover the opening of the housing 921 and has an operating surface 92a that can be touched by the occupant U operating the equipment. The operating surface 92a is formed by covering it with a flexible, expandable body that extends two-dimensionally and can induce irregularities on that plane. The specific configuration of the operating surface 92a is not limited to this and is arbitrary as long as the surface shape can be changed. Images from the projector 91 can be projected onto the operating surface 92a. This makes it possible to perform effects such as projection mapping on the operating surface 92a. 【0037】 The shape-changing section 10 is a mechanism that changes the surface shape of the operating surface 92a. The shape-changing section 10 is housed in an internal space defined by the housing 921 and the expandable sheet 922, and is configured to deform the uneven shape of the operating surface 92a of the expandable sheet 922. Figure 5 is an exploded view showing an example of the configuration of the shape-changing section 10. As shown in Figure 5, the shape-changing section 10 comprises a support plate 11, a solenoid 12, a push bar 13, a support column 14, a peripheral wall 15, and a cushioning material 16. 【0038】 The support plate 11 is a rectangular plate that can be housed in the housing 921. 【0039】 The solenoid 12 is positioned to penetrate the support plate 11 perpendicularly to the surface on which the support plate 11 extends. The solenoids 12 are arranged two-dimensionally at equal intervals within the plane of the support plate 11. The number and position of the solenoids 12 are arbitrary. The solenoids 12 can drive an internal motor by an external electrical signal. 【0040】 The push bar 13 is a cylindrical member that extends perpendicularly to the surface on which the support plate 11 extends. It is configured to extend and retract in this vertical direction in conjunction with the drive of the solenoid 12 motor. 【0041】 The support column 14 is a cylindrical member that extends perpendicularly to the surface on which the support plate 11 extends, and is located on the outer edge of the surface on which the support plate 11 extends. In this embodiment, 10 support columns 14 are arranged on the surface. 【0042】 The peripheral wall 15 is connected to the outer edge of the support plate 11 and is positioned along the circumferential surface of the support column 14. The peripheral wall 15 serves to protect, for example, the solenoid 12 and the push bar 13 that penetrate the support plate 11. 【0043】 The cushioning material 16 is a sheet-like member positioned to contact the ends of the push bar 13 and the support column 14. The cushioning material 16 is, for example, a gel sheet and is configured to be elastically deformable at least perpendicular to the surface on which the support plate 11 extends. As the position of the push bar 13 changes due to the driving of the solenoid 12, the uneven shape of the cushioning material 16 changes. As shown in Figures 4 and 5, the cushioning material 16 is configured to contact the back surface of the operating surface 92a of the expandable sheet 922. The push bar 13 can change the surface shape (in this case, the uneven shape) of the operating surface 92a via the cushioning material 16. At this time, the cushioning material 16 can protect the push bar 13 by distributing the stress that the push bar 13 applies to the expandable sheet 922. 【0044】 Returning to Figure 3, the contact detection unit 93 is configured to detect the contact state of the passenger U with respect to the operating surface 92a. For example, the contact detection unit 93 is configured to detect the position of the passenger U's hand UH in contact with the operating surface 92a. In this embodiment, the contact detection unit 93 is a camera that images the operating surface 92a. The contact detection unit 93 generates a surface image IM2 as a result of imaging the operating surface 92a. The contact detection unit 93 is arranged in a manner that allows it to image the entire operating surface 92a and the hand UH located on the operating surface 92a. Based on the position of the hand UH relative to the operating surface 92a in the surface image IM2, it is possible to identify the contact state and contact position of the hand UH. Note that the contact detection unit 93 does not need to actually detect whether the hand UH is in contact with the operating surface 92a; it is sufficient to determine that there is a possibility of contact. In other words, detecting a contact state is not limited to detecting that the passenger U is actually in contact with the operating surface 92a, but may also include detecting the possibility of contact, such as by holding a hand UH over the operating surface 92a. Furthermore, the specific form of the contact detection unit 93 is not limited to non-contact methods such as cameras, but may also be a touch sensor that detects direct contact, for example. 【0045】 Next, we will briefly explain the relationship between the in-vehicle camera 4, biosensor 5, car audio system 6, and information processing device 8 in relation to the HMI system 9. Details of the information processing performed by these devices will be described in subsequent sections. 【0046】 The information processing device 8 acquires an in-vehicle image IM1 from the in-vehicle camera 4 and a biosensor D1 from the biosensor 5. The information processing device 8 also transmits a video signal s1. The video signal s1 is a signal indicating the image projected onto the operating surface 92a by the projector 91. The projector 91 projects the image onto the operating surface 92a based on the video signal s1. Subsequently, the contact detection unit 93 captures an image of the operating surface 92a on which the image has been projected and generates a surface image IM2. The generated surface image IM2 is transmitted to the information processing device 8, and the processor 83 of the information processing device 8 determines the position of the hand UH on the operating surface 92a (or whether the hand UH is in contact with the operating surface 92a) based on the surface image IM2 using an image classifier with known image processing techniques. The processor 83 may also further determine the uneven shape of the surface image IM2 (for example, the position of depressions or protrusions) based on the surface image IM2. 【0047】 Next, the processor 83 of the information processing device 8 changes the surface shape of the operating surface 92a using the shape changing unit 10 in a predetermined operating mode, according to the contact state detection result by the contact detection unit 93. For example, the processor 83 changes the surface shape (particularly the uneven shape) of the operating surface 92a by transmitting a control signal s2 to the shape changing unit 10 to drive the solenoid 12 of the shape changing unit 10. Here, the operating mode M includes a mode in which the surface shape of the operating surface 92a is changed according to the behavior of the occupant U, and a mode in which the surface shape of the operating surface 92a is changed regardless of the behavior of the occupant U, and may include, for example, an operating mode M1 and a standby mode M2. 【0048】 Operation mode M1 is executed when contact of the passenger U with the operating surface 92a is detected. In operation mode, the processor 83 changes the surface shape of the operating surface 92a to track the point of contact with the user, and changes the operating mode (e.g., volume) of the car audio 6 in conjunction with the change in the surface shape of the operating surface 92a. Operation mode M1 allows the HMI device 92 to function as a controller for controlling the operation of the car audio 6. When the passenger U performs an operation on the car audio 6, the processor 83 of the information processing device 8 sends an output signal s3 to the car audio 6, and the car audio 6 changes its operation based on the output signal s3. Operation mode M1 is an example of a mode in which the surface shape of the operating surface 92a is changed according to the behavior of the passenger U. 【0049】 Standby mode M2 ​​is executed when no contact by the occupant U with the operating surface 92a is detected. In standby mode M2, the processor 83 deforms the surface shape of the operating surface 92a regardless of the occupant U's behavior toward the operating surface 92a. With this configuration, when the occupant U controls the device, they can control the device through contact with the operating surface 92a and tactilely confirm the content of the operation as a deformation of the operating surface 92a's shape, thus providing the occupant U with a more reliable sense of operation compared to simply using touch. On the other hand, even when no operation is being performed, the deformation of the surface shape visually presents the position of the operating surface 92a to the occupant U. Therefore, a control system with higher operability can be provided, for example, even when the vehicle 2 is in motion. Standby mode M2 ​​is an operating mode that indicates a so-called standby state, where the occupant U is not operating the car audio 6 through the HMI device 92. In this case, the random change in the surface shape of the operating surface 92a reduces the possibility of giving the occupant U an inorganic and mechanical impression of the HMI device 92, and makes the interior space of the vehicle 2 more comfortable. The standby mode M2 ​​may include a mode in which the surface shape of the operating surface 92a is changed in response to the occupant U's behavior other than operation on the operating surface 92a, and a mode in which the surface shape of the operating surface 92a is changed regardless of the occupant U's behavior. 【0050】 Figure 6 shows an example of an HMI device 92 on which an image is projected. Figure 7 shows a specific example of the HMI device 92 shown in Figure 6. As shown in Figures 6 and 7, the projected image IM3 displayed by the projector 91 is projected onto the operating surface 92a of the HMI device 92, and the position and shape of the recess Rc on the operating surface 92a are configured to change. 【0051】 Figure 8 is a diagram illustrating an example of the behavior of the HMI device 92 in operation mode M1. Figure 9 is a diagram showing the behavior when a hand is placed over the operating surface of a specific example of the HMI device 92 shown in Figure 7. As shown in Figures 8 and 9, when a passenger U places their hand over the operating surface 92a and it is determined that the hand UH is in contact with the operating surface 92a, the operation mode M is set to operation mode M1. In operation mode M1, the push bar 13 in the area where it is determined that the hand UH and the operating surface 92a are in contact retracts, forming a recess in that area. Subsequently, the push bar 13 corresponding to the area to which the hand UH is moving retracts to follow the movement of the hand UH on the operating surface 92a. At this time, the push bar 13 corresponding to the area where contact with the hand UH has been released protrudes again. In this way, the processor 83 changes the surface shape of the operating surface 92a. At this time, the volume of the car audio 6 is increased or decreased according to the direction and amount of movement of the hand UH. In other words, in operation mode M1, when a swipe operation on the operating surface 92a is detected as contact by the passenger U, the processor 83 deforms the surface shape of the operating surface 92a to follow the swipe operation, and changes the operating mode of the device according to the direction of the swipe operation and the amount of movement of the contact point of the passenger U. With this configuration, even in situations such as when the vehicle 2 is in motion, the fact that an operation has been performed on the device can be presented to the passenger U in a more tactile and emphasized manner. Note that operations on the operating surface 92a are not limited to such swipe operations, but may include any operation by the hand UH, such as tap operations, double tap operations, and flick operations. Here, the HMI device 92 may spontaneously form a new indentation on the operating surface 92a and spontaneously change its surface shape without requiring an action to push in the push bar 13 by pressing with the hand UH. 【0052】 4. An example of information processing This section describes specific examples of information processing performed in the control system 1 mentioned above. 【0053】 4.1. Information Processing Flow Figure 10 is a flowchart illustrating an example of the information processing flow performed in control system 1. Note that this information processing may include arbitrary exception handling not shown. Exception handling includes interrupting the information processing or omitting individual processes. The selections or inputs made in this information processing may be based on user operation or performed automatically without user operation. 【0054】 [Step S1] As shown in Figure 10, in step S1, the control system 1 first supplies power to the electrical components in the vehicle 2 in response to an operation from the occupant U. An operation from the occupant U is, for example, turning on the ignition (IG) power or accessory (ACC) power of the vehicle 2. This supplies power from the vehicle 2's battery to the onboard camera 4, biosensor 5, car audio 6, display unit 7, information processing device 8, and HMI system 9, making these electrical components operational. At this time, the HMI system 9 projects an image onto the operating surface 92a using the projector 91, and the shape change unit 10 operates in a predetermined standby mode (in particular, standby mode A, which will be described later), randomly changing the surface shape of the operating surface 92a. The contact detection unit 93 generates a surface image IM2 by capturing images of the operating surface 92a as the image is projected and the surface shape changes. The processor 83 detects the contact state of the occupant U with the operating surface 92a based on the generated surface image IM2. Furthermore, the processor 83 may detect the driving pattern of the vehicle 2 (for example, the direction of the curve, the speed of travel, acceleration, deceleration, etc.) based on the steering angle of the steering wheel H, the driving state of the accelerator, changes in the in-vehicle image IM1, etc. In this case, the processor 83 can function as a driving detection unit that detects the driving pattern of the vehicle 2. 【0055】 [Step S2] Next, in step S2, the processor 83 acquires the contact state detection result from the contact detection unit 93. The processor 83 may further acquire the driving mode of the vehicle 2, the in-vehicle image IM1 from the in-vehicle camera 4, the biosignal D1 from the biosensor 5, etc. In particular, when the processor 83 acquires the in-vehicle image IM1, it may calculate a complexity parameter indicating the degree of the change in the environment based on the time change of the in-vehicle image IM1, which is an example of the detection result of a change in the environment. 【0056】 For example, the processor 83 calculates a complexity parameter based on the difference between the latest image in the in-vehicle image IM1 and an image in the in-vehicle image IM1 from a predetermined period prior to a certain image (e.g., the current image). With this configuration, the difference between an image from a predetermined period prior and the current image can be visually grasped. When the temporal element is represented using frames, the predetermined period is specifically, for example, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 frames, and may be within the range of any two of the numbers exemplified here. In this embodiment, the processor 83 calculates a complexity parameter based on the difference between the latest image in the in-vehicle image IM1 and the image immediately preceding the latest image in the in-vehicle image IM1 (in other words, the image one frame prior). Specifically, the processor 83 identifies the number of pixels that differ between the latest image and the previous image, and calculates the complexity parameter from a relational expression relating the number of pixels to the complexity parameter. With this configuration, the time required to calculate the complexity parameter can be shortened, and the complexity of changes in the in-vehicle image IM1 can be evaluated in more real time. If the in-vehicle image IM1 does not change, the number of such pixels will be approximately 0, and the number of such pixels will increase as the complexity of the changes in the in-vehicle image IM1 increases. Therefore, the number of nearly different pixels correlates with the complexity of the changes in the in-vehicle image IM1. Note that the method of calculating the complexity parameter is not limited to this and is arbitrary; for example, the complexity parameter may be calculated using a pre-trained model that takes multiple images (e.g., two) as input and outputs the complexity parameter for those multiple images. 【0057】 [Step S3] Next, in step S3, the processor 83 sets the operating mode M of the HMI device 92 according to the results obtained in step S2. The operating mode M may include the operation mode M1 and the standby mode M2, as described above. 【0058】 [Step S4] Next, in step S4, the processor 83 controls the shape changing unit 10 according to the set operating mode M, thereby changing the surface shape of the operating surface 92a of the HMI device 92. For example, the processor 83 drives each of the multiple solenoids 12 in a pattern corresponding to the set operating mode M, individually changing the position of the push bar 13 that pushes up (or pulls down) the operating surface 92a. As a result, the height of the area on the operating surface 92a corresponding to the push bar 13 whose position has changed changes, and the uneven shape of the operating surface 92a changes. 【0059】 [Step S5] Next, in step S5, the processor 83 determines whether the set operating mode M is operation mode M1 or not. 【0060】 [Step S6] If the set operating mode M is operation mode M1, the process proceeds to step S6, where the processor 83 changes the operating mode (in this case, volume) of the car audio 6 in response to the operation by the passenger U. If it is determined in step S5 that the operating mode M is set to standby mode M2, the process in step S6 is omitted. 【0061】 [Step S7] Next, in step S7, it is determined whether or not the power supply to the HMI device 92, etc., has been stopped. If the power supply is continuing, the process returns to step S2, and the processor 83 obtains the latest contact status detection results, etc., and in step S3, updates the operation mode M according to the latest obtained results. 【0062】 [Step S8] On the other hand, if the power supply is stopped in step S7, the control system 1 stops operating in step S8, and this information processing ends. 【0063】 4.2. Details of the operating modes This section describes the details of the operation mode M set in step S3 as described in the previous section. Figure 11 is a diagram summarizing the information regarding each operation mode M in this embodiment. As shown in Figure 11, in this embodiment, the standby mode M2 ​​may include, for example, four types of standby modes A to D. Each operation mode M is associated with setting conditions and priority. The processor 83 adopts the operation mode M that satisfies the setting conditions as the operation mode M set in step S3. In this embodiment, if multiple setting conditions are satisfied simultaneously, the processor 83 adopts the one with the highest priority (here, the one with the smallest numerical value assigned as priority) among the multiple operation modes M that satisfy the setting conditions as the operation mode M set in step S3. However, it is not limited to this, and the processor 83 may perform multiple operation modes M simultaneously by superimposing modified patterns of multiple operation modes M that satisfy the setting conditions. 【0064】 As described above, operation mode M1 is an operation mode having a deformation pattern that causes the area of ​​the operating surface 92a in contact with the occupant U's hand UH to be indented to follow the hand UH. Operation mode M1 has a priority of 1, which is higher than all other operation modes M. Therefore, operation mode M1 is executed with exclusive priority over standby mode M2. 【0065】 Standby mode A is an operating mode M that can be set when power is supplied to the HMI device 92, and has the lowest priority among the four types of standby modes A to D. Standby mode A is the operating mode that is executed first, for example, when power is supplied from the vehicle 2. In standby mode A, the processor 83 randomly changes the surface shape of the operating surface 92a. For example, the processor 83 randomly selects a solenoid 12 that retracts the push bar 13 from among several solenoids 12 and drives the solenoid 12 to deform the operating surface 92a in a random pattern. At this time, the processor 83 randomly updates the pattern at a predetermined constant update frequency. Hereinafter, for the sake of explanation, the update frequency of the deformation pattern of the operating surface 92a in standby mode A will be called the reference frequency. The random pattern can be determined regardless of the control mode of equipment such as the car audio 6. In relation to standby mode C, which will be described later, standby mode A is executed when the vehicle 2 is stopped. In other words, in standby mode A, which is an example of the first standby mode, the processor 83 can change the surface shape of the operating surface 92a in a random first pattern independent of the operating mode of the equipment when the vehicle 2 is stopped. With such a configuration, the visual spatial effect inside the vehicle 2 when the vehicle 2 is stopped can be enhanced. Standby mode A, which is an example of the first standby mode, is an example of a mode in which the surface shape of the operating surface 92a is changed regardless of the behavior of the occupant U. 【0066】 Standby mode B is an operating mode that is executed when a biological signal D1 is acquired from the biosensor 5. The processor 83 executes the operation in standby mode B, for example, when the biological signal D1 is not a value that indicates an unacquirable state (e.g., Null). In standby mode B, the processor 83 updates the uneven pattern randomly, similar to standby mode A. The update frequency at this time is changed from a reference frequency based on the biological signal D1. For example, the processor 83 changes the update frequency in standby mode B to a frequency corresponding to a first characteristic frequency (e.g., the specific frequency described above). For example, if the biological signal D1 is respiration, the processor 83 sets the update frequency to a specific frequency that characterizes the respiration period, for example. If the biological signal D1 is pulsation, the processor 83 sets the update frequency to a value of 1 / 4 to 1 / 2 times the specific frequency that characterizes the pulsation period, for example. As a result, the pattern on the operating surface 92a changes in a cycle that regulates the breathing of the passenger U, thereby promoting psychological relaxation of the passenger U and encouraging safer driving. In other words, in standby mode B, which is an example of the fourth standby mode, the processor 83 can deform the shape of the operating surface 92a in a fourth pattern that changes according to the detection result of the biosignal D1 by the biosensor 5. With such a configuration, for example, changes in the biosignals of the passenger U, such as the driver or passenger in the vehicle 2, can be visually observed by the change in the deformation pattern of the operating surface 92a, thereby promoting more comfortable driving for the passenger U through an objective understanding of the passenger U's state. Standby mode B is executed in priority over standby mode A. To put it another way, the fourth pattern is defined so that the shape of the operating surface 92a changes at a timing synchronized with the cycle that characterizes breathing or pulsation. With this configuration, the internal rhythm of the passenger U, expressed by breathing and pulsation, can be visually represented as a deformation of the shape of the operating surface 92a, thereby encouraging the passenger U to objectively observe their own internal rhythm and providing a more comfortable space within the vehicle 2. Standby mode B is an example of a mode in which the surface shape of the operating surface 92a is changed in response to behavior other than operation on the operating surface 92a (in this case, physiological behavior).Furthermore, if the behavior is limited to conscious operation of equipment such as vehicle 2, standby mode B can be considered as an example of a mode that changes the surface shape of the operating surface 92a regardless of the behavior of the occupant U. 【0067】 Standby mode C is an operating mode that is executed when the calculated complexity parameter is greater than or equal to a predetermined threshold, and is executed in priority over standby modes A and B. In standby mode C, the processor 83 randomly updates the surface shape of the operating surface 92a at an update frequency corresponding to the complexity parameter. For example, the processor 83 drives the solenoid 12 such that the update frequency increases as the complexity parameter increases. In this case, the update frequency in standby mode C is greater than the update frequency in standby mode B. Specifically, for example, the update frequency in standby mode C is 1.5, 1.6, 1.7, 1.8, 1.9, or 2 times the update frequency in standby mode B, and may be within the range of any two of the values ​​exemplified here. Thus, in standby mode C, which is an example of a third standby mode, the processor 83 can deform the shape of the operating surface 92a in a third pattern that changes according to a complexity parameter that indicates the degree of environmental condition change, calculated based on the detection result of environmental condition change by the environment detection unit. With this configuration, for example, it is possible to increase the likelihood of drivers and passengers noticing changes in the surrounding environment during driving, which they may not otherwise notice, by changing the deformation pattern of the shape of the operating surface 92a. Therefore, it is possible to increase the opportunities to encourage safer driving. Standby mode C is an example of a mode in which the surface shape of the operating surface 92a is changed regardless of the behavior of the occupant U. 【0068】 In standby mode D, when it is detected that vehicle 2 is curving, the position of the existing indentation is continuously shifted in parallel according to the direction of the curve. For example, if vehicle 2 is curving to the left, the processor 83 shifts the position of the existing indentation to the left, and if vehicle 2 is curving to the right, it shifts the position of the existing indentation to the right. The processor 83 may change the amount of this parallel shift per unit time according to the sharpness of the curve (in other words, the amount of steering of the steering wheel H), and may increase the amount of this shift per unit time as the amount of steering increases. Furthermore, even when such parallel shifts are being performed, the processor 83 may randomly update the position of the existing indentation at predetermined time intervals and further shift the updated existing indentation. Alternatively, the processor 83 may drive the solenoid 12 to induce an indentation along a certain direction, rather than a random indentation, and shift that indentation in parallel. Figure 12 is a photograph showing a specific example of an HMI device 92 in which the surface shape of the operating surface 92a changes so that an indentation Rc along the vertical direction shifts in the horizontal direction. Thus, in standby mode D as an example of the second standby mode, the processor 83 may deform the surface shape of the operating surface 92a in a second pattern that changes according to the detection result of the driving manner of the vehicle 2 by the processor 83 as a driving detection unit. With this configuration, the driving state of the vehicle 2 can be visually grasped from the change in the shape of the operating surface 92a, thereby encouraging an objective understanding of the driving manner. Note that the driving manner is not limited to the direction of the curve of the vehicle 2 and the amount of steering, but may include any behavior of the vehicle 2 that the occupant U can operate while driving, such as the driving speed of the vehicle 2, acceleration and deceleration, the amount the accelerator pedal is pressed, the amount the brake pedal is pressed, the state of the lights, the operation of the turn signals, and the behavior of the wipers. Standby mode D is an example of a mode in which the surface shape of the operating surface 92a is changed in response to behavior other than operation on the operating surface 92a (for example, operation on the vehicle 2 such as steering wheel operation). 【0069】 Furthermore, the processor 83 may change the display mode of the image in accordance with the deformation of the surface shape based on the operating mode. Figures 13 and 14 show an example of an HMI device 92 that changes the display mode of the image projected onto the operating surface 92a in accordance with changes in the surface shape. As shown in Figures 13 and 14, the processor 83 may modulate the image on the operating surface 92a, such as an image representing a water surface, by generating bright areas of color like ripples starting from where the shape of the operating surface 92a has changed. The processor 83 may also change the projected image IM3 itself that is projected onto the operating surface 92a when the vehicle 2 is stopped and when it is moving. For example, when the vehicle 2 is moving, the processor 83 may display a projected image IM3 with a more noticeable color on the operating surface 92a compared to when the vehicle 2 is stopped. With such a configuration, the driving mode of the vehicle 2 can be more clearly visualized through the HMI device 92. In this case, the processor 83 may, for example, change the display mode (e.g., color) of the image in a certain area on the operating surface 92a according to the height of that area (e.g., the depth of the recess Rc or the height of the protrusion). For example, the processor 83 may change the display mode of the image so that the appearance of the recess Rc on the operating surface 92a is relatively darker than the appearance of the protrusion. With such a configuration, the changes in the surface shape of the operating surface 92a can be presented to the user in a more three-dimensional way. For example, the processor 83 may change the display mode of the image so that the appearance of the recess Rc becomes relatively darker as the depth of the recess Rc increases. The processor 83 may, for example, estimate the depth from the driving mode of the solenoid 12 and change the display mode of the image according to the estimated depth. Conversely, the processor 83 may change the display mode of the image so that the appearance of the protrusion on the operating surface 92a is relatively brighter than the appearance of the recess Rc on the operating surface 92a. 【0070】 [others] The above embodiments may be implemented as appropriate based on, for example, the following embodiments. 【0071】 The method for changing the surface shape of the operating surface 92a is not limited to using the solenoid 12; for example, it may also involve inducing local deformation by applying air locally to the operating surface 92a. 【0072】 The devices controlled by this control system 1 are not limited to the car audio system 6. For example, this control system 1 may also control the interior lights, change the display content of the display unit 7, change the angle of the side mirrors, etc., through operations on the HMI device 92. 【0073】 The control system 1 is not limited to controlling equipment in the vehicle 2, but can be used in any application where an HMI system is applicable. 【0074】 The control system 1 described above can also be used as a means to solve the problem of providing a new human-machine interface that is visually or tactilely user-friendly. 【0075】 The information processing device 8 may be on-premise or in a cloud-based configuration. In the case of a cloud-based information processing device 8, for example, it may provide the above-mentioned functions and processing in the form of SaaS (Software as a Service) or cloud computing. 【0076】 In the above embodiment, the information processing device 8 performed various storage and control functions, but instead of the information processing device 8, multiple external devices may be used. That is, various information and programs may be stored in a distributed manner across multiple external devices using blockchain technology or the like. 【0077】 The above embodiment is not limited to the control system 1, but may also be a control method. That is, the control method includes the following steps: In the detection step, the contact state of the occupant U with the operating surface 92a that the occupant U can contact to operate the equipment is detected. In the shape deformation step, the surface shape of the operating surface 92a is changed using a shape changing unit that changes the surface shape of the operating surface 92a in a predetermined operating mode M, according to the detection result of the contact state in the detection step. The operating modes include a mode in which the surface shape is changed according to the behavior of the occupant U, and a mode in which the surface shape is changed regardless of the behavior of the occupant U. 【0078】 The control system 1 described above may be provided in any of the following embodiments. 【0079】 (1) A control system comprising at least one control unit, wherein the control unit is configured to change the surface shape of the operating surface using a shape changing unit that changes the surface shape of the operating surface in a predetermined operating mode, in accordance with the result of detecting the contact state by a contact detection unit that detects the contact state of the user with respect to an operating surface that the user can touch, and wherein the operating mode includes a mode in which the surface shape is changed according to the user's behavior and a mode in which the surface shape is changed regardless of the user's behavior. 【0080】 With this configuration, when a user controls the device, the shape of the operating surface changes in response to the user's actions, such as contact with the operating surface, allowing the user's actions to be objectively observed. On the other hand, the existence of a mode that changes the surface shape regardless of the user's actions allows the user to perceive the autonomous operation of the operating surface, providing a user-friendly experience. Such diverse functions can be expressed in combination through changes in surface shape. 【0081】 (2) The control system described in (1) above, wherein the control system controls the equipment of a vehicle through operation on the operating surface, and the operating mode includes an operating mode that is executed when contact of the user with the operating surface is detected and a standby mode that is executed when contact of the user with the operating surface is not detected, wherein in the operating mode the surface shape is changed to follow the contact point with the user on the operating surface, and the operating mode of the equipment is changed in conjunction with the change in the surface shape, and in the standby mode the surface shape of the operating surface is deformed independently of the user's operation on the operating surface. 【0082】 With this configuration, when a user controls a device, they can control the device through contact with the operating surface and tactilely confirm the operation as a deformation of the operating surface's shape. This provides the user with a more reliable sense of control compared to simple touch operation. On the other hand, even when no operation is being performed, the deformation of the surface shape visually indicates the position of the operating surface to the user. Therefore, a control system with higher operability can be provided, for example, while driving a vehicle. 【0083】 (3) A control system as described in (2) above, wherein the standby mode includes a first standby mode, and in the first standby mode, the control unit deforms the surface shape of the operating surface in a random first pattern independent of the operating mode of the equipment when the vehicle is stopped. 【0084】 This configuration enhances the visual spatial design effect inside the vehicle when it is stationary. 【0085】 (4) A control system according to (2) or (3) above, further comprising a driving detection unit for detecting the driving manner of the vehicle, wherein the standby mode includes a second standby mode, and in the second standby mode, the control unit deforms the surface shape of the operating surface in a second pattern that changes according to the detection result of the driving manner of the vehicle by the driving detection unit. 【0086】 With this configuration, the vehicle's driving state can be visually grasped from changes in the shape of the control surface, thus encouraging an objective understanding of the driving style. 【0087】 (5) A control system according to any one of (2) to (4) above, further configured to generate a target image by imaging at least a portion of the area surrounding the vehicle, wherein the standby mode further includes a third standby mode, in which the control unit deforms the shape of the operating surface in a third pattern that is modulated according to a complexity parameter, which is calculated based on the target image and indicates the complexity of the time change of the target image. 【0088】 With this configuration, for example, it becomes possible to increase the likelihood of drivers and passengers noticing subtle changes in the surrounding environment while driving, which they might otherwise miss, by changing the deformation pattern of the control surface. Therefore, it can increase the opportunities to encourage safer driving. 【0089】 (6) A control system according to any one of (2) to (5) above, further comprising a biosignal detection unit for detecting the user's biosignals, wherein the standby mode further includes a fourth standby mode, and in the fourth standby mode, the control unit deforms the shape of the operating surface in a fourth pattern that changes according to the result of the biosignal detection unit detecting the biosignals. 【0090】 With this configuration, for example, changes in the biosignals of users such as the driver or passengers in the vehicle can be visually observed through changes in the deformation pattern of the control surface. This allows for a more comfortable driving experience for the user through an objective understanding of their condition. 【0091】 (7) A control system as described in (6) above, wherein the biosignal includes information relating to the user's respiration or pulsation, and the fourth pattern is defined such that the shape of the operating surface changes at a timing synchronized with the period characterizing the respiration or pulsation. 【0092】 With this configuration, the user's internal rhythm, expressed through breathing and pulsation, can be visually represented as a deformation of the shape of the control surface. This encourages the user to objectively observe their own internal rhythm and provides a more comfortable space within the vehicle. 【0093】 (8) A control system according to any one of (2) to (7) above, wherein in the operation mode, when a swipe operation on the operating surface is detected as contact by the user, the control unit deforms the surface shape of the operating surface to follow the swipe operation and changes the operating mode of the device according to the direction of the swipe operation and the amount of movement of the user's contact point. 【0094】 With this configuration, it is possible to more tactilely emphasize to the user that an operation has been performed on the device, even when the vehicle is in motion. 【0095】 (9) A control system according to any one of (2) to (8) above, further comprising the vehicle equipped with the aforementioned equipment. 【0096】 (10) A control system according to any one of (1) to (9) above, further comprising a video display unit, wherein the video display unit is configured to display a predetermined video on the operating surface, and the control unit further changes the display mode of the video in accordance with the deformation of the surface shape based on the operating mode. 【0097】 (11) A control method for controlling equipment in a vehicle, comprising the following steps: a detection step of detecting the contact state of a user with respect to an operating surface that a user operating the equipment can touch; and a shape deformation step of changing the surface shape of the operating surface using a shape changing unit that changes the surface shape of the operating surface in a predetermined operating mode, wherein the operating mode includes a mode in which the surface shape is changed according to the user's behavior and a mode in which the surface shape is changed regardless of the user's behavior. Of course, this is not always the case. 【0098】 Finally, while various embodiments relating to this disclosure have been described, these are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols] 【0099】 1: Control System 2: Vehicles 3: Sheet 4: In-car camera 5: Biosensors 6: Car Audio 7:Display section 8: Information Processing Device 80: Communications bus 81: Communications Department 82: Storage section 83: Processor 84: Input section 9: HMI System 91: Projector 92: HMI devices 92a: Operation surface 921: Cabinet 922: Stretchable sheet 93: Contact detection unit 10: Shape change section 11: Support plate 12: Solenoid 13: Push bar 14: Strut 15: Peripheral wall 16: Cushioning material D1: Biological signals H: Handle IM1: In-vehicle image IM2: Surface image IM3: Projected image Rc: dent U: Passenger UH: Hand s1: Video signal s2: Control signal s3: Output signal

Claims

[Claim 1] A control system, It comprises at least one control unit, The control unit is configured to change the surface shape of the operating surface using a shape changing unit in a predetermined operating mode, according to the results of the contact state detection by the contact detection unit which detects the user's contact state with the operating surface that the user can touch, and here, The control system includes a mode in which the surface shape is changed according to the user's behavior and a mode in which the surface shape is changed regardless of the user's behavior. [Claim 2] In the control system according to claim 1, The control system is for controlling the vehicle's equipment through operation on the aforementioned operating surface. The aforementioned operating modes include an operating mode that is executed when contact of the user with respect to the operating surface is detected, and a standby mode that is executed when contact of the user with respect to the operating surface is not detected. In the aforementioned operating mode, the surface shape is changed to follow the contact points with the user on the operating surface, and the operating mode of the device is changed in conjunction with the change in the surface shape. In the standby mode, the control system deforms the surface shape of the operating surface independently of the user's operation on the operating surface. [Claim 3] In the control system according to claim 2, The standby mode includes a first standby mode, In the first standby mode, the control unit is a control system that, when the vehicle is stopped, deforms the surface shape of the operating surface in a random first pattern independent of the operating mode of the device. [Claim 4] In the control system according to claim 2, Furthermore, it includes a driving detection unit that detects the driving manner of the vehicle, The standby mode includes a second standby mode, In the second standby mode, the control unit deforms the surface shape of the operating surface in a second pattern that changes according to the detection result of the driving mode of the vehicle by the driving detection unit. [Claim 5] In the control system according to claim 2, Furthermore, the system is configured to generate a target image by imaging at least a portion of the area surrounding the vehicle. The aforementioned standby mode further includes a third standby mode, In the third standby mode, the control unit deforms the shape of the operating surface in a third pattern modulated according to a complexity parameter that indicates the complexity of the time change of the target image, which is calculated based on the target image. [Claim 6] In the control system according to claim 2, Furthermore, it includes a biosignal detection unit that detects the user's biosignals, The aforementioned standby mode further includes a fourth standby mode, In the fourth standby mode, the control unit is a control system that deforms the shape of the operating surface in a fourth pattern that changes according to the detection result of the biological signal by the biological signal detection unit. [Claim 7] In the control system according to claim 6, The biosignals include information relating to the user's respiration or pulsation. The fourth pattern is a control system in which the shape of the operating surface changes at a timing synchronized with the period characterizing the respiration or pulsation. [Claim 8] In the control system according to claim 2, In the aforementioned operating mode, when a swipe operation on the operating surface is detected as contact by the user, the control unit deforms the surface shape of the operating surface to follow the swipe operation, and changes the operating mode of the device according to the direction of the swipe operation and the amount of movement of the user's contact point. [Claim 9] A control system according to claim 2, Furthermore, a control system including the vehicle equipped with the aforementioned equipment. [Claim 10] In the control system according to any one of claims 1 to 9, Furthermore, it is equipped with a video display unit, The aforementioned video display unit is configured to display a predetermined video on the operating surface, The control unit further includes a control system that changes the display mode of the image in accordance with the deformation of the surface shape based on the operating mode. [Claim 11] A control method for controlling equipment in a vehicle, The following steps are included: In the detection step, the user's contact state with the operating surface that the user can touch when operating the device is detected. In the shape deformation step, the surface shape is changed using a shape changing unit that changes the surface shape of the operating surface in a predetermined operating mode, according to the detection result of the contact state in the detection step, and here, The aforementioned operating mode is A control method including a mode for changing the surface shape in accordance with the user's behavior and a mode for changing the surface shape regardless of the user's behavior.

Images