Device control method, device, medium and device in movable space

Abstract

The disclosed embodiment discloses a device control method, apparatus, medium and device in a movable space, wherein the method includes: in response to obtaining voice information of a user in the movable space, determining the target type of the device to be controlled and the control intention information of the user according to the voice information; determining the historical devices and types of historical devices corresponding to the historical sight of the user within a preset time period; according to the types of historical devices, determining the historical devices of the target type from the historical devices as the devices to be controlled; and controlling the state of the devices to be controlled according to the control intention information. The disclosed embodiment can prevent the user from staring at the device to be controlled for a long time, effectively reduce the user's distraction time, and improve driving safety.

Description

Technical Field

[0001] The present disclosure relates to intelligent control technology, and in particular to a method, device, medium and equipment for controlling equipment in a movable space. Background Art

[0002] In movable spaces such as vehicles, airplanes, and ships, drivers and other users often need to control the state of a device in the space during driving, such as opening or closing a window, which can easily distract the driver and affect driving safety. Summary of the invention

[0003] Embodiments of the present disclosure provide a method, apparatus, medium, and device for controlling equipment in a movable space to avoid driver distraction and improve driving safety.

[0004] A first aspect of an embodiment of the present disclosure provides a method for controlling a device in a movable space, comprising: in response to acquiring voice information of a user in the movable space, determining, according to the voice information, a target type of a device to be controlled and control intention information of the user; determining historical devices corresponding to the user's historical line of sight within a preset time period and the types of the historical devices; according to the types of the historical devices, determining, from the historical devices, a historical device of the target type as the device to be controlled; and controlling the state of the device to be controlled according to the control intention information.

[0005] According to a second aspect of the embodiments of the present disclosure, there is provided a device control apparatus in a movable space, comprising: a first processing module, for responding to voice information of a user in the movable space, and determining, according to the voice information, a target type of a device to be controlled and control intention information of the user; a second processing module, for determining historical devices and types of historical devices corresponding to the historical line of sight of the user within a preset time period; a third processing module, for determining, according to the type of the historical device, a historical device of the target type from the historical devices as the device to be controlled; and a control module, for controlling the state of the device to be controlled according to the control intention information.

[0006] According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the device control method in a movable space described in any of the above embodiments of the present disclosure.

[0007] According to the fourth aspect of the embodiments of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the device control method in a movable space described in any of the above embodiments of the present disclosure.

[0008] A fifth aspect of the embodiments of the present disclosure provides a computer program product. When the instructions in the computer program product are executed by a processor, the device control method in the movable space provided by any of the above embodiments of the present disclosure is executed.

[0009] Based on the device control method, apparatus, medium and equipment in the movable space provided by the above-mentioned embodiments of the present disclosure, when the type of device to be controlled by the user (i.e., the target type) and the user's control intention information are determined according to the user's voice information, the types of historical devices that the user has historically gazed at within a preset time period can be combined to determine the device to be controlled by the user (i.e., the device to be controlled) from these historical devices, and then the state of the device to be controlled can be controlled according to the user's control intention information. As a result, after the user sees that a certain device in the movable space needs to be controlled, he can quickly look at the road ahead and issue voice commands while ensuring driving safety, which can avoid the user staring at the device to be controlled for a long time, effectively reducing the user's distraction time and improving driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is an exemplary application scenario of the device control method in a movable space provided by the present disclosure;

[0011] Figure 2 is a flowchart of a method for controlling a device in a movable space provided by an exemplary embodiment of the present disclosure;

[0012] Figure 3 is a flow chart of a method for controlling a device in a movable space provided by another exemplary embodiment of the present disclosure;

[0013] Figure 4 is a flowchart of a method for controlling a device in a movable space provided by yet another exemplary embodiment of the present disclosure;

[0014] Figure 5 is a schematic diagram of a sight line direction vector provided by an exemplary embodiment of the present disclosure;

[0015] Figure 6 is a flowchart of a method for controlling a device in a movable space provided by another exemplary embodiment of the present disclosure;

[0016] Figure 7is a flowchart of a method for controlling a device in a movable space provided by yet another exemplary embodiment of the present disclosure;

[0017] Figure 8 is a schematic structural diagram of a device control apparatus in a movable space provided by an exemplary embodiment of the present disclosure;

[0018] Fig. 9 is a structural schematic diagram of a device control apparatus in a movable space provided by another exemplary embodiment of the present disclosure;

[0019] Fig.10 is a structural schematic diagram of a device control apparatus in a movable space provided by yet another exemplary embodiment of the present disclosure;

[0020] Fig.11 is a structural schematic diagram of a device control apparatus in a movable space provided by another exemplary embodiment of the present disclosure;

[0021] Fig.12 is a structural schematic diagram of a device control apparatus in a movable space provided by yet another exemplary embodiment of the present disclosure;

[0022] Fig.13 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.

[0024] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0025] SUMMARY OF THE DISCLOSURE

[0026] In the process of realizing the present disclosure, the inventors found that in a movable space such as a vehicle, an airplane, or a ship, a driver or other user often needs to control the state of a device in the space during driving, such as controlling the opening or closing of a window. In the related art, the driver needs to look at the device to be controlled and issue a voice command to control the device, which causes the driver to look at the device for a long time, easily distracting the driver and affecting driving safety.

[0027] Exemplary Overview

[0028] Figure 1 This is an exemplary application scenario of the device control method in a movable space provided by the present disclosure. Figure 1 As shown, the movable space 10 may include a variety of controllable devices. For example, the air conditioner 11, display screen 12, window 13, etc. in the figure. During driving, the driver 14 can use the device control method in the movable space disclosed in the present invention (hereinafter referred to as the method) to control the devices in the movable space 10. The method disclosed in the present invention can be executed in the terminal device in the movable space 10. The movable space 10 can be, for example, the cabin space of a movable device such as a vehicle, an airplane, or a ship. The embodiment of the present invention takes the cabin space of a vehicle (hereinafter referred to as the vehicle space) as an example, and the vehicle terminal executes the method disclosed in the present invention. Specifically, in response to obtaining the voice information of the user (ie, the driver) in the movable space, according to the voice information, the target type of the device to be controlled and the control intention information of the user are determined; the historical device and the type of the historical device corresponding to the historical sight of the user within a preset time length are determined; according to the type of the historical device, the historical device of the target type is determined from the historical device as the device to be controlled; according to the control intention information, the state of the device to be controlled is controlled. As a result, when the driver sees that a device in the movable space needs to be controlled, he can quickly look at the road ahead and issue voice commands while ensuring driving safety. This can prevent the user from staring at the device to be controlled for a long time, effectively reduce the user's distraction time, and improve driving safety.

[0029] Exemplary Methods

[0030] Figure 2 1 is a flow chart of a device control method in a movable space provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as a computing platform on a movable device (such as a vehicle-mounted terminal), such as Figure 2 As shown, the method of the embodiment of the present disclosure may include the following steps:

[0031] Step 210, in response to acquiring the voice information of the user in the movable space, determining the target type of the device to be controlled and the control intention information of the user according to the voice information.

[0032] The movable space is the space where the driver of the movable device is located, and the movable device may include, for example, a vehicle, an airplane, a ship, etc. Taking a vehicle as an example, the movable space is the cabin space of the vehicle. The user may be the driver or other users.

[0033] In some optional embodiments, the user's voice information can be collected by a voice collection device. The voice collection device may include a microphone, a microphone array, etc. The user information collected by the voice collection device is sent to a computing platform of the mobile device, and the computing platform can obtain the user's voice information.

[0034] In some optional embodiments, the target type of the device to be controlled refers to the candidate type of the device to be controlled that is preliminarily determined based on the user's voice information, and the target type may include one or more types. Taking the cockpit of a vehicle as an example, for example, the target type is a window. Since there may be multiple windows, it is temporarily impossible to determine which window the device to be controlled is based on the voice information, and it can only be determined that the target type of the device to be controlled is a window. For another example, the target type includes multiple types with the same or similar control attributes. For example, the voice information is "open a little wider", and the device types that can support the control attribute of opening a little wider include air conditioners, windows, etc. According to the voice information, it is temporarily impossible to determine whether the device to be controlled is a window or an air conditioner, so both the air conditioner and the window are used as target types. The user's control intention information is information indicating the control method of the device to be controlled. For example, the control intention information may include "open", "close", "open a little wider", "turn down", etc.

[0035] In some optional embodiments, the target type of the device to be controlled and the user's control intention information may be determined through voice recognition and semantic analysis.

[0036] Step 220 , determining the historical devices and types of the historical devices that the user has gazed at during the preset time period.

[0037] Among them, the preset duration is the historical duration before the current moment. The preset duration can be set to any duration according to actual needs. For example, the preset duration can be 5 seconds, 10 seconds, 20 seconds, etc., and there is no specific limitation. The historical sight is the sight of the user in the historical time period of the preset duration. The historical device is the device that the user has seen in the historical time period. The type of the historical device is the type of the device that the user has seen in the historical time period. For example, the user sees that the front passenger window is open at a certain moment t1, and then quickly shifts his sight to the front of the road to drive normally. At the moment t2 after t1, a voice "close the window" is issued. Then, according to the voice information, the target type of the device to be controlled (i.e., the window) and the user's control intention information (i.e., close) can be determined. In this case, it is determined that the historical device corresponding to the historical sight of the user within the preset duration may include the historical device that is observed at the moment t1, i.e., the front passenger window, and the type of the historical device is the window. For another example, at time t3, the user sees that the air conditioner is turned on low, and then quickly turns his sight to the road ahead and utters a voice message "turn it up a bit". At time t4 after time t3, it is determined based on the voice information that the target type of the device to be controlled may be a car window or an air conditioner; and then it is determined that the historical devices that the user's historical sight line corresponds to within the preset time period include air conditioners, and the historical device type is air conditioners. The historical devices that are watched within the preset time period may also include other types of devices, such as devices that are watched before time t3, which are specifically determined based on the user's actual sight conditions. The specific historical devices and the way they are represented are not limited. For example, a historical device can be represented by a device identifier, and the device type can be a type number or other methods, as long as the corresponding semantics can be expressed, and the embodiments of the present disclosure are not limited.

[0038] In some optional embodiments, the device that the user's line of sight corresponds to can be determined in real time or at a specified frequency, and the device that the user's line of sight corresponds to, the device type, time information, etc. are stored accordingly, so as to be used in subsequent moments to determine the historical devices and types of historical devices that the user's line of sight corresponds to.

[0039] Step 230 : According to the type of the historical device, determine a historical device of a target type from the historical devices as a device to be controlled.

[0040] The target type can be matched with the type of historical device, and the historical device of the target type can be determined from the historical devices according to the matching result. For example, the historical devices and their corresponding types include device 1-window type, device 2-air conditioner type, and the target type is air conditioner. Then, according to the type of historical device, device 2 of the air conditioner type is determined from the historical devices as the device to be controlled.

[0041] In some optional examples, the user sees that the front passenger window is open at a certain time t1, and then quickly shifts his sight to the road ahead to drive normally. At time t2 after t1, a voice message "close the window" is uttered. Then, the target type of the device to be controlled (i.e., the window) and the user's control intention information (i.e., close) can be determined based on the voice information. In this case, it is determined that the historical device that the user's historical sight corresponds to within the preset time includes the historical device that is observed at time t1, i.e., the front passenger window. The type of the historical device is a window, and the target type is a window. Then, based on the type of the historical device, the historical device of the target type (front passenger window) can be determined from the historical devices as the device to be controlled. The device to be controlled is accurately determined in combination with the user's historical sight and the current voice information.

[0042] In some optional embodiments, if the user is familiar with the specific name of the device to be controlled, the specific device to be controlled and the control intention to the device to be controlled can be directly indicated through voice information. For example, if the passenger window is closed, the device to be controlled and the control intention information can be directly determined based on the voice information. Steps 220 and 230 can be skipped. The disclosed embodiments do not exclude the function of directly controlling the device by voice.

[0043] Step 240: Control the state of the device to be controlled according to the control intention information.

[0044] After the device to be controlled is determined, the state of the device to be controlled can be controlled according to the user's control intention information. For example, if the control intention information is "close" and the device to be controlled is the front passenger window, a command to close the front passenger window is generated according to the control intention information, and the generated command is sent to the controller of the front passenger window to control the front passenger window to close.

[0045] The device control method in the movable space provided by the present embodiment can determine the type of device to be controlled by the user (i.e., the target type) and the user's control intention information based on the user's voice information, and can then combine the types of historical devices that the user has historically gazed at within a preset period of time to determine the device to be controlled by the user (i.e., the device to be controlled) from these historical devices. As a result, after the user sees that a certain device in the movable space needs to be controlled, the user can quickly shift his or her gaze to the road ahead and issue voice commands while ensuring driving safety. This can prevent the user from staring at the device to be controlled for a long time, effectively reduce the user's distraction time, and improve driving safety.

[0046] Figure 3 It is a flowchart of a method for controlling a device in a movable space provided by another exemplary embodiment of the present disclosure.

[0047] In some optional embodiments, in the above Figure 2 Based on the embodiment shown, Figure 3As shown, in response to acquiring the voice information of the user in the movable space, step 210 determines the target type of the device to be controlled and the control intention information of the user according to the voice information, which may include:

[0048] Step 2110, in response to acquiring the voice information of the user in the movable space, performing voice recognition processing on the voice information to obtain text information corresponding to the voice information.

[0049] In some optional embodiments, the speech information may be recognized and processed by a pre-configured, pre-trained speech recognition model to obtain text information corresponding to the speech information.

[0050] In some optional embodiments, the speech recognition model may adopt any model in the relevant technology that can convert speech into text. For example, the speech recognition model may be a model based on a convolutional neural network, a model based on a recurrent neural network, a model based on a deep neural network, etc., which is not limited in the embodiments of the present disclosure.

[0051] Step 2120: perform semantic parsing on the text information to obtain the target type and control intent information of the device to be controlled.

[0052] In some optional embodiments, the text information can be semantically parsed based on preconfigured parsing rules to obtain the target type and control intention information of the device to be controlled. The parsing rules can be set according to the types of various devices in the movable space, control attributes and other related information. The parsing rules may include a first mapping rule between a first preset keyword and the type of the device, and a second mapping rule between a second preset keyword and the control intention. The first preset keyword may include a set of at least one keyword, for example, the first preset keyword may include keywords related to the device name (such as car windows, air conditioners, music, display screens, etc.), keywords related to device control attributes (such as open, close, increase, decrease, etc.). The second preset keyword may include a set of at least one keyword, and the second preset keyword may include keywords related to device control attributes, such as open, close, increase, decrease, etc. The second preset keyword and the first preset keyword may have some of the same keywords. The first mapping rule and the second mapping rule may be set according to the specific circumstances of the controllable device. For example, the first mapping rule may include: first determine whether the text information contains keywords related to the device name. If so, the target type can be obtained based on the device name. If no name-related keywords exist, then determine whether the text information contains keywords related to control attributes. If there are keywords related to control attributes, determine all possible target types from each device based on the keywords related to control attributes and the pre-configured control attributes supported by each device.

[0053] In some optional embodiments, a first preset keyword related to the type can be extracted from the text information, and the first preset keyword can be mapped to the target type of the device to be controlled according to a first mapping rule. A second preset keyword related to the control intention can be extracted from the text information, and the second preset keyword can be mapped to the user's control intention information according to a second mapping rule.

[0054] The embodiments of the present disclosure, by performing semantic analysis on the text information obtained by voice recognition, can parse the text information into possible types of the user's device to be controlled and control intention information, achieve preliminary screening of the types of devices to be controlled, provide effective candidate type references for the determination of the devices to be controlled, and further combine the historical devices that the user's historical line of sight has gazed at and the possible types preliminarily determined by voice to accurately determine the devices to be controlled, thereby ensuring the accuracy of the devices to be controlled.

[0055] Figure 4 It is a flowchart of a method for controlling a device in a movable space provided by yet another exemplary embodiment of the present disclosure.

[0056] In some optional embodiments, based on any of the above embodiments, Figure 4 As shown, step 220 of determining the historical devices and types of historical devices that the user's historical sight lines correspond to within a preset time period may include:

[0057] Step 2210, determining the sight direction vector of the user in the first coordinate system according to the first image including the user's eyes captured by the first camera in the movable space at the first moment.

[0058] The first moment is a moment within a preset time period; the first coordinate system is a coordinate system with the center point of the user's eye as the origin. The user's sight line direction vector in the first coordinate system is a vector along the sight line direction with the center of the user's pupil as the starting point. The first camera is a camera used to monitor the driver.

[0059] In some optional embodiments, the user's sight direction vector in the first coordinate system may be obtained based on a driver monitoring system (Driver Monitor System, DMS for short).

[0060] In some optional embodiments, the gaze direction vector of the user in the first coordinate system may be determined based on a gaze detection algorithm.

[0061] In some optional embodiments, the sight direction vector of the user in the first coordinate system may be determined in real time based on the first image at the first moment. Alternatively, the sight direction vector of the user in the first coordinate system may be determined at the current moment based on the first image at the first moment.

[0062] Step 2220: Determine the historical device and the type of the historical device that the user has gazed at at the first moment according to the gaze direction vector.

[0063] In some optional embodiments, after determining the user's sight direction vector, the device the user is looking at can be determined as the historical device looked at at the first moment based on the spatial vector of the sight direction vector in the movable space and the relative position relationship between each device in the movable space. The type of the historical device is obtained according to the pre-configured correspondence between the device identifier and the type.

[0064] In some optional embodiments, step 220 may be performed in real time at the first moment to determine and store the device and the type of the device that the user is looking at at the first moment. Alternatively, step 220 may be performed at the current moment to obtain the historical devices and the types of the historical devices that the user is looking at at the first moment.

[0065] Step 2230, determining the historical devices and the types of the historical devices that the user's historical sights gazed at within the preset time period according to the historical devices and the types of the historical devices that the user's historical sights gazed at at each moment within the preset time period.

[0066] The historical devices and types of historical devices observed by the historical sight at each moment within the preset time can be stored in a specified format, thereby recording the historical devices and types of historical devices observed by the historical sight within the preset time. The specified format can be, for example, a list, an array, etc., which is not specifically limited.

[0067] In some optional embodiments, a time window with a preset time length can be slid to maintain in real time the historical devices and types of historical devices that were observed by the historical sight for the preset time length before the current moment. For example, if the preset time length is 30 seconds, the historical devices and types observed by the historical sight within the 30 seconds closest to the current moment are always recorded through the time window. Every time data of a new moment is generated, the window is slid, the data of the earliest moment is deleted, and the data of the latest moment is stored.

[0068] In the embodiments of the present disclosure, a first camera is used to monitor the user's line of sight direction vector, so that the device that the user is looking at at each moment can be determined based on the user's line of sight direction vector, and the type of the device can be obtained. When the user issues a voice command, the historical devices and types that the user has looked at at each historical moment within a preset time period can be combined to assist in determining the user's device to be controlled, thereby ensuring the accuracy of the device to be controlled, reducing the user's distraction time, and improving driving safety.

[0069] In some optional embodiments, determining the sight direction vector of the user in the first coordinate system according to the first image including the user's eyes captured by the first camera in the movable space at the first moment in step 2210 may include:

[0070] According to the first image, determine the first position information of the user's eyes in a pre-calibrated second coordinate system, the key point information of the user's eyes in the first image, and the user's line of sight angle information; establish a first coordinate system according to the key point information and the first position information; determine the line of sight direction vector according to the line of sight angle information.

[0071] Among them, the pre-calibrated second coordinate system can be the camera coordinate system (or camera coordinate system) of the first camera, or the pre-calibrated second coordinate system can be a coordinate system with a specified position in the movable space as the origin. The first position information of the user's eye in the second coordinate system may include the position coordinates of the center point of the user's eye in the second coordinate system, and may also include the plane position information of the user's eye plane in the second coordinate system. The key point information of the user's eye in the first image is the coordinate information of the key point of the user's eye detected in the first image. The user's line of sight angle information refers to the pitch angle (pitch), yaw angle (yaw), etc. of the user's line of sight in the second coordinate system.

[0072] In some optional embodiments, the second coordinate system is the camera coordinate system of the first camera. Eye key point detection can be performed based on the first image to obtain key point information such as the center key point, pupil key point and / or iris key point, orbital key point, etc. of the user's eye in the first image. Alternatively, the center key point can be calculated based on the orbital key point. The center key point is converted to the second coordinate system as the first position information of the user's eye. According to the line connecting the designated key point of the user's eye in the first image and the origin of the second coordinate system (i.e., the light emitted from the optical center of the first camera to the pixel corresponding to the designated key point in the image coordinate system), the user's line of sight angle information is determined. For example, a first line is formed between the key point of the edge of the pupil or iris close to the designated eye corner (such as the inner or outer eye corner of the right eye) and the origin of the second coordinate system, and a second line is formed between the designated eye corner and the origin of the second coordinate system, and the angle between the first line and the second line is used as the yaw angle of the user's eye. For another example, the key point of the pupil or iris close to the upper eyelid (or lower eyelid) and the origin of the second coordinate system form a third line, the upper eyelid (or lower eyelid) and the origin of the second coordinate system form a fourth line, and the angle between the third line and the fourth line is used as the pitch angle of the user's eye. The pitch angle and yaw angle of the user's eye are used as the user's sight angle information.

[0073] In some optional embodiments, an eye key point detection algorithm model may be used to perform key point detection on the first image to obtain key point information of the user's eyes in the first image. According to the key point information of the user's eyes in the first image, the first position information and sight angle information of the user's eyes in the second coordinate system are determined. The eye key point detection algorithm model may adopt any model, such as a model based on a convolutional neural network, a model based on a deep neural network, etc., which is not limited in the embodiments of the present disclosure.

[0074] In some optional embodiments, the first position information can be used as the position of the origin of the first coordinate system in the second coordinate system, and the central key point in the key point information can be used as the origin of the first coordinate system. The directions of the three axes of the first coordinate system are respectively the horizontal and vertical directions of the eye plane, and the direction perpendicular to the eye plane from the origin, which is similar to the camera coordinate system.

[0075] In some optional embodiments, after the first coordinate system is established, the sight line direction vector in the first coordinate system may be determined based on the association between the sight line angle information and the first coordinate system. For example, the coordinates of the pupil center in the first coordinate system may be used as the starting point of the sight line direction vector, the sight line direction may be determined based on the sight line angle information, and the sight line direction vector may be determined based on the starting point and the sight line direction.

[0076] In some optional embodiments, Figure 5 is a schematic diagram of a sight line direction vector provided by an exemplary embodiment of the present disclosure. Figure 5 As shown, the first coordinate system is the coordinate system O1X1Y1Z1 of the user's eye A, O1 is the center point of the eye, X1 is the lateral direction of the eye plane, Y1 is the longitudinal direction of the eye plane, and Z1 is the direction perpendicular to the eye plane emitting from O1. The second coordinate system is the camera coordinate system O2X2Y2Z2 of the first camera B, O2 is the optical center of the first camera, X2 is the lateral direction of the camera coordinate system, Y2 is the longitudinal direction of the camera coordinate system, and Z2 is the optical axis direction. The key points P of the user's eye A are the black dots in the eye sockets and the gray dots at the edge of the iris in the figure. The figure shows the key points schematically marked on the user's eye in space. The key points on the image captured by the first camera are similar to the marks in the figure. The sight line direction vector V is a vector along the sight line direction with the center of the pupil of the user's eye A as the starting point.

[0077] The embodiments of the present disclosure, by acquiring a first image containing the user's eyes, are used to determine the position information of the user's eyes in a second coordinate system, the key point information of the user's eyes, and the user's line of sight angle information, thereby providing an effective reference basis for determining the user's line of sight direction vector, thereby effectively determining the line of sight direction vector.

[0078] In some optional embodiments, determining the historical device and the type of the historical device that the user's historical sight line gazed at at the first moment according to the sight line direction vector in step 2220 may include:

[0079] Perform spatial point sampling on the line of sight direction vector to obtain a set of first spatial points in the first coordinate system; convert each first spatial point to the second coordinate system according to the conversion relationship between the first coordinate system and the pre-calibrated second coordinate system to obtain the second spatial point of each first spatial point in the second coordinate system; determine the historical device and the type of historical device that the user's historical line of sight at the first moment is looking at according to the device information corresponding to each second spatial point and the device in the pre-configured second coordinate system.

[0080] Among them, the conversion relationship between the first coordinate system and the second coordinate system can be determined based on the position information of the origin of the first coordinate system in the second coordinate system and the plane position information of the XY plane of the first coordinate system in the second coordinate system. The plane position information represents the posture of the first coordinate system in the second coordinate system. Combined with the position and posture of the origin of the first coordinate system in the second coordinate system, the posture (i.e., external parameter) of the first coordinate system relative to the second coordinate system is obtained. The external parameter represents the conversion relationship between the first coordinate system and the second coordinate system. The conversion relationship may include a translation vector and a rotation matrix. The device information corresponding to the device in the second coordinate system may include the device identification, type information, location information and size information of the device in the second coordinate system (or area information occupied by the device) of each device in the space.

[0081] In some optional embodiments, spatial points may be selected along the sight direction vector at a specified distance interval to obtain a set of first spatial points (eg Figure 5 pi), and then according to the conversion relationship between the first coordinate system and the second coordinate system, each first spatial point pi is converted to the second coordinate system to obtain the second spatial point of each first spatial point in the second coordinate system, thereby unifying the spatial points on the sight direction vector and each device in the space into the same coordinate system. Then, according to the second spatial point and each device (such as Figure 5 The relative position relationship between the second spatial point and each device can be included outside the device and inside the device.

[0082] In some optional embodiments, each second space point can be traversed, and if there is at least one second space point in the area of ​​a device, it is determined that the sight direction vector corresponds to the device. Based on this, the historical device that the user's historical sight line gazed at at the first moment can be determined. Then, the type of the historical device can be determined from the pre-configured device information.

[0083] The embodiments of the present disclosure perform spatial point sampling on the gaze direction vector and convert the sampled spatial points into the same coordinate system as the device, so that the device the user is looking at can be effectively determined based on the positional relationship between the spatial points and the device, thereby providing an accurate and effective historical gaze record for determining the device to be controlled.

[0084] In some optional embodiments, the historical devices that the user has gazed at at the first moment can be determined based on the intersection of the extension of the gaze direction vector and the area of ​​each device. For example, the gaze direction vector can be converted to the second coordinate system, the straight line corresponding to the gaze direction vector in the second coordinate system can be determined, and based on the device information of each device in the second coordinate system, multiple planes of each device in the second coordinate system can be determined, and the intersection of the straight line where the gaze direction vector is located and the planes of each device can be determined based on the intersection of the straight line and the plane. If it intersects with a certain device, it is determined that the gaze is gazed at the device. Based on this, the historical devices that the user has gazed at at the first moment can be determined.

[0085] In some optional embodiments, determining the historical device and the type of the historical device that the user's historical sight line gazed at at the first moment according to each second spatial point and the device information corresponding to the device in the preconfigured second coordinate system may include:

[0086] Based on the device information, determine the area information and type corresponding to each device in the second coordinate system; match each second spatial point with the area information corresponding to each device to obtain a matching result; and determine the historical devices and types of the historical devices that the user's historical line of sight was looking at at the first moment based on the matching result and the type corresponding to each device.

[0087] The device information may include the position, size, type, etc. of each device in the second coordinate system. Therefore, the region information and type corresponding to each device in the second coordinate system may be determined based on the device information. The matching result of any second spatial point and the region information of any device includes two situations: matching or not matching.

[0088] In some optional embodiments, for any second spatial point, the position information of the second spatial point can be matched with the area information of each device to determine whether the second spatial point is located in the area of ​​the device. If the second spatial point is located in the area of ​​the device, the matching result between the second spatial point and the device is matched; if the second spatial point is located outside the area of ​​the device, the matching result between the second spatial point and the device is not matched. If at least one second spatial point is located in the area of ​​the device, it indicates that the device is the device that the user is looking at.

[0089] The embodiments of the present disclosure can effectively determine the device that the gaze direction vector is looking at by matching the area information of each device in the second coordinate system with the second spatial point on the gaze direction vector, thereby providing an effective historical gaze device reference for determining the device to be controlled, and combining with the current voice command to achieve accurate control of the device to be controlled.

[0090] Figure 6 It is a flowchart of a method for controlling a device in a movable space provided by yet another exemplary embodiment of the present disclosure.

[0091] In some optional embodiments, based on any of the above embodiments, Figure 6 As shown, step 230 of determining a historical device of a target type from the historical devices as the device to be controlled according to the type of the historical device may include:

[0092] Step 2310: According to the type of the historical device, determine a historical device of the target type from the historical devices as a candidate device.

[0093] Among them, the target type can be matched with the type of historical device, and the historical device that matches (or is consistent with) the target type can be determined from the type of historical device as a candidate device. Since there may be one or more historical devices that match the target type in the historical devices, when there are multiple historical devices that match the target type, it is necessary to further determine which of the multiple historical devices is the device to be controlled. Therefore, the device of the target type in the historical setting is used as a candidate device. For example, if the target type is a car window, and the historical devices include two historical devices of the target types of the co-driver window and the driver window, the historical devices of these two target types are used as candidate devices.

[0094] Step 2320, determine the status information of the candidate device.

[0095] The state of the candidate device refers to the current state of the candidate device, and the state information of the candidate device may include, for example, an open state, a closed state, etc. The specific device state may be set according to the properties of the device.

[0096] In some optional embodiments, the state information of the candidate device can be determined based on the state machine of each device maintained in real time. The state machine of the device is a mathematical model that records the actual state changes of the device in real time during the control process of the device, and will not be described in detail.

[0097] Step 2330: Determine the device to be controlled from the candidate devices according to the state information and control intention information of the candidate devices.

[0098] Among them, the control intention information represents the state that the user intends to convert the state of the device to be controlled to, and the state information of the candidate device represents the current state of the candidate device. In the case of multiple candidate devices, the state information and control intention information of the candidate devices can be further combined to determine the device to be controlled from the candidate devices. For example, the candidate devices include a front passenger window and a main passenger window. The current state information of the front passenger window is closed, and the current state information of the main passenger window is open. If the control intention information is to close the window, it can be determined that the main passenger window is to be closed, and the main passenger window is the device to be controlled. If the control intention information is to open the window, it can be determined that the window to be opened is the front passenger window, and the front passenger window is the device to be controlled, so that the device to be controlled can be accurately determined.

[0099] In the embodiments of the present disclosure, after determining candidate devices from historical devices according to target types, the devices to be controlled are further determined from the candidate devices in combination with control intention information, which can further improve the accuracy of the devices to be controlled, allowing users to effectively control devices in the space through brief visual gaze combined with simple voice commands, thereby improving user experience and driving safety.

[0100] Figure 7 It is a flowchart of a method for controlling a device in a movable space provided by yet another exemplary embodiment of the present disclosure.

[0101] In some optional embodiments, based on any of the above embodiments, Figure 7 As shown, step 230 of determining a historical device of a target type from the historical devices as the device to be controlled according to the type of the historical device may include:

[0102] Step 23a0: According to the type of the historical device, determine candidate historical devices of the target type from the historical devices and the gaze time information corresponding to each candidate historical device.

[0103] Among them, the candidate historical devices (also referred to as candidate devices), the gaze time information corresponding to each candidate historical device can be determined by determining the frequency of the device gazed (or the corresponding time interval), and the number of frames of continuous gaze on the same device. For example, the time interval is T1 milliseconds, that is, the device the user is gazing at is determined once every T1 milliseconds based on the first image collected. If the device being gazed is determined to be the same device for m consecutive frames, the starting gaze time of the device, the gaze duration (m*T1) and other information are recorded. When the user voice controls the device, the candidate device of the target type can be determined from the historical devices, and the gaze time information corresponding to each candidate device can be determined from the historical records.

[0104] In some optional embodiments, in the process of maintaining historical records, devices with too short gaze time (gaze time less than a threshold) may be deleted to reduce historical records and improve subsequent processing efficiency.

[0105] Step 23b0: Determine the device to be controlled from the candidate historical devices according to the gaze time information corresponding to each candidate historical device.

[0106] The candidate historical devices may be sorted according to the length of the watching time, and the candidate historical device with the longest watching time may be used as the device to be controlled.

[0107] In the case where multiple candidate devices of the target type are determined from historical devices, the embodiments of the present disclosure further combine the gaze time of the candidate devices to effectively determine the device to be controlled, thereby ensuring the accuracy of the device to be controlled.

[0108] In some optional embodiments, after the device to be controlled is determined, the to-be-confirmed information corresponding to the device to be controlled can also be output, and the device to be controlled can be controlled after confirmation by the user. The output method of the information to be confirmed can include, for example, voice output, voice + display output, etc. For example, the voice output is "Do you want to close the passenger seat window?", and the user can confirm through voice interaction. After obtaining the user's confirmation instruction, the state of the device to be controlled is controlled according to the control intention information. By interactively confirming with the user, the accuracy of device control can be further improved.

[0109] In the technical solution of this disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the relevant laws and regulations and do not violate public order and good morals. In addition, in the technical solution of this disclosure, the collection and use of user personal information involved are all carried out with the knowledge and authorization of the user, and do not involve illegal collection and illegal use of user personal information.

[0110] The above-mentioned embodiments of the present disclosure may be implemented separately or in any combination without conflict. The specific configuration may be based on actual needs and the present disclosure does not limit this.

[0111] Any device control method in a movable space provided by the embodiments of the present disclosure can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: electronic devices such as terminal devices and servers. Alternatively, any device control method in a movable space provided by the embodiments of the present disclosure can be executed by a processor, such as the processor executing any device control method in a movable space mentioned in the embodiments of the present disclosure by calling the corresponding instructions stored in the memory. This will not be repeated below.

[0112] Exemplary Devices

[0113] Figure 8 is a schematic diagram of the structure of a device control device in a movable space provided by an exemplary embodiment of the present disclosure. The device of this embodiment can be used to implement the corresponding method embodiment of the present disclosure, such as Figure 8 The device shown may include: a first processing module 51 , a second processing module 52 , a third processing module 53 and a control module 54 .

[0114] The first processing module 51 is used to determine the target type of the device to be controlled and the control intention information of the user according to the voice information in response to acquiring the voice information of the user in the movable space.

[0115] The second processing module 52 is used to determine the historical devices and types of the historical devices that the user's historical sight lines correspond to within a preset time period.

[0116] The third processing module 53 is used to determine a historical device of a target type from the historical devices according to the type of the historical device as the device to be controlled.

[0117] The control module 54 is used to control the state of the device to be controlled according to the control intention information.

[0118] Fig. 9 It is a structural schematic diagram of a device control apparatus in a movable space provided by another exemplary embodiment of the present disclosure.

[0119] In some optional embodiments, in the above Figure 8 Based on the illustrated embodiment, the first processing module 51 includes:

[0120] The first processing unit 511 is used for performing voice recognition processing on the voice information in response to acquiring the voice information of the user in the movable space, and obtaining text information corresponding to the voice information.

[0121] The second processing unit 512 is used to perform semantic parsing on the text information to obtain the target type and control intention information of the device to be controlled.

[0122] Fig.10 It is a structural schematic diagram of a device control apparatus in a movable space provided by yet another exemplary embodiment of the present disclosure.

[0123] In some optional embodiments, based on any of the above embodiments, Fig.10 As shown, the second processing module 52 may include: a third processing unit 521 , a fourth processing unit 522 and a fifth processing unit 523 .

[0124] The third processing unit 521 is used to determine the sight direction vector of the user in the first coordinate system according to the first image including the user's eyes acquired by the first camera in the movable space at the first moment.

[0125] Among them, the first moment is a moment within a preset time length; the first coordinate system is a coordinate system with the center point of the user's eye as the origin.

[0126] The fourth processing unit 522 is used to determine the historical device and the type of the historical device that the user gazed at at the first moment according to the gaze direction vector.

[0127] The fifth processing unit 523 is used to determine the historical devices and the types of the historical devices that the user's historical sights gazed at within the preset time period according to the historical devices and the types of the historical devices that the user's historical sights gazed at at each moment within the preset time period.

[0128] In some optional embodiments, the third processing unit 521 is specifically configured to:

[0129] According to the first image, determine the first position information of the user's eyes in a pre-calibrated second coordinate system, the key point information of the user's eyes in the first image, and the user's line of sight angle information; establish a first coordinate system according to the key point information and the first position information; determine the line of sight direction vector according to the line of sight angle information.

[0130] In some optional embodiments, the fourth processing unit 522 is specifically configured to:

[0131] Perform spatial point sampling on the line of sight direction vector to obtain a set of first spatial points in the first coordinate system; convert each first spatial point to the second coordinate system according to the conversion relationship between the first coordinate system and the pre-calibrated second coordinate system to obtain the second spatial point of each first spatial point in the second coordinate system; determine the historical device and the type of historical device that the user's historical line of sight at the first moment is looking at according to the device information corresponding to each second spatial point and the device in the pre-configured second coordinate system.

[0132] In some optional embodiments, the fourth processing unit 522 is specifically configured to:

[0133] Based on the device information, determine the area information and type corresponding to each device in the second coordinate system; match each second spatial point with the area information corresponding to each device to obtain a matching result; and determine the historical devices and types of the historical devices that the user's historical line of sight was looking at at the first moment based on the matching result and the type corresponding to each device.

[0134] Fig.11 It is a structural schematic diagram of a device control apparatus in a movable space provided by yet another exemplary embodiment of the present disclosure.

[0135] In some optional embodiments, based on any of the above embodiments, Fig.11 As shown, the third processing module 53 may include: a first determining unit 531 , a second determining unit 532 , and a third determining unit 533 .

[0136] The first determining unit 531 is configured to determine, according to the type of the historical device, a historical device of a target type from the historical devices as a candidate device.

[0137] The second determining unit 532 is configured to determine the status information of the candidate device.

[0138] The third determining unit 533 is configured to determine the device to be controlled from the candidate devices according to the state information and the control intention information of the candidate devices.

[0139] Fig.12 It is a structural schematic diagram of a device control apparatus in a movable space provided by yet another exemplary embodiment of the present disclosure.

[0140] In some optional embodiments, based on any of the above embodiments, Fig.12 As shown, the third processing module 53 may include: a fourth determining unit 53a and a fifth determining unit 53b.

[0141] The fourth determining unit 53a is used to determine candidate historical devices of the target type and gaze time information corresponding to each candidate historical device from the historical devices according to the type of the historical device.

[0142] The fifth determining unit 53b is configured to determine the device to be controlled from the candidate historical devices according to the gaze time information corresponding to each candidate historical device.

[0143] The beneficial technical effects corresponding to the exemplary embodiment of the present device can be found in the corresponding beneficial technical effects of the above exemplary method section, which will not be repeated here.

[0144] Exemplary Electronic Devices

[0145] Fig.13 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure, including at least one processor 91 and a memory 92.

[0146] The processor 91 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 90 to perform desired functions.

[0147] The memory 92 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 91 may execute one or more computer program instructions to implement the methods and / or other desired functions of the various embodiments of the present disclosure described above.

[0148] In one example, the electronic device 90 may further include: an input device 93 and an output device 94 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0149] The input device 93 may also include, for example, a touch screen, a microphone, various sensors, and the like. Sensors may include, for example, image sensors (such as cameras, cameras, etc.), laser radars, millimeter-wave radars, ultrasonic radars, positioning sensors, pressure sensors, air quality sensors, temperature sensors, and the like. Image sensors, laser radars, millimeter-wave radars, ultrasonic radars, and the like may be used to sense the surrounding environment, that is, to detect dynamic and static objects in the surrounding environment. Dynamic and static objects may include, for example, static objects such as lane lines, curbs, arrows, signs, trees, and buildings, as well as dynamic objects such as surrounding vehicles, pedestrians, and cyclists. Positioning sensors are used to realize the positioning of movable devices (such as self-vehicles, robots, etc.) where electronic devices are located. Positioning sensors may include, for example, inertial measurement units (Inertial Measurement Unit, referred to as: IMU), global positioning systems (Global Positioning System, referred to as: GPS), and the like. Pressure sensors may be used to detect seat pressure. Temperature sensors may be used to detect the temperature in the vehicle cabin. Air quality sensors may be used to detect the air quality in the vehicle cabin.

[0150] The output device 94 can output various information to the outside, and may include, for example, a display, a speaker, a communication network and a remote output device connected thereto, and the like.

[0151] Of course, to simplify, Fig.13 Only some of the components related to the present disclosure in the electronic device 90 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 90 may also include any other appropriate components.

[0152] Exemplary computer program products and computer-readable storage media

[0153] In addition to the above methods and devices, embodiments of the present disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the methods of various embodiments of the present disclosure described in the above “Exemplary Methods” section.

[0154] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0155] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps in the method of various embodiments of the present disclosure described in the above “Exemplary Method” section.

[0156] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium is, for example, but not limited to, a system, device or device including electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0157] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and cannot be considered as necessary for each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, rather than limitation, and the above details do not limit the present disclosure to being implemented by adopting the above specific details.

[0158] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present disclosure claims and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A device control method in a movable space, comprising: In response to acquiring voice information of a user in the movable space, determining a target type of a device to be controlled and control intention information of the user according to the voice information; Determine the historical devices that the user's historical sight lines correspond to within a preset time period and the types of the historical devices; According to the type of the historical device, determining a historical device of the target type from the historical devices as the device to be controlled; According to the control intention information, the state of the device to be controlled is controlled.

2. The method according to claim 1, wherein: The step of determining, according to the type of the historical device, a historical device of the target type from the historical devices as the device to be controlled includes: According to the type of the historical device, determine the historical device of the target type from the historical devices as a candidate device; Determining status information of the candidate device; The device to be controlled is determined from the candidate devices according to the state information of the candidate devices and the control intention information.

3. The method according to claim 1, wherein: The step of determining, according to the type of the historical device, a historical device of the target type from the historical devices as the device to be controlled includes: According to the type of the historical device, determining candidate historical devices of the target type and gaze time information corresponding to each candidate historical device from the historical devices; The device to be controlled is determined from the candidate historical devices according to the gaze time information corresponding to each of the candidate historical devices.

4. The method according to claim 1, wherein: The determining of the historical devices that the historical sight lines of the user correspond to within a preset time period and the types of the historical devices includes: Determine, according to a first image including the user's eyes, captured by a first camera in the movable space at a first moment, a sight line direction vector of the user in a first coordinate system; the first moment is a moment within the preset time length; the first coordinate system is a coordinate system with a center point of the user's eyes as an origin; Determine, according to the sight direction vector, a historical device that the user has historically gazed at at the first moment and a type of the historical device; According to the historical devices gazed at by the historical sight at each moment within the preset time period and the types of the historical devices, the historical devices gazed at by the historical sight of the user within the preset time period and the types of the historical devices are determined.

5. The method according to claim 4, wherein: The determining, according to a first image including the user's eyes and captured by a first camera in the movable space at a first moment, a sight line direction vector of the user in a first coordinate system comprises: Determine, according to the first image, first position information of the user's eye in a pre-calibrated second coordinate system, key point information of the user's eye in the first image, and sight angle information of the user; Establishing the first coordinate system according to the key point information and the first position information; The sight line direction vector is determined according to the sight line angle information.

6. The method according to claim 4, wherein: The determining, according to the sight direction vector, a historical device that the user has historically gazed at at the first moment and a type of the historical device includes: Performing spatial point sampling on the sight line direction vector to obtain a set of first spatial points in the first coordinate system; According to the conversion relationship between the first coordinate system and the pre-calibrated second coordinate system, each of the first spatial points is converted to the second coordinate system to obtain a second spatial point of each of the first spatial points in the second coordinate system; According to the device information corresponding to each of the second spatial points and the device in the preconfigured second coordinate system, the historical device that the user has historically gazed at at the first moment and the type of the historical device are determined.

7. The method according to claim 6, wherein: The determining, according to each of the second spatial points and the device information corresponding to the device in the preconfigured second coordinate system, the historical device that the user has historically gazed at at the first moment and the type of the historical device includes: Based on the device information, determine the area information and type corresponding to each device in the second coordinate system; Matching each of the second spatial points with the area information corresponding to each device to obtain a matching result; According to the matching result and the types corresponding to the devices, the historical devices that the user has historically gazed at at the first moment and the types of the historical devices are determined.

8. The method according to any one of claims 1 to 7, wherein: The step of determining the target type of the device to be controlled and the control intention information of the user according to the voice information includes: Performing voice recognition processing on the voice information to obtain text information corresponding to the voice information; The text information is semantically parsed to obtain the target type of the device to be controlled and the control intention information.

9. A device control apparatus in a movable space, comprising: A first processing module, configured to, in response to acquiring voice information of a user in the movable space, determine a target type of a device to be controlled and control intention information of the user according to the voice information; The second processing module is used to determine the historical devices that the user's historical sight lines correspond to within a preset time period and the types of the historical devices; A third processing module, configured to determine, according to the type of the historical device, a historical device of the target type from the historical devices as the device to be controlled; A control module is used to control the state of the device to be controlled according to the control intention information.

10. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the method according to any one of claims 1 to 8.

11. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1-8.

Images