Interaction control method and device

Abstract

The invention provides an interaction control method and device which are applied to the technical field of man-machine interaction, and the method comprises the steps: collecting various forms of action parameters from interaction actions of a user through different collection modes; and fusing the action parameters in various forms, and determining the intention confidence of the interactive action. And under the condition that the intention confidence is greater than a confidence threshold, determining a target control instruction for controlling the target vehicle according to the interaction action. The problems of poor user experience and low reliability in a traditional man-machine interaction system can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, and in particular to an interaction control method and device. BACKGROUND

[0002] With the rapid development of intelligent cockpit technology, gesture interaction has become an important research direction of vehicle human-computer interaction due to its naturalness and convenience. At present, gesture interaction technology mainly includes two technical paths: biological perception and visual recognition.

[0003] In the field of biological perception technology, the current mainstream solution mainly uses wearable surface electromyography sensors to collect the muscle electrical signals of the user to achieve accurate recognition of gesture actions. However, such sensors usually need to be worn on the user's forearm or hand, which not only reduces the user's operation convenience, but also may cause skin irritation and other discomfort reactions.

[0004] In terms of visual recognition, most current vehicle systems mainly use camera-based visual recognition technology, which only relies on hand geometry features and motion trajectories to determine the user's operation intention. However, during vehicle driving, the jolt of the vehicle can easily cause non-autonomous displacement of the user's hand, and such motion is often misjudged by the system as an effective operation instruction, resulting in low reliability of gesture interaction.

[0005] As can be seen, both wearable solutions and visual recognition solutions have certain limitations. SUMMARY

[0006] The purpose of the present application is to provide an interaction control method and device to solve the problems of poor user experience and low reliability in traditional human-computer interaction systems.

[0007] In a first aspect, an interaction control method is provided, which includes: collecting multiple forms of action parameters from the user's interaction action through different collection methods. Fusing the multiple forms of action parameters to determine the intention confidence of the interaction action. In the case where the intention confidence is greater than a confidence threshold, determining a target control instruction for controlling a target vehicle according to the interaction action.

[0008] The interaction control method provided by the embodiments of the present application obtains various forms of action parameters from user interaction actions through different collection methods, and performs fusion analysis on the action parameters, so as to more accurately determine the intention confidence of the user interaction action. When the intention confidence is higher than a set confidence threshold, a target control instruction is generated according to the interaction action. This process accurately judges the user intention, effectively avoids the error operation caused by the intention misjudgment, significantly improves the reliability and safety of the interaction control, provides the user with a more stable and reliable interaction control experience, ensures the accuracy and stability of the vehicle control, and further enhances the trust and satisfaction of the user to the interaction system.

[0009] In a possible implementation, the intention confidence of the interaction action is determined by fusing the various forms of action parameters, including: extracting corresponding action features from each form of action parameter. The intention confidence of the interaction action is determined by fusing each form of action feature.

[0010] In a possible implementation, when the action parameter includes the hand reflection signal of the user, the corresponding action feature is extracted from each form of action parameter, including: performing Fourier transform on the reflection signal to calculate the muscle group spectrum energy of a preset muscle group on the hand of the user. The muscle micro-vibration feature corresponding to the reflection signal is extracted from the muscle group spectrum energy.

[0011] In this possible implementation, the hand reflection signal of the user is processed by Fourier transform, and the muscle micro-vibration feature is extracted, so that the subtle action information of the hand of the user can be accurately captured, and accurate muscle micro-vibration features are provided for subsequent intention recognition and control instruction generation.

[0012] In a possible implementation, the interaction control method provided by the embodiments of the present application further includes: collecting a pulse excitation generated by the user by performing an instruction action and a response signal of the vehicle to the user. Based on the pulse excitation and the response signal, a vibration transfer function of the vehicle and the user is identified. According to the vibration transfer function, an inverse filter is constructed. The reflection signal is preprocessed by using the inverse filter.

[0013] In a possible implementation, when the action parameter includes the hand blood flow signal of the user, the corresponding action feature is extracted from each form of action parameter, including: calculating the blood flow acceleration of the hand of the user according to the blood flow signal. The muscle contraction strength of the hand of the user is determined according to the blood flow acceleration.

[0014] In this possible implementation, the blood flow acceleration of the hand of the user is calculated according to the hand blood flow signal of the user, so that the dynamic change of the blood flow can be quantified, and the muscle contraction strength of the hand of the user is determined, and accurate muscle contraction strength is provided for subsequent intention recognition and control instruction generation.

[0015] In a possible implementation, when the action parameter is a key node signal of a key node of the hand of the user, a corresponding action feature is extracted from each form of the action parameter, including: determining a position change of each key node of the hand of the user in consecutive frames according to the key node signal of each key node of the hand of the user; and determining a motion trajectory feature of the hand of the user in a three-dimensional space based on the position changes of all the key nodes of the hand of the user in the consecutive frames.

[0016] This possible implementation can comprehensively understand the motion intention of the hand of the user by comprehensively considering the position changes of all the key nodes, thereby providing more accurate and reliable motion trajectory features for subsequent interactive control.

[0017] In a possible implementation, the interactive control method provided by the embodiment of the application further includes: calculating a joint angular velocity of the hand of the user according to the position changes of each key node of the hand of the user in consecutive frames; and when the joint angular velocity in a preset number of consecutive frames is greater than an angular velocity threshold, removing the key node signals in the preset number of consecutive frames from the action parameter, and generating an abnormality warning to indicate an abnormal interactive action.

[0018] In a possible implementation, each form of the action feature is fused to determine an intention confidence of the interactive action, including: obtaining an environmental parameter of a target vehicle; determining a feature weight corresponding to each form of feature according to the environmental parameter; and determining the intention confidence of the interactive action according to each form of action feature and the corresponding feature weight.

[0019] In a possible implementation, the action parameter includes a reflection signal of the hand of the user, a blood flow signal of the hand of the user, and a key node signal of a key node of the hand of the user. The multiple forms of action parameters are collected from the interactive action of the user, including: transmitting continuous waves to the user to receive the reflection signal of the hand of the user; and / or using a preset light source to obtain the blood flow signal of the hand of the user; and / or collecting a trajectory of each key node of the hand of the user to obtain the key node signal of the key node of the hand of the user.

[0020] This possible implementation can more comprehensively capture the detailed features of the interactive action of the user by collecting multiple forms of action parameters of the interactive action of the user.

[0021] In a second aspect, the embodiment of the application provides an interactive control device, which includes: a collection module, a fusion module, and a determination module.

[0022] The collection module is configured to collect multiple forms of action parameters from the interactive action of the user by different collection manners.

[0023] The fusion module is configured to fuse the multiple forms of action parameters to determine an intention confidence of the interactive action.

[0024] determining, according to the interaction action, a target control instruction for controlling the target vehicle, when the intention confidence is greater than the confidence threshold.

[0025] In a third aspect, an embodiment of the present application provides an interactive control device, which has a function of implementing the interactive control method of the first aspect or any possible implementation manner of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0026] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores instructions, when the instructions are run on a computer, the computer can execute the interactive control method of the first aspect or any possible implementation manner of the first aspect.

[0027] In a fifth aspect, an embodiment of the present application provides a computer program product containing instructions, when the instructions are run on a computer, the computer can execute the interactive control method of the first aspect or any possible implementation manner.

[0028] The technical effects brought by any implementation manner of the second aspect to the fifth aspect can refer to the technical effects brought by the possible implementation manners of the first aspect, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the description of the embodiments or the prior art. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0030] Figure 1 A system architecture diagram of an interactive control system provided by an embodiment of the present application; Figure 2 A flowchart of an interactive control method provided by an embodiment of the present application; Figure 3 A specific example diagram of a key node of a hand provided by an embodiment of the present application; Figure 4 A structural schematic diagram of an interactive control device provided by an embodiment of the present application; Figure 5 Another system architecture diagram of an interactive control system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0031] The technical solutions and advantages of the embodiments of the present application will be more apparent from the following description of the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0032] Therefore, the detailed description of the embodiments of the present application provided below in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by a person of ordinary skill in the art without creative labor based on the embodiments in the present application are within the scope of protection of the present application.

[0033] Currently, in the process of gesture interaction control of a vehicle, a wearable sensor is usually worn on the forearm or hand of a user, which not only reduces the operation convenience of the user, but also may cause skin allergy and other discomfort reactions. Meanwhile, in the process of driving of the vehicle, the bumping of the vehicle is likely to cause non-autonomous displacement of the hand of the user, and such motion is often misjudged as an effective operation instruction by the system, resulting in low reliability of gesture interaction.

[0034] Based on this, the embodiments of the present application provide an interactive control method, which comprises: collecting a plurality of forms of action parameters from an interactive action of a user. Fusing the plurality of forms of action parameters to determine an intention confidence of the interactive action. In the case that the intention confidence is greater than a confidence threshold, determining a target control instruction for controlling a target vehicle according to the interactive action.

[0035] The interactive control method provided by the embodiments of the present application acquires a plurality of forms of action parameters from the interactive action of the user through different collection methods, and analyzes these action parameters to determine the intention confidence of the interactive action of the user more accurately. When the intention confidence is higher than the set confidence threshold, the target control instruction is generated according to the interactive action. This process accurately judges the intention of the user, effectively avoids the error operation caused by misjudgment of intention, significantly improves the reliability and safety of interactive control, provides the user with a more stable and reliable interactive control experience, ensures the accuracy and stability of vehicle control, and further enhances the trust and satisfaction of the user to the interactive system.

[0036] The solutions provided by the embodiments of the present application will be described below in conjunction with specific accompanying drawings.

[0037] In one aspect, the embodiments of the present application provide an interactive control system. As shown in Figure 1As shown, the interaction control system 100 can include an interaction module 101, a sensor 102, a control module 103 and a driving module 104.

[0038] The interaction module 101 is configured to implement the interaction between the user and the interaction control system.

[0039] Specifically, the interaction module 101 is configured to receive the input signal input by the user through gestures, voice or other ways. For example, the interaction action of the user's hand is received.

[0040] The sensor 102 is configured to collect the action parameters of the user at the same time when the interaction module 101 receives the instruction input by the user. For example, the sensor 102 can include a millimeter wave radar, an optical sensor and a vision system.

[0041] For example, the millimeter wave radar can use AWR1843 radar chip, emit 60GHz frequency modulation continuous wave (FMCW) to the user's hand, and receive the reflection signal of the user's hand. The optical sensor can use a 1300nm broadband light source (bandwidth 100nm) to obtain the blood flow signal of the user's hand skin under 2mm depth. The vision system can be used to identify the key nodes of the user's hand, and collect the key node signals of the key nodes of the user's hand.

[0042] The control module 103 is configured to receive the interaction action sent by the interaction module 101 and the action parameters of the user collected by the sensor 102, and determine the intention confidence of the interaction action through the interaction control method provided by the embodiment of the application. In the case where the intention confidence is greater than the confidence threshold, the control module 103 determines the target control instruction of the control target vehicle according to the interaction action.

[0043] The driving module 104 is configured to receive the target control instruction sent by the control module 103, and drive the corresponding device or component to perform operation according to the target control instruction. For example, the driving module 104 can drive the motor, the steering engine and other mechanical components to realize the motion control of the vehicle. The driving module 104 can drive the display screen to display information such as text, image or video according to the target control instruction. The driving module 104 can drive the loudspeaker to emit sound or adjust the volume according to the target control instruction.

[0044] The interaction module 101 can also be used to display the intention confidence of the control module 103 to the user, so as to facilitate the user to adjust or re-interact in time. The interaction module 101 is also used to display the operation execution result to the user after the driving module 104 drives the corresponding device or component to perform operation according to the target control instruction.

[0045] It should be noted that the above Figure 1The schematic interactive control system 100 is only an example of the application scenario of the application scheme, and is not a limitation on the application scenario of the application scheme.

[0046] In one aspect, the application embodiment provides an interactive control method. The method can be executed by Figure 1 The schematic interactive control system 100. As shown in the figure, the method can include the following steps. Figure 2

[0047] S201, collecting multiple forms of action parameters from the user's interactive action through different collection methods.

[0048] Among them, the action parameters can include the user's hand reflection signal, the user's hand blood flow signal and the key node signal of the user's hand key node.

[0049] Specifically, when detecting that the user initiates an interactive action, multiple forms of action parameters are collected from the user's interactive action through the sensors of the vehicle.

[0050] A possible implementation, through a millimeter wave radar to the user's hand sends continuous millimeter wave signal, and collects the user's hand reflection signal.

[0051] Among them, the frequency and power emission of the millimeter wave signal can be adjusted according to the application scenario and safety standards to ensure that the activity range of the user's hand can be covered while avoiding harm to the human body.

[0052] For example, the AWR1843 radar chip emits a 60GHz frequency-modulated continuous wave (FMCW) to the user's hand and receives the reflection signal reflected from the user's hand. The reflection signal can be represented as:

[0053] Among them, is the reflection signal, K is the number of scattering points, Ak, fk, and ϕk are the amplitude, Doppler frequency and phase of each scattering point, respectively.

[0054] Another possible implementation, through an optical sensor to the user's hand emits a light source of a predetermined wavelength, and detects the reflected or transmitted light signal to obtain the blood flow information of the hand.

[0055] Among them, the wavelength of the light source can be selected according to actual needs. For example, infrared light or visible light can be selected. Among them, the infrared light can penetrate the deep tissue of the skin and be used to detect the subcutaneous blood flow. Visible light can be used to detect the blood flow changes on the surface of the skin.

[0056] ​For example, a 1300 nm broadband light source (bandwidth 100 nm) is used to obtain blood flow signals at a depth of 2 mm under the skin. The blood flow signal can be represented as:

[0057] wherein, is the blood flow signal, is the reference arm light intensity, and is the sample arm light intensity, Δf is the Doppler shift, and Δϕ is the initial phase difference.

[0058] In another possible implementation, an image or video of the user's hand is captured by a camera or other imaging device, key nodes of the hand are identified and tracked, and their motion trajectories are recorded.

[0059] As shown in Figure 3 , a specific example of a key node of the hand is shown. A high-resolution camera or depth sensor can be used to capture an image of the user's hand. The key nodes of the user's hand are identified through image processing algorithms. The motion trajectories of each key node, including the position, velocity, and acceleration of the key nodes, are tracked and recorded. Figure 3 Further, after collecting various forms of action parameters from the user's interactive actions, the various forms of action parameters are synchronized by establishing a unified space-time coordinate system.

[0060] Specifically, first, a unified space-time coordinate system is established, and the user's hand reflection signal (100 Hz), the user's hand blood flow information (10 Hz), and the key node signal of the user's hand key node (30 fps) are interpolated and synchronized, aligning the user's hand reflection signal, the user's hand blood flow information, and the key node signal of the user's hand key node. The interpolation synchronization process can be achieved through the following formula.

[0061]

[0062] wherein,

[0063] is the aligned time, is the radar measurement time. S202, the various forms of action parameters are fused to determine the intent confidence of the interactive action.

[0064] Specifically, the corresponding action features are extracted from each form of action parameter. Each form of action feature is fused to determine the intent confidence of the interactive action.

[0065]

[0066] ​In one possible implementation, the action parameter includes a hand reflection signal of the user, and a Fourier transform is performed on the reflection signal to calculate muscle group spectral energy of a preset muscle group on the hand of the user. Muscle micro-vibration features corresponding to the reflection signal are extracted from the muscle group spectral energy.

[0067] In an example, the original signal is segmented into multiple time windows, and a Fourier transform is performed on the signal in each time window. The selection of the time window can be optimized according to the characteristics of the signal to balance the time resolution and the frequency resolution. The frequency spectrum in each time window is calculated to obtain muscle group spectral energy of a preset muscle group on the hand of the user. Muscle micro-vibration features corresponding to the reflection signal are extracted from the muscle group spectral energy. The spectral energy can be determined by the following formula.

[0068]

[0069] wherein, is the muscle group spectral energy, n is the time window index, m is the frequency bin number, N is the number of FFT points, and S is the reflection signal.

[0070] Further, before the corresponding muscle micro-vibration features are extracted from the hand reflection signal, the hand reflection signal can be preprocessed to remove noise and interference.

[0071] Specifically, when the user is seated, the user is guided to perform an indication action, and the impulse excitation generated by the user by performing the indication action and the response signal of the vehicle to the user are collected. Based on the impulse excitation and the response signal, a vibration transfer function of the vehicle and the user is identified. According to the vibration transfer function, an inverse filter is constructed. The inverse filter is used to preprocess the reflection signal.

[0072] In an example, the user taps the steering wheel with a finger or a specific tool to generate an impulse excitation. The vehicle collects the response signal of the vehicle to the user through a sensor. Based on the impulse excitation and the response signal, a vibration transfer function of the vehicle and the user is identified. The vibration transfer function can be determined by the following formula.

[0073]

[0074] wherein, is the natural frequency, which can be obtained by a peak detection method. ζ is the damping ratio, which can be calculated by a half-power bandwidth method.

[0075] According to the identified transfer function, an inverse filter is constructed. The inverse filter can be represented as:

[0076] wherein, is the preprocessed reflection signal, H(jω) is the vibration transfer function, and j is the imaginary unit.

[0077] Finally, the inverse filter is constructed to pre-process the reflected signal, and compensate for the vibration distortion.

[0078] Further, when the user is seated, the user can be guided to perform a standard gesture, such as clenching and unclenching the fist 3 times, to establish a personalized baseline signal of the user, which can be used for subsequent signal analysis and comparison.

[0079]

[0080] wherein, is the baseline signal.

[0081] In one possible implementation, when the action parameter includes a hand blood flow signal of the user, a blood flow acceleration of the hand of the user is calculated according to the blood flow signal. A muscle contraction strength of the hand of the user is determined according to the blood flow acceleration.

[0082] Specifically, the blood flow acceleration is calculated by a phase-resolved Doppler algorithm according to the blood flow signal.

[0083]

[0084] wherein, is the blood flow acceleration, is the center wavelength of the light source. n is the refractive index of the tissue, for example, 1.38. T is the adjacent A-scan time interval.

[0085] Further, the muscle contraction strength of the hand of the user is determined according to the blood flow acceleration. The muscle contraction strength can be determined by the following formula.

[0086]

[0087] wherein, is the muscle contraction strength, and k is a constant, for example, k can be 0.03.

[0088] In one possible implementation, when the action parameter is a key node signal of a key node of the hand of the user, a position change of each key node of the hand of the user in consecutive frames is determined according to the key node signal of each key node of the hand of the user. A motion trajectory feature of the hand of the user in the three-dimensional space is determined based on the position changes of all the key nodes of the hand of the user in the consecutive frames.

[0089] Specifically, a deep learning model can be used to extract node coordinates of each key node of the user's hand from the video frames. According to the node coordinates of each key node, the position change of each key node in the continuous frames is calculated using a difference operation. Then, according to the position change of all key nodes of the user's hand in the continuous frames, the motion trajectory feature of the user's hand in the three-dimensional space is determined.

[0090] Further, after determining the position change of all key nodes of the user's hand in the continuous frames, the joint angular velocity of the user's hand can also be calculated according to the position change of each key node of the user's hand in the continuous frames. When the joint angular velocity in a preset number of continuous frames is greater than an angular velocity threshold, the key node signal in the preset number of continuous frames is excluded from the action parameters, and an abnormal warning is generated to indicate an abnormal interaction action.

[0091] Specifically, the joint angular velocity of the user's hand is calculated by establishing a four-bar hand model.

[0092]

[0093] wherein, is the joint angular velocity. and is used to represent the position change of the key node in the continuous frames.

[0094] If the key node signal is detected in three continuous frames > 180° / s, it can be determined that the key node signal is an abnormal signal, and the key node signal is excluded from the action parameters to avoid abnormal data affecting subsequent gesture recognition and analysis. At the same time, an abnormal warning is immediately generated to prompt the user that the current gesture action is invalid or abnormal.

[0095] Further, when it is detected that the user's hand is occluded, a ST-GAN (Spatial-Temporal Generative Adversarial Network) network can be used to generate a reasonable gesture trajectory prediction.

[0096] Specifically, when the hand reflection signal shows an abnormal reflection pattern or the camera cannot capture the complete image of the user's hand, it can be determined that the user's hand is occluded. When the occlusion is detected, the ST-GAN network is started, and a reasonable gesture trajectory prediction is generated using the millimeter wave micro-Doppler spectrum as a constraint.

[0097] wherein, the ST-GAN network can include a generator G and a discriminator D.

[0098] The input data of the ST-GAN network includes an occlusion mask (64x64) and a historical key point sequence (10 frames x 21 points). The occlusion mask is used to represent the area of the hand being occluded. The historical key point sequence is used to represent the historical hand key point data captured before the hand is occluded.

[0099] The generator G processes the input data using 3 layers of spatio-temporal convolution (Conv3D) to extract spatio-temporal features. The historical key point sequence is processed through a gated recurrent unit (GRU) to capture the temporal dependency. Finally, the spatio-temporal features and sequence information are combined to predict the hand gesture trajectory during the occlusion period.

[0100] The discriminator D adopts a dual-stream network structure, including a visual stream CNN and a radar stream STFT-CNN, which process visual information and radar information respectively. The discriminator D processes the generated gesture trajectory through the visual stream to determine the rationality of the gesture trajectory. The discriminator D processes the millimeter wave micro-Doppler spectrum through the radar stream to determine whether the radar features of the generated trajectory are consistent with the real data.

[0101] The performance of the generator and discriminator is further optimized by combining the trajectory L2 loss and the radar feature similarity. The trajectory L2 loss is used to measure the difference between the generated trajectory and the real trajectory. The radar feature similarity is used to measure the difference between the radar features of the generated trajectory and the real trajectory.

[0102]

[0103] wherein, is the loss function of the generator G. Y represents the generated trajectory. Y represents the real trajectory. and are weight parameters. and represent the radar features.

[0104] Further, after extracting the corresponding action features from each form of action parameter, the environmental parameters of the target vehicle are obtained. According to the environmental parameters, the feature weights corresponding to each form of feature are determined. According to each form of action feature and the corresponding feature weight, the intention confidence of the interactive action is determined.

[0105] For example, the current environmental parameters of the target vehicle are obtained. The environmental parameters can include the hand occlusion ratio of the user, the vehicle vibration signal-to-noise ratio, and the light intensity.

[0106] According to the hand occlusion ratio of the user, the weight of the motion trajectory feature of the user's hand in the three-dimensional space is adjusted . For example, the weight is adjusted by the following formula .

[0107]

[0108] Adjust the weight of muscle micro-vibration feature according to the signal-to-noise ratio of vehicle vibration For example, adjust the weight by the following formula .

[0109]

[0110] Adjust the weight of muscle contraction intensity feature according to the intensity of light For example, adjust the weight by the following formula .

[0111]

[0112] After determining the feature weight corresponding to each form of action parameter, the intention confidence of the interactive action is determined according to each form of action feature and the corresponding feature weight. For example, the intention confidence of the interactive action can be determined by the following formula.

[0113]

[0114] Wherein, is used to represent the intention confidence of the interactive action. i represents the category of the action feature. represents the confidence of the i-th action feature.

[0115] This process can adapt to different operating conditions and environmental changes by dynamically adjusting the weight and confidence by using real-time environmental changes and sensor data, and can improve the accuracy and robustness of interactive action recognition.

[0116] Further, the weight distribution model is corrected in reverse according to the determined biomechanical verification result. This process can be implemented by the following formula.

[0117]

[0118] Wherein, is the new value of the weight of the i-th action feature. is the old value of the weight of the i-th action feature. is a flag bit.

[0119] S203, in the case where the intention confidence is greater than the confidence threshold, determining a target control instruction for controlling the target vehicle according to the interactive action.

[0120] Specifically, after determining the intention confidence of the interactive action, first evaluate whether the intention confidence of the recognized interactive action is greater than the preset confidence threshold. The confidence threshold can be a predetermined threshold.

[0121] In the case that the intention confidence of the interaction action is greater than the confidence threshold, the gesture is converted into a specific target control instruction according to a predefined mapping relationship. For example, “single finger right swipe” may correspond to increasing the volume, and “double finger pinch” may correspond to map zooming. The control instruction is then sent to various modules of the vehicle, such as the entertainment system, air conditioner, navigation, etc., through a controller area network (CAN) bus to perform corresponding operations.

[0122] Further, the response time of the user operation is monitored in real time. If the response time exceeds 300 milliseconds, the system automatically triggers a self-check process to determine whether there is a problem in the current vehicle interaction process.

[0123] Still further, after the target control instruction is executed, the system adjusts the weight distribution model parameters through feedback optimization according to the user's subsequent operation (such as gesture correction or voice confirmation) to improve the accuracy and response speed of the system.

[0124] Finally, when the vehicle is detected to be turned off or the user leaves the vehicle, the system gradually reduces the sensor sampling frequency to a standby mode to save vehicle energy. At the same time, the various forms of action parameters collected during the current driving period, the target control instruction, and the control result are saved to facilitate subsequent adjustment of the weight of each form of action parameter in the interaction control method.

[0125] For example, the millimeter wave radar and optical sensor of the vehicle are turned off, and only the vision system is retained to monitor the cockpit state at a low frame rate. After detecting that the user reseats, the vehicle interaction control process is automatically restarted and full-function operation is restored.

[0126] The above mainly introduces the scheme provided by the embodiments of the present application from the perspective of the working principle of the device. It can be understood that the interaction control device includes hardware structures and / or software modules corresponding to the execution of each function in order to implement the above functions. Those skilled in the art should easily realize that, in combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0127] The embodiments of the present application can divide the functional modules of the interactive control device according to the above method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated in one processing module. The integrated module can be realized in the form of hardware or in the form of a software functional module.

[0128] It should be noted that the division of the modules in the embodiments of the present application is illustrative, and is only a logical functional division. In actual implementation, another division manner can be used. In the case of dividing each functional module according to each function, Figure 4 A possible composition schematic diagram of the interactive control device involved in the above and embodiments is shown. As shown in the figure, Figure 4 The interactive control device 400 can include a collection module 401, a fusion module 402 and a determination module 403.

[0129] The collection module 401 is configured to support the interactive control device 400 to perform Figure 2 S201 in the schematic interactive control method.

[0130] The fusion module 402 is configured to support the interactive control device 400 to perform Figure 2 S202 in the schematic interactive control method.

[0131] The determination module 403 is configured to support the interactive control device 400 to perform Figure 2 S203 in the schematic interactive control method.

[0132] In one possible implementation, the interactive control device provided by the embodiments of the present application can also be used to extract the corresponding action features from each form of action parameter. The action features of each form are fused to determine the intention confidence of the interactive action.

[0133] In one possible implementation, when the action parameter includes the hand reflection signal of the user, the interactive control device provided by the embodiments of the present application can also be used to perform Fourier transform on the reflection signal, and calculate the muscle group spectrum energy of the preset muscle group on the hand of the user. The muscle micro-vibration features corresponding to the reflection signal are extracted from the muscle group spectrum energy.

[0134] In one possible implementation, the interactive control device provided by the embodiments of the present application can also be used to collect the pulse excitation generated by the user by performing the indication action and the response signal of the vehicle to the user. Based on the pulse excitation and the response signal, the vibration transfer function of the vehicle and the user is identified. According to the vibration transfer function, an inverse filter is constructed. The inverse filter is used to preprocess the reflection signal.

[0135] In a possible implementation, when the action parameter comprises a hand blood flow signal of the user, the interaction control apparatus provided in the embodiments of the present application can further be used to calculate a blood flow acceleration of the hand of the user according to the blood flow signal. The muscle contraction strength of the hand of the user is determined according to the blood flow acceleration.

[0136] In a possible implementation, when the action parameter is a key node signal of a key node of the hand of the user, the interaction control apparatus provided in the embodiments of the present application can further be used to determine a position change of each key node of the hand of the user in consecutive frames according to the key node signal of each key node of the hand of the user. The motion trajectory feature of the hand of the user in the three-dimensional space is determined based on the position changes of all the key nodes of the hand of the user in the consecutive frames.

[0137] In a possible implementation, the interaction control apparatus provided in the embodiments of the present application can further be used to calculate an angular velocity of a joint of the hand of the user according to the position changes of each key node of the hand of the user in the consecutive frames. When the angular velocity of the joint is greater than an angular velocity threshold in a preset number of consecutive frames, the key node signal in the preset number of consecutive frames is removed from the action parameter, and an abnormal warning is generated to indicate that the interaction action is abnormal.

[0138] In a possible implementation, the interaction control apparatus provided in the embodiments of the present application can further be used to obtain an environmental parameter of the target vehicle. The feature weight corresponding to each form feature is determined according to the environmental parameter. The intention confidence of the interaction action is determined according to each form action feature and the corresponding feature weight.

[0139] In a possible implementation, the action parameter comprises a hand reflection signal of the user, a hand blood flow signal of the user, and a key node signal of a key node of the hand of the user. The interaction control apparatus provided in the embodiments of the present application can further be used to send a continuous wave to the user to receive the hand reflection signal of the user. In addition, the hand blood flow signal of the user can be obtained by using a preset light source. In addition, the trajectory of each key node of the hand of the user can be collected to obtain the key node signal of the key node of the hand of the user.

[0140] It should be noted that all related contents of each step involved in the method embodiments described above can be cited to the function description of the corresponding function module, and will not be described here.

[0141] The interaction control apparatus 400 provided in the embodiments of the present application is used to execute the interaction control method shown in the above Figure 2 , and thus can achieve the same effect as the above interaction control method.

[0142] The embodiments of the present application further provide an interaction control device, which can execute the interaction control method and related steps in the method embodiments.

[0143] An embodiment of the present application also provides a computer-readable storage medium having instructions stored thereon, which, when executed, execute the interactive control method and related steps in the above method embodiment.

[0144] An embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the interactive control method and related steps in the above method embodiment.

[0145] In some embodiments, the methods described herein may be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of manufacture.

[0146] The embodiment of the present application also provides an interactive control system 100, such as Figure 5 As shown, the interactive control system 100 includes at least one processor 501 and at least one interface circuit 502 .

[0147] As an example, when the interactive control system 100 includes a processor and an interface circuit, the processor may be Figure 5 The processor 501 shown in the solid line frame (or the processor 501 shown in the dotted line frame) may be Figure 5 The interface circuit 502 shown in the solid line frame (or the interface circuit 502 shown in the dotted line frame). When the interactive control system 100 includes two processors and two interface circuits, the two processors include Figure 5 The processor 501 shown in the solid line frame and the processor 501 shown in the dotted line frame, the two interface circuits include Figure 5 The interface circuit 502 shown in the solid line frame and the interface circuit 502 shown in the dotted line frame are not limited to this.

[0148] The processor 501 and the interface circuit 502 can be interconnected via a line. For example, the interface circuit 502 can be used to receive signals. For another example, the interface circuit 502 can be used to send signals to other devices (such as the processor 501). For example, the interface circuit 502 can read computer instructions stored in the memory and send the computer instructions to the processor 501. The processor 501 executes the instructions and, in conjunction with the input and output devices, implements the various steps in the above embodiments, such as implementing Figure 2 Of course, the interactive control system may also include other discrete components, which are not specifically limited in the embodiments of the present application.

[0149] Through the description of the above embodiments, those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional modules is exemplified, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0150] In several embodiments provided in the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0151] The units described as separate components can or can not be physically separated, and the components displayed as units can be one physical unit or multiple physical units, that is, can be located in one place or can be distributed to multiple different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0152] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0153] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a readable storage medium. Based on such understanding, the technical scheme of the embodiment of the present application essentially or the part that contributes or the whole or part of the technical scheme can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing an apparatus (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0154] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any change or replacement within the technical scope disclosed by the present application should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An interactive control method, characterized in that: The method comprises: Collect various types of action parameters from user interaction actions through different collection methods; Fusing multiple forms of action parameters to determine the intention confidence of the interactive action; In a case where the intention confidence is greater than a confidence threshold, a target control instruction for controlling the target vehicle is determined according to the interactive action.

2. The method according to claim 1, characterized in that The fusion of multiple forms of action parameters to determine the intention confidence of the interactive action includes: Extract the corresponding action features from the action parameters of each form; The action features of each form are fused to determine the intention confidence of the interactive action.

3. The method according to claim 2, characterized in that When the motion parameter includes a hand reflection signal of the user, extracting corresponding motion features from each form of the motion parameter includes: Performing Fourier transform on the reflected signal to calculate the muscle group spectrum energy of the preset muscle group on the user's hand; The muscle micro-vibration characteristics corresponding to the reflection signal are extracted from the muscle group spectrum energy.

4. The method according to claim 3, characterized in that The method further comprises: collecting a pulse excitation generated by the user performing an indication action and a response signal of the vehicle to the user; identifying a vibration transfer function between the vehicle and the user based on the pulse excitation and the response signal; constructing an inverse filter according to the vibration transfer function; The reflected signal is preprocessed using the inverse filter.

5. The method according to claim 2, characterized in that When the motion parameter includes a blood flow signal of the user's hand, extracting corresponding motion features from each form of motion parameter includes: calculating the blood flow acceleration of the user's hand according to the blood flow signal; The muscle contraction strength of the user's hand is determined based on the blood flow acceleration.

6. The method according to claim 2, characterized in that When the motion parameter is a key node signal of a key node of the user's hand, extracting corresponding motion features from each form of the motion parameter includes: determining, according to the key node signal of each key node of the user's hand, a position change of each key node of the user's hand in consecutive frames; Based on the position changes of all key nodes of the user's hand in consecutive frames, the motion trajectory characteristics of the user's hand in the three-dimensional space are determined.

7. The method according to claim 6, characterized in that The method further comprises: Calculating the angular velocity of the user's hand joints based on the position change of each key node of the user's hand in consecutive frames; When it is determined that the joint angular velocity in a preset number of consecutive frames is greater than an angular velocity threshold, the key node signals in the preset number of consecutive frames are removed from the action parameters, and an abnormality warning is generated to indicate that the interactive action is abnormal.

8. The method according to claim 2, characterized in that The step of fusing the action features of each form to determine the intention confidence of the interactive action includes: Acquiring environmental parameters of the target vehicle; Determining a feature weight corresponding to each form feature according to the environmental parameters; The intention confidence of the interactive action is determined based on the action features and corresponding feature weights of each form.

9. The method according to claim 1, characterized in that The motion parameters include a hand reflection signal of the user, a hand blood flow signal of the user, and key node signals of key nodes of the user's hand; The method collects various types of action parameters from the user's interactive actions through different collection methods, including: sending a continuous wave to the user and receiving a hand reflection signal of the user; and / or, obtaining a hand blood flow signal of the user using a preset light source; And / or, the trajectory of each key node of the user's hand is collected to obtain the key node signal of the key node of the user's hand.

10. An interactive control device, characterized in that: The device comprises: The collection module is used to collect various forms of action parameters from the user's interactive actions through different collection methods; A fusion module, configured to fuse various forms of action parameters to determine the intention confidence of the interactive action; A determination module is used to determine a target control instruction for controlling the target vehicle according to the interaction action when the intention confidence is greater than a confidence threshold.

Images