Data processing method, device, system, storage medium and electronic device

Abstract

The present application provides a data processing method, device, system, storage medium and electronic device, which relate to the field of human-computer interaction technology. The data processing method includes: determining the audio signal and image signal of the target sound zone in the cockpit of a movable device; if it is determined based on the audio signal and the image signal that a user is speaking in the target sound zone, then determining the user's speech type; if the user's speech type is human-machine speech, then determining the control instruction based on the audio signal; if the user's speech type is human-human speech, then determining the user's behavior pattern based on the image signal. Through the dual-modal features of the user's voice signal and image signal in the present application, the accuracy of speech recognition can be improved and the false alarm of user speech detection can be reduced. In addition, by judging the user's speech type, the human-computer interaction system actively performs operations in the human-machine speech scene or the human-human speech scene, realizes the wake-up-free function, and improves the human-computer interaction experience.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and in particular to a data processing method, device, system, storage medium and electronic device. Background Art

[0002] With the development of intelligent cars, the interaction between users and cars has become more frequent, and the interaction mode has also evolved from the earliest button-based interaction to touch screen interaction, and then to today's voice-based interaction. Among them, voice-based interaction mainly relies on speech recognition technology, natural language processing technology, and speech synthesis technology.

[0003] However, the relevant voice interaction technology is easily affected by the background noise in the car, especially when the user's voice is too small relative to the background noise, which will reduce the accuracy of voice recognition. In addition, the relevant voice interaction technology usually collects the voice information of users in each voice zone first, and then performs corresponding operations based on the voice information. This "one question and one answer" interaction method is too mechanical and the user experience is average. Summary of the invention

[0004] In order to solve the above technical problems, the present application is proposed. The embodiments of the present application provide a data processing method, device, system, storage medium and electronic device.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, which is applied to a human-computer interaction system in a mobile device, the method comprising: determining an audio signal and an image signal of a target sound zone in a cockpit of the mobile device; if it is determined based on the audio signal and the image signal that a user is speaking in the target sound zone, determining the user's speech type, the speech types including human-computer speech and human-human speech; if the user's speech type is human-computer speech, determining a control instruction based on the audio signal so that the human-computer interaction system performs an operation matching the control instruction; if the user's speech type is human-human speech, determining the user's behavior pattern based on the image signal so that the human-computer interaction system performs an operation matching the user's behavior pattern.

[0006] In combination with the first aspect, in certain implementations of the first aspect, control instructions are determined based on the audio signal, including: determining the user's speech content and speech scene based on the audio signal and the image signal; and determining control instructions that match the user's speech intention based on the speech content and the speech scene.

[0007] In combination with the first aspect, in certain implementations of the first aspect, determining the user's behavior pattern based on the image signal includes: determining the user's head posture deflection angle information, eye information, and lip shape information based on the image signal; determining the user's behavior pattern based on the head posture deflection angle information, eye information, and lip shape information.

[0008] In combination with the first aspect, in certain implementations of the first aspect, the behavior mode includes a cross-audio zone chat mode, and after determining the user's behavior mode, it also includes: if the user's behavior mode is a cross-audio zone chat mode, determining the audio zones of other users in the cockpit with whom the user speaks and communicates; generating a call function activation instruction between the target audio zone and the audio zones of other users in the cockpit; and sending a call function activation instruction to a removable device to activate the call function between the target audio zone and the audio zones of other users in the cockpit.

[0009] In combination with the first aspect, in certain implementations of the first aspect, after determining the audio signal and image signal of the target sound zone in the cockpit of the movable device, it also includes: if it is determined based on the audio signal and the image signal that the target sound zone includes a user, but the user is not speaking, then determining facial feature information of the user included in the target sound zone based on the image signal; based on the facial feature information, determining an audio tuning command that matches the user included in the target sound zone.

[0010] In combination with the first aspect, in certain implementations of the first aspect, the facial feature information includes eye information and head posture deflection angle information, and based on the facial feature information, an audio tuning command matching the user included in the target sound zone is determined, including: if based on the user's eye information and head posture deflection angle information, it is determined that the eyes are in a closed state and the head posture deflection angle falls within a first head posture deflection range, then the audio tuning command is determined to adjust the volume of the target sound zone to a preset rest volume range; if based on the user's eye information and head posture deflection angle information, it is determined that the eyes are in a closed state and the head posture deflection angle falls within a second head posture deflection range, then the audio tuning command is determined to adjust the volume of the target sound zone to a preset sleep volume range.

[0011] In combination with the first aspect, in certain implementations of the first aspect, before determining that a user is speaking in the target sound area based on the audio signal and the image signal, it also includes: extracting speech features on the audio signal to obtain speech timing feature information; extracting lip shape features on the image signal to obtain lip shape timing feature information; and determining whether a user is speaking in the target sound area based on the speech timing feature information and the lip shape timing feature information.

[0012] In a second aspect, an embodiment of the present application provides a data processing device, which is applied to a human-computer interaction system in a movable device, and the device includes: a first determination module, which is used to determine the audio signal and image signal of a target sound zone in a cockpit of the movable device; a second determination module, which is used to determine the user's speech type if it is determined that a user is speaking in the target sound zone based on the audio signal and the image signal, and the speech types include human-computer speech and human-human speech; a third determination module, which is used to determine the control instruction based on the audio signal if the user's speech type is human-computer speech, so that the human-computer interaction system performs an operation matching the control instruction; and a fourth determination module, which is used to determine the user's behavior pattern based on the image signal if the user's speech type is human-computer speech, so that the human-computer interaction system performs an operation matching the user's behavior pattern.

[0013] In a third aspect, an embodiment of the present application provides a human-computer interaction system, which is applied to a movable device, and the human-computer interaction system includes: a data acquisition module, which is used to determine the audio signal and image signal of the target sound zone in the cockpit of the movable device; a dialogue intention judgment module, which is used to determine the user's speech type if it is determined based on the audio signal and the image signal that a user is speaking in the target sound zone, and the speech types include human-computer speech and human-human speech; a semantic engine module, which is used to determine the control instruction based on the audio signal if the user's speech type is human-computer speech, so that the human-computer interaction system performs an operation matching the control instruction; a user behavior recognition module, which is used to determine the user's behavior pattern based on the image signal if the user's speech type is human-human speech, so that the human-computer interaction system performs an operation matching the user's behavior pattern.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is used to execute the data processing method described in the first aspect.

[0015] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; and the processor is used to execute the data processing method described in the first aspect.

[0016] The data processing method provided in the embodiment of the present application first determines whether there is a user speaking in the target sound zone based on the audio signal and image signal of the target sound zone, and then determines the speech type of the speaking user. If the user's speech type is human-computer dialogue, the control instruction is determined so that the human-computer interaction system performs an operation matching the control instruction; if the user's speech type is human-human dialogue, the user's behavior pattern is determined based on the user's image signal so that the human-computer interaction system performs an operation matching the user's behavior pattern. In the embodiment of the present application, firstly, whether the user in the target sound zone is speaking is determined by the dual-mode characteristics of the user's voice signal and image signal, which greatly improves the accuracy of speech recognition, especially the accuracy of speech recognition in a high-noise environment. In addition, by further judging the user's speech type, the human-computer interaction system actively executes the corresponding operations in the human-computer speech scenario or the human-human speech scenario, realizes the wake-up-free function, and improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0018] Figure 1 Shown is a schematic diagram of a scenario applicable to an embodiment of the present application.

[0019] Figure 2 Shown is a flowchart of a data processing method provided by an exemplary embodiment of the present application.

[0020] Figure 3 Shown is a schematic diagram of a flow chart of determining a control instruction provided by an exemplary embodiment of the present application.

[0021] Figure 4 Shown is a flowchart of determining a user's behavior pattern provided by an exemplary embodiment of the present application.

[0022] Figure 5 Shown is a flowchart of a data processing method provided by another exemplary embodiment of the present application.

[0023] Figure 6 Shown is a flowchart of a data processing method provided by yet another exemplary embodiment of the present application.

[0024] Figure 7 It is a schematic diagram of a flow chart of determining an audio tuning command matching a user included in a target sound zone provided by an exemplary embodiment of the present application.

[0025] Figure 8 Shown is a flowchart of a data processing method provided by yet another exemplary embodiment of the present application.

[0026] Fig. 9 Shown is a schematic diagram of the structure of a data processing device provided by an exemplary embodiment of the present application.

[0027] Fig.10 Shown is a schematic diagram of the structure of a human-computer interaction system provided by an exemplary embodiment of the present application.

[0028] Fig.11 Shown is a structural schematic diagram of a human-computer interaction system provided by another exemplary embodiment of the present application.

[0029] Fig.12 Shown is a schematic structural diagram of a facial feature detection module provided by an exemplary embodiment of the present application.

[0030] Fig.13 FIG. 1 is a schematic diagram of the structure of a multimodal activity detection module provided by an exemplary embodiment of the present application.

[0031] Fig.14 Shown is a schematic diagram of the structure of a dialogue intention identification module provided by an exemplary embodiment of the present application.

[0032] Fig.15 Shown is a schematic diagram of the structure of a multimodal speech recognition module provided by an exemplary embodiment of the present application.

[0033] Fig.16 Shown is a schematic diagram of the structure of a user behavior identification module provided by an exemplary embodiment of the present application.

[0034] Fig.17 Shown is a schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] With the development of intelligent automobiles, the interaction between users and automobiles generally adopts voice interaction. Voice interaction technology usually includes voice recognition, natural language processing and speech synthesis. Existing voice interaction technology has realized multi-zone human-computer interaction in the car, that is, users in different positions in the car can interact with the car system by voice. However, this method is easily disturbed by the background noise in the car, such as the speaker playing volume is too loud, the noise level of wind noise and road noise is too high, and multiple people are talking at the same time; or the user's voice is too small relative to the background noise, which will affect the accuracy of voice recognition. In addition, when multiple people are talking in the car, the relevant voice recognition technology usually requires the user to say the wake-up word first to avoid false triggering of the voice recognition function, and then say the user's needs. All of the above problems will reduce the user experience.

[0037] In order to solve the above problems, the embodiment of the present application proposes a data processing method, which combines the audio signal and image signal of the target sound zone to determine whether the user in the target sound zone is in a speaking state, thereby reducing the influence of background noise and improving the accuracy of speech recognition. If it is determined that there is a user in the target sound zone who is in a speaking state, the user's speech type is determined so that the human-computer interaction system performs an operation that matches the user's speech intention based on the speech type, thereby realizing the wake-up-free function and improving the user's experience.

[0038] Combine the following Figures 1 to 16 The data processing method, device, system, storage medium and electronic device provided in the embodiments of the present application are introduced in detail.

[0039] Figure 1 The figure shows a schematic diagram of a scenario to which the embodiment of the present application is applicable. Figure 1 As shown, the scene includes a human-computer interaction system 110, a vehicle-mounted audio system 120, and a vehicle-mounted navigation system 130. Among them, there is a communication connection relationship between the human-computer interaction system 110, the vehicle-mounted audio system 120, and the vehicle-mounted navigation system 130.

[0040] Specifically, the human-computer interaction system 110 is used to determine the audio signal and image signal of the target sound zone, and when it is determined that a user is speaking in the target sound zone based on the audio signal and the image signal, further determine the user's speech type. If the user's speech type is everyone speaking, then determine the control instruction corresponding to the user's audio signal; if the user's speech type is everyone speaking, then determine the user's behavior pattern based on the user's image signal. Further, the matching operation can be that the human-computer interaction system 110 sends an instruction to adjust the audio parameters to the vehicle audio system 120, or it can be that the human-computer interaction system 110 sends an instruction to the vehicle audio system 120 to turn on the function of each sound zone talking to each other, or it can be that the human-computer interaction system 110 sends an instruction to navigate to the destination to the vehicle navigation system 130. The embodiment of the present application does not limit the specific matching operation. In addition, in addition to the vehicle audio system 120 and the vehicle navigation system 130 listed in the embodiment of the present application, the human-computer interaction system can also communicate with other vehicle control systems to meet the needs of users.

[0041] Figure 2 FIG. 1 is a flow chart of a data processing method provided by an exemplary embodiment of the present application. For example, the method is applied to a human-computer interaction system in a mobile device. Figure 2 As shown, the data processing method provided in the embodiment of the present application includes the following steps.

[0042] Step S210: determining the audio signal and the image signal of the target sound zone in the cockpit of the movable device.

[0043] For example, for a driving compartment of a mobile device (such as a car), the target sound zone may be a driver's position, a co-driver's position, and a rear position relative to the driver's position. In addition, the number of target sound zones may be one or more.

[0044] Specifically, the audio signal of the target sound zone can be determined by the microphone array in the human-computer interaction system, and the image signal of the target sound zone can be determined by the camera device in the human-computer interaction system. Of course, the embodiment of the present application only exemplifies the device for determining the audio signal and image signal of the target sound zone, and those skilled in the art can also select other appropriate devices according to specific circumstances.

[0045] For example, if a camera device is used to collect image signals of the target sound zone, in order to ensure the acquisition of high-quality image signals, it is recommended that the angle between the camera and the face is less than 60°. Preferably, the angle between the camera and the face is equal to 30°. Furthermore, the camera is generally installed at the bottom of the A-pillar, above the steering column, and near the rearview mirror. In addition, the camera channel mode is recommended to support IR, IR+RGB, with a resolution of 720P and above, and the light source type is LED or VCSEL.

[0046] Step S220: If it is determined based on the audio signal and the image signal that a user is speaking in the target audio zone, the user's speech type is determined.

[0047] Exemplarily, the speech types include human-to-human speech and human-to-machine speech.

[0048] Step S230: If the user's speech type is human-machine speech, a control instruction is determined based on the audio signal.

[0049] Step S240: If the user's speech type is everyone speaking, the user's behavior pattern is determined based on the image signal.

[0050] Specifically, the human-computer interaction system performs matching operations based on control instructions or user behavior patterns. The operation may be sending a request instruction to adjust audio parameters to the vehicle audio system, or sending a route request instruction to navigate to the destination to the vehicle navigation system, or sending a request instruction to call the target contact to the vehicle call system, etc. The present application does not limit the type of operation performed by the human-computer interaction system, and those skilled in the art may select the specific operation type according to actual conditions.

[0051] The data processing method provided in the embodiment of the present application first determines whether there is a user speaking in the target sound zone based on the audio signal and image signal of the target sound zone, and then determines the speech type of the speaking user. If the user's speech type is human-computer dialogue, the control instruction is determined so that the human-computer interaction system performs an operation matching the control instruction; if the user's speech type is human-human dialogue, the user's behavior pattern is determined based on the user's image signal so that the human-computer interaction system performs an operation matching the user's behavior pattern. In the embodiment of the present application, firstly, whether the user in the target sound zone is speaking is determined by the dual-mode characteristics of the user's voice signal and image signal, which greatly improves the accuracy of speech recognition, especially the accuracy of speech recognition in a high-noise environment. In addition, by further judging the user's speech type, the human-computer interaction system actively executes the corresponding operations in the human-computer speech scenario or the human-human speech scenario, realizes the wake-up-free function, and improves the user experience.

[0052] Figure 3 FIG. 1 is a flow chart of determining a control instruction provided by an exemplary embodiment of the present application. Figure 2 Based on the embodiment shown, Figure 3 The embodiment shown is described below in detail. Figure 3 The embodiment shown and Figure 2 The differences and similarities of the illustrated embodiments are not described in detail.

[0053] like Figure 3 As shown, in an embodiment of the present application, determining a control instruction based on an audio signal includes the following steps.

[0054] Step S310: determining the user's speech content and speech scene based on the audio signal and the image signal.

[0055] Specifically, the user's lip image is first extracted according to the image signal, and the user's lip shape information is further obtained according to the lip image. In addition, the user's voice information is extracted according to the audio signal.

[0056] Exemplarily, the user's lip shape information is input into a multi-layer neural network to obtain the time series coding of the lip shape information in a high-dimensional space, and the user's voice information is input into a multi-layer neural network to obtain the time series coding of the voice information in a high-dimensional space, and the output voice time series coding sequence frequency and lip shape time series coding sequence frequency must be synchronized. The voice time series coding sequence and the lip shape time series coding sequence are further fused, and phoneme recognition is performed to obtain a phoneme probability distribution time series sequence. Then, the phoneme probability distribution time series sequence is decoded to finally determine the user's speech content. It should be noted that those skilled in the art may also use other methods to obtain the time series coding of lip shape information and voice information in a high-dimensional space, and the embodiments of the present application are not limited to this.

[0057] Furthermore, semantic understanding is performed based on the user's speech content, and a preset voice application scenario is matched based on the semantic understanding result.

[0058] The extracted lip image of the user must meet the requirements shown in Table 1.

[0059] Table 1 Lip image requirements

[0060] name index Spatial frequency response Cycles / face ≥80 Image distortion ≤10％ Dynamic Range Effective grayscale ≥ 16 levels Lip effective pixels ≥90×90; recommended 120×120 Frame rate 25 / 30FPS image <100ms

[0061] Step S320: Determine a control instruction that matches the user's speech intention based on the speech content and the speech scene.

[0062] Specifically, the control instruction is a pre-set command that matches the user's speech content and speech scene. Exemplarily, the control instruction may be a request instruction for making a phone call, or a request instruction for listening to music, or a request instruction for listening to the radio, or a request instruction for navigation.

[0063] In the embodiment of the present application, the user's speech scene and speech intention are determined through the user's audio signal and image signal, and the control instruction is further determined based on the speech scene and speech intention, so that the human-computer interaction system actively executes the operation matching the control instruction. Through the scheme in the embodiment of the present application, the human-computer interaction system is more intelligent, abandoning the traditional "one question and one answer" interaction mode, and the user does not need to wake up the human-computer interaction system through the wake-up word, which greatly improves the user's interaction experience.

[0064] Figure 4 FIG. 1 is a flow chart of determining a user's behavior pattern provided by an exemplary embodiment of the present application. Figure 2 Based on the embodiment shown, Figure 4 The embodiment shown is described below in detail. Figure 4 The embodiment shown and Figure 2 The differences and similarities of the illustrated embodiments are not described in detail.

[0065] like Figure 4 As shown, in an embodiment of the present application, determining the user's behavior pattern based on the image signal includes the following steps.

[0066] Step S410: determining the user's head posture deflection angle information, eye information, and lip shape information based on the image signal.

[0067] Specifically, according to the image signal, the image coordinates of the face rectangle frame of each target sound zone in the image are first determined, and then the face image corresponding to each target sound zone is obtained. The face key point detection is performed on the face image of each sound zone to obtain the coordinates of 68 face key points in the face image of each target sound zone. Then, the head posture deflection angle information of the face image of each target sound zone in the reference coordinate system is calculated, specifically including the yaw angle, pitch angle, and roll angle. Exemplarily, the reference coordinate system is the camera coordinate system. In addition, the user's eye information and lip shape information are obtained according to the image signal. For example, the user's eye information and lip shape information can be further obtained by obtaining the user's eye image and lip image.

[0068] The acquired lip image of the user must meet the requirements shown in Table 1. The acquired eye image of the user must meet the requirements shown in Table 2.

[0069] Table 2 Eye image requirements

[0070]

[0071]

[0072] Step S420: determining the user's behavior pattern based on the head posture deflection angle information, eye information, and lip shape information.

[0073] In the embodiment of the present application, the user's head posture deflection angle information, eye information and lip shape information are determined through the user's image signal, and the user's behavior pattern is further determined based on the user's head posture deflection angle information, eye information and lip shape information, so that the human-computer interaction system actively performs operations that match the user's behavior pattern. Similarly, the solution in the embodiment of the present application makes the human-computer interaction system more intelligent, abandons the traditional "one question and one answer" interaction method, and does not require the user to wake up the human-computer interaction system through a wake-up word, which greatly improves the user's interaction experience.

[0074] Figure 5 FIG. 1 is a flow chart of a data processing method provided by another exemplary embodiment of the present application. Figure 2 Based on the embodiment shown, Figure 5 The embodiment shown is described below in detail. Figure 5 The embodiment shown and Figure 2 The differences and similarities of the illustrated embodiments are not described in detail.

[0075] like Figure 5 As shown, in an embodiment of the present application, the behavior pattern includes a cross-audio zone chat mode. After determining the user's behavior pattern, the following steps are also included.

[0076] Step S510, determining whether the user's behavior mode is a cross-audio zone chat mode.

[0077] Exemplarily, in actual application, if the judgment result of step S510 is yes, that is, the user's behavior mode is a cross-audio zone chat mode, then steps S520 to S540 are executed; if the judgment result of step S510 is no, that is, the user's behavior mode is not a cross-audio zone chat mode, then step S550 is executed.

[0078] Step S520, determining the voice zones of other users in the cockpit with whom the user is speaking and communicating.

[0079] Specifically, the voice zones of other users in the cockpit may be in a left-right voice zone relationship or a front-back voice zone relationship with the voice zone of the current speaker. That is, at least two users in speech communication may be in a left-right voice zone chat mode or a front-back voice zone chat mode.

[0080] Step S530, generating a call function activation instruction between the target audio zone and the audio zones of other users in the cockpit.

[0081] Step S540: Send a call function activation instruction to the mobile device to activate the call function between the target audio zone and the audio zones of other users in the cockpit.

[0082] Step S550, end.

[0083] In the embodiment of the present application, according to the user's chat mode, the operation corresponding to the user's chat mode is actively performed, which can maximize the satisfaction of the needs matching the user's current state and greatly improve the user's interactive experience.

[0084] Figure 6 FIG. 1 is a flow chart of a data processing method provided by another exemplary embodiment of the present application. Figure 2 Based on the embodiment shown, Figure 6 The embodiment shown is described below in detail. Figure 6 The embodiment shown and Figure 2 The differences and similarities of the illustrated embodiments are not described in detail.

[0085] like Figure 6 As shown, in the embodiment of the present application, after determining the audio signal and image signal of the target sound zone in the driving compartment of the movable device, the following steps are also included.

[0086] Step S610: If it is determined based on the audio signal and the image signal that the target audio zone includes a user, but the user does not speak, facial feature information of the user included in the target audio zone is determined based on the image signal.

[0087] Exemplarily, the user's facial feature information includes the user's eye information, lip shape information, head posture yaw angle, head posture pitch angle, and head posture roll angle.

[0088] As mentioned above, firstly, the image coordinates of the face rectangle frame of each target sound zone in the image are sorted out based on the image signal, and then the face image corresponding to each target sound zone is obtained. Then, the face key point detection is performed according to the face image of each target sound zone, and the coordinates of 68 face key points in the face image of each sound zone are obtained. For example, the Ensemble of Regression Tress (ERT) algorithm can be used to obtain the coordinates of the face key points in the face image of each sound zone. Then, the head posture deflection angle information of the face image of each sound zone in the reference coordinate system is further calculated by the coordinates of the face key points in the face image, including the yaw angle, the pitch angle, and the roll angle. Then, according to the face image of each sound zone, the lip image and the eye image corresponding to each sound zone are extracted respectively. And it is judged whether the user's eyes are open according to the eye image, and it is judged whether the user is speaking according to the lip image.

[0089] Step S620: Determine, based on the facial feature information, an audio tuning command that matches the user included in the target audio zone.

[0090] In an embodiment of the present application, if it is determined that there is a user in the target sound zone but the user does not speak, the facial feature information of the users included in the target sound zone is used to determine the audio tuning command that matches the users included in the target sound zone. Similarly, the solution in the embodiment of the present application abandons the "question-and-answer" mode in traditional human-computer interaction systems, as well as the "wake-up" function based on wake-up words commonly found in traditional human-computer interaction systems, greatly improving the user experience. In addition, the solution in the embodiment of the present application further enriches the human-computer interaction scenarios. Even if the user in the target sound zone does not speak, the human-computer interaction system can still actively perform corresponding operations through the user's facial features.

[0091] Figure 7 FIG. 1 is a flow chart of determining an audio tuning command matching a user included in a target sound zone provided by an exemplary embodiment of the present application. Figure 6 Based on the embodiment shown, Figure 7 The embodiment shown is described below in detail. Figure 7 The embodiment shown and Figure 6 The differences and similarities of the illustrated embodiments are not described in detail.

[0092] like Figure 7 As shown, in an embodiment of the present application, based on facial feature information, determining an audio tuning command that matches a user included in a target sound zone includes the following steps.

[0093] Step S710: determining the range within which the eye state and the head posture deflection angle fall based on the user's eye information and head posture deflection angle information.

[0094] Specifically, the head posture deflection angle includes the head posture yaw angle, the head posture pitch angle, and the head posture roll angle.

[0095] Exemplarily, if the execution result of step S710 is that the eyes are closed and the head posture deflection angle range falls within the first head posture deflection angle range, step S720 is executed; if the execution result of step S710 is that the eyes are closed and the head posture deflection angle range falls within the second head posture deflection angle range, step S730 is executed.

[0096] Exemplarily, the criteria for determining the range within which the user's eye state and head posture deflection angle fall may be referred to as shown in Table 3.

[0097] Table 3 Criteria for determining the range of the user's eye state and head posture deflection angle

[0098]

[0099] Exemplarily, in combination with Table 3, the first head posture deflection range in the embodiment of the present application is (15°, 45°), and the second head posture deflection angle range is greater than 45°. When the user's eyes are closed and the head posture deflection angle falls within the first head posture deflection range, the human-computer interaction system considers that the user is in rest mode; when the user's eyes are closed and the head posture deflection angle falls within the second head posture deflection range, the human-computer interaction system considers that the user is in sleep mode.

[0100] Step S720: determining that the audio tuning command is to adjust the volume of the target sound zone to a preset rest volume range.

[0101] Step S730 , determining that the audio tuning command is to adjust the volume of the target sound zone to a preset sleeping volume range.

[0102] The preset rest volume range and the preset sleep volume range in the embodiment of the present application are both preset manually, and those skilled in the art can set the specific rest volume range and sleep volume range according to actual conditions.

[0103] For example, the voice zone adjustment strategy may refer to that shown in Table 4.

[0104] Table 4 Automatic adjustment scheme of sound zone

[0105]

[0106]

[0107] It should be noted that Table 4 is only an example of a sound zone adjustment strategy and is not a limitation of the sound zone adjustment strategy. Those skilled in the art may also select other types of sound zone tuning schemes, such as reducing the decibel level of the sound zone where the user is located by 20-30 dB when the user is in rest mode, and directly turning off the volume of the sound zone where the user is located when the user is in sleep mode.

[0108] Through the scheme in the embodiments of the present application, the human-computer interaction system actively adjusts the volume of the target sound zone to a preset rest volume range, or adjusts the volume of the target sound zone to a preset sleep volume range according to the user's eye information and head deflection angle information, so as to maximize the hearing needs of users in each sound zone, and there is no need for users in each sound zone to control the human-computer interaction system through wake-up words or specific voice commands, thereby improving the human-computer interaction experience.

[0109] Figure 8 FIG. 1 is a flow chart of a data processing method provided by another exemplary embodiment of the present application. Figure 2 Based on the embodiment shown, Figure 8 The embodiment shown is described below in detail. Figure 8 The embodiment shown and Figure 2 The differences and similarities of the illustrated embodiments are not described in detail.

[0110] like Figure 8 As shown, in the embodiment of the present application, before determining that a user is speaking in the target sound zone based on the audio signal and the image signal, the following steps are also included.

[0111] Step S810: extract speech features from the audio signal to obtain speech time sequence feature information.

[0112] Specifically, based on the audio signal, a beamforming algorithm can be used to obtain the audio signal of each target sound zone, and then the speech signal in the audio signal in each target sound zone is extracted. Then, based on the speech signal of each target sound zone, the speech timing feature information corresponding to each target sound zone is extracted.

[0113] Step S820: extract lip shape features from the image signal to obtain lip shape temporal feature information.

[0114] For example, since the image signal collected by the camera may contain image signals of multiple target sound zones, after acquiring the image signal, the image coordinates of the face rectangle frame of each target sound zone in the image are first sorted out, and the face image signal corresponding to each target sound zone is determined according to the image coordinates. Then, according to the face image signal corresponding to each target sound zone, the lip shape timing feature information of the user in each target sound zone is determined.

[0115] Step S830: Determine whether a user is speaking in the target sound zone based on the speech timing feature information and the lip shape timing feature information.

[0116] Specifically, based on the speech timing feature information and the lip shape timing feature information, the posterior probability of the speech and lip shape bimodal activities is calculated, and then based on the posterior probability, frame level classification coding is implemented, and finally based on the frame level classification coding, it is determined whether there is a user speaking in the target sound area.

[0117] In an embodiment of the present application, the speech timing feature information and lip shape timing feature information of the target sound zone user can be used to determine in real time whether the user in each target sound zone is speaking, thereby improving the accuracy of speech recognition and reducing false alarms in the target sound zone user speech detection.

[0118] Combination of the above Figures 2 to 8 , describes in detail the data processing method embodiment of the present application, and the following is combined with Figures 9 to 16 , describes the device and system embodiments of the present application in detail. It should be understood that the description of the method embodiment corresponds to the description of the device and system embodiment, so the parts not described in detail can refer to the previous method embodiment.

[0119] Fig. 9FIG. 1 is a schematic diagram of the structure of a data processing device provided by an exemplary embodiment of the present application. Fig. 9 As shown, the data processing device 90 provided in the embodiment of the present application includes:

[0120] A first determination module 910, configured to determine an audio signal and an image signal of a target sound zone;

[0121] A second determination module 920 is configured to determine the user's speech type if it is determined based on the audio signal and the image signal that the user is speaking in the target audio zone, where the speech type includes human-machine speech and human-human speech;

[0122] The third determination module 930 is used to determine the control instruction based on the audio signal if the user's speech type is human-computer speech, so that the human-computer interaction system performs an operation matching the control instruction;

[0123] The fourth determination module 940 is used to determine the user's behavior pattern based on the image signal if the user's speech type is everyone's speech, so that the human-computer interaction system performs an operation matching the user's behavior pattern.

[0124] In one embodiment of the present application, the third determination module 930 is also used to determine the user's speech content and speech scene based on the audio signal and the image signal; based on the speech content and speech scene, determine the control instruction that matches the user's speech intention.

[0125] In one embodiment of the present application, the fourth determination module 940 is also used to determine the user's head posture deflection angle information, eye information and lip shape information based on the image signal; and determine the user's behavior pattern based on the head posture deflection angle information, eye information and lip shape information.

[0126] In one embodiment of the present application, the fourth determination module 940 is also used to, if the user's behavior mode is a cross-voice zone chat mode, determine the voice zones of other users in the cockpit with whom the user speaks and communicates; generate a call function activation instruction between the target voice zone and the voice zones of other users in the cockpit; and send a call function activation instruction to a removable device to activate the call function between the target voice zone and the voice zones of other users in the cockpit.

[0127] In one embodiment of the present application, the fourth determination module 940 is also used to, if it is determined based on the audio signal and the image signal that the target sound zone includes a user, but the user does not speak, determine the facial feature information of the user included in the target sound zone based on the image signal; and determine the audio tuning command that matches the user included in the target sound zone based on the facial feature information.

[0128] In one embodiment of the present application, the fourth determination module 940 is also used to determine the audio tuning command to adjust the volume of the target sound zone to a preset rest volume range if it is determined based on the user's eye information and head posture deflection angle information that the eyes are in a closed state and the head posture deflection angle falls within a first head posture deflection range; if it is determined based on the user's eye information and head posture deflection angle information that the eyes are in a closed state and the head posture deflection angle falls within a second head posture deflection range, then the audio tuning command is determined to adjust the volume of the target sound zone to a preset sleep volume range.

[0129] In one embodiment of the present application, the first determination module 910 is also used to extract speech features from the audio signal to obtain speech timing feature information; to extract lip shape features from the image signal to obtain lip shape timing feature information; and to determine whether a user is speaking in the target sound area based on the speech timing feature information and the lip shape timing feature information.

[0130] Fig.10 FIG. 1 is a schematic diagram of the structure of a human-computer interaction system provided by an exemplary embodiment of the present application. Fig.10 As shown, the human-computer interaction system 100 provided in the embodiment of the present application includes:

[0131] The data acquisition module 1010 is used to determine the audio signal and image signal of the target sound zone in the driving cabin of the movable device;

[0132] The dialogue intention identification module 1020 is used to determine the user's speech type if it is determined based on the audio signal and the image signal that the user is speaking in the target audio zone, and the speech type includes human-machine speech and human-human speech;

[0133] The semantic engine module 1060 is used to determine the control instruction based on the audio signal if the user's speech type is human-computer speech, so that the human-computer interaction system performs an operation matching the control instruction;

[0134] The user behavior recognition module 1080 is used to determine the user's behavior pattern based on the image signal if the user's speech type is human speech, so that the human-computer interaction system performs an operation matching the user's behavior pattern.

[0135] Next, combine Figures 11 to 16 , and specifically introduces the working process of the human-computer interaction system.

[0136] Fig.11 FIG. 2 is a schematic diagram of a human-computer interaction system provided by another exemplary embodiment of the present application. Fig.11As shown, the data acquisition module 1010 first collects the audio signal and image signal of the target sound zone user, and the facial feature detection module 1020 determines the facial features corresponding to the target sound zone user based on the image signal, such as eye image, lip image, head posture yaw angle, head posture pitch angle, head posture roll angle, etc. The multimodal activity detection module 1030 determines the speech timing information and lip shape timing information of the target sound zone user according to the audio signal and lip image of the target sound zone user, and then determines whether the target sound zone user is speaking again.

[0137] Further, if the multimodal activity detection module 1030 determines that there is a user in the target audio zone, but no user is speaking, the user behavior recognition module 1080 is entered. The user behavior recognition module 1080 preliminarily determines the behavior mode of the user in each target audio zone, such as sitting mode, rest mode, sleep mode, etc., based on the facial feature information such as the eye image, lip image, head posture yaw angle, head posture deflection angle, and head posture roll angle obtained by the facial feature detection module 1020. The audio zone automatic adjustment module 1090 performs an operation that matches the user's behavior mode based on the judgment result of the user behavior recognition module 1080. For example, if the user is in the sitting mode, the audio zone automatic adjustment module 1090 does not perform any operation; if the user is in the rest mode, the audio zone automatic adjustment module 1090 determines that the audio tuning command is to adjust the volume of the target audio zone to the preset rest volume range.

[0138] Furthermore, if the multimodal activity detection module 1030 determines that there is at least one user speaking in the target audio zone, the conversation intention determination module 1040 is entered. The conversation intention determination module 1040 can distinguish whether the current speaking user is talking to the machine or to other passengers in the car, that is, the conversation intention determination module can determine whether the current speaking user is talking to a machine or to a person.

[0139] Furthermore, if the conversation intention identification module 1040 determines that the current speaking user is a person-to-person conversation, the user behavior recognition module 1080 is entered. The user behavior recognition module 1080 determines whether the behavior mode of the current speaking user is a cross-sound zone chat mode, and if so, determines the sound zones of other users in the cockpit who are communicating with the current speaking user. The sound zone automatic adjustment module 1090 sends a call function activation instruction to the mobile device, so that the mobile device activates the call function between the sound zone where the current speaking user is located and the sound zone where other users are located.

[0140] Furthermore, if the dialogue intention identification module 1040 determines that the current speaking user is a human-computer dialogue, the multimodal speech recognition module 1050 is entered. The multimodal speech recognition module 1050 reads the speech information and lip shape information of the user in the target voice zone, predicts the content of the user's speech, and finally outputs the recognition result of the speech content. Then, the semantic engine module 1060 performs semantic understanding on the recognition result of the speech content and matches the corresponding voice instructions according to the preset voice application scenario. Then, the voice application module 1070 operates the corresponding system according to the instructions matched by the semantic engine module 1060, for example, operates the car audio system to meet the personalized needs of the user in the target voice zone.

[0141] It should be noted that if no valid face image, eye information or lip shape information is obtained, the human-computer interaction system in the embodiment of the present application stops working, and further performs human-computer interaction through other interaction methods set in the vehicle. For example, human-computer interaction can be performed through a wake-up word.

[0142] Below, according to Figures 12 to 16 The specific working processes of the facial feature detection module 1020 , the multimodal activity detection module 1030 , the conversation intention identification module 1040 , the multimodal speech recognition module 1050 and the user behavior recognition module 1080 are further introduced.

[0143] Fig.12 FIG. 2 is a schematic diagram of the structure of a facial feature detection module provided by an exemplary embodiment of the present application. Fig.12 As shown, the facial feature detection module includes a face detection unit, a face key point detection unit, a head posture detection unit, an eye occlusion detection unit and an eye image extraction unit, and a lip occlusion detection unit and a lip image extraction unit.

[0144] Specifically, the face detection unit sorts out the coordinates of the face rectangle frame of each target sound zone in the face image, thereby obtaining the face image of each target sound zone. The face key point detection unit obtains the coordinates of 68 face key points in the face image of each target sound zone. The head posture detection unit calculates the head posture deflection angle information of the face image of each target sound zone in the reference coordinate system, including the head posture yaw angle, the head posture pitch angle, and the head posture roll angle. According to the head posture deflection angle information, it is further determined whether the face angle of the user in each target sound zone meets the eye image extraction requirements and the lip image extraction requirements. The eye occlusion detection unit determines whether the eyes of the face image of each target sound zone are occluded, and then obtains the eye image of each target sound zone through the eye image extraction unit. The mouth occlusion detection unit determines whether the mouth of the face image of each target sound zone is occluded, and then obtains the lip image of each target sound zone through the lip image extraction unit.

[0145] If the face detection unit does not recognize a face in a target sound zone, the human-computer interaction system is notified that there is no face information in the target sound zone. If a face in the target sound zone is recognized, but the eye image or lip image extraction requirements cannot be met due to eye occlusion or lip occlusion, or the face angle is too large, the human-computer interaction system is notified that the eye image or lip image cannot be extracted in the sound zone.

[0146] Fig.13 FIG. 2 is a schematic diagram of the structure of a multimodal activity detection module provided by an exemplary embodiment of the present application. Fig.13 As shown, the multimodal activity detection module includes a speech activity detection unit, a lip activity detection unit, an activity fusion unit, a post-processing unit, and a result output unit.

[0147] Specifically, the voice activity detection unit extracts voice features based on the audio signal, and the lip shape activity detection unit extracts lip shape features based on the lip image. Both the voice activity detection unit and the lip shape activity detection unit can use the neural network algorithm to complete the above functions. The activity fusion unit calculates the posterior probability of multimodal activity detection based on the extracted voice features and lip shape features. The post-processing unit mainly realizes the encoding of the posterior probability to the frame level classification. The result output unit finally outputs the result of multimodal activity detection.

[0148] Fig.14 FIG. 1 is a schematic diagram of the structure of a dialogue intention identification module provided by an exemplary embodiment of the present application. Fig.14 As shown, the dialogue intention identification module includes a preprocessing unit, a real-time semantic judgment unit and a classification result output unit.

[0149] The dialogue intention recognition module is a new technology based on the in-vehicle environment that can recognize the user's speaking intention without the user waking up the human-computer interaction system with a wake-up word. The dialogue intention recognition module can return to the voice scene represented by the sentence before the voice ends, such as listening to music, listening to the radio, navigating, making calls or in-vehicle control operations, and can also intelligently distinguish whether the current speech type is a human-computer dialogue or a human-to-human dialogue, so as to achieve more intelligent intention recognition.

[0150] Specifically, the audio signal first passes through the preprocessing unit to filter out the channel information and compress the audio features to extract a certain amount of semantic information. For example, the preprocessing unit can use a multi-layer convolutional neural network to extract semantic information. The real-time semantic judgment unit performs real-time step encoding on the voice signal after removing the channel information, and semantically classifies the voice signal. This process can also be processed using a recurrent neural network structure. The classification result output unit outputs the classification results, such as human-computer dialogue, human-to-human dialogue, or more detailed listening to music, listening to the radio, navigation, making phone calls, and other vehicle control scenarios.

[0151] Fig.15FIG. 2 is a schematic diagram of the structure of a multimodal speech recognition module provided by an exemplary embodiment of the present application. Fig.15 As shown, the multimodal speech recognition module includes a speech encoder, a video encoder, an audio fusion unit, a phoneme recognition unit, a decoder, and a recognition result output unit.

[0152] Specifically, the speech encoder inputs filter group parameters, and outputs the time sequence coding of speech information in high-dimensional space through a multi-layer neural network, and the output sequence frequency is synchronized with the video side. The video encoder inputs lip shape information, and outputs the time sequence coding of lip shape information in high-dimensional space through a multi-layer neural network. The audio and video fusion unit merges the speech time sequence coding sequence and the lip shape time sequence coding sequence into an audio and video fusion coding sequence as the input of the phoneme recognition unit. The phoneme recognition unit outputs the phoneme probability distribution time sequence, which is decoded by a separate decoder combined with a language model, and the result output unit outputs the final recognition result.

[0153] Fig.16 FIG. 1 is a schematic diagram of the structure of a user behavior identification module provided by an exemplary embodiment of the present application. Fig.16 As shown, the user behavior recognition module includes a facial behavior detection unit and a chat mode discrimination unit.

[0154] Specifically, the facial behavior detection unit is used to determine the user's behavior mode based on the user's eye information, lip shape information, and head posture deflection angle information, for example, whether the user is in a sitting mode, a speaking mode, or a resting mode. When the user is in a speaking mode, the chat mode determination unit further determines whether the user is in a left-right audio zone chat mode or a front-back audio zone chat mode.

[0155] pass Figures 11 to 16 In the embodiment shown, the multimodal activity detection module detects voice activity by combining the timing information of voice and lip shape, thereby determining in real time whether the user in each target audio zone is speaking, thereby improving the accuracy of voice recognition, especially the accuracy of voice recognition in a high-noise environment, and reducing the false alarm of the target audio zone user's speech detection. When used in conjunction with the dialogue intention discrimination module, the wake-up-free function of the human-computer interaction system is realized, thereby improving the user's experience. In addition, based on the user's eye information, lip information, and head posture deflection angle information of the target audio zone user, the user's current behavior state is determined, and the sound effects of each audio zone are automatically adjusted to maximize the sound effects of each audio zone to meet the user's hearing needs, that is, to create a virtual personal assistant for each audio zone in the car, further improving the experience of human-computer interaction.

[0156] Below, reference Fig.17 To describe an electronic device according to an embodiment of the present application. Fig.17 Shown is a schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application.

[0157] like Fig.17 As shown, the electronic device 170 includes one or more processors 1701 and a memory 1702 .

[0158] The processor 1701 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 170 to perform desired functions.

[0159] The memory 1702 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1701 may run the program instructions to implement the data processing methods of the various embodiments of the present application described above and / or other desired functions. Various contents such as audio signals and image signals including target sound zones, user speech types, etc. may also be stored in the computer-readable storage medium.

[0160] In one example, the electronic device 170 may further include: an input device 1703 and an output device 1704 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0161] The input device 1703 may include, for example, a keyboard, a mouse, etc.

[0162] The output device 1704 can output various information to the outside, including audio signals and image signals of the target sound zone, user's speech type, etc. The output device 1704 can include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0163] Of course, to simplify, Fig.17 Only some of the components related to the present application in the electronic device 170 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 170 may also include any other appropriate components.

[0164] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the data processing method according to various embodiments of the present application described above in this specification.

[0165] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0166] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the data processing method according to various embodiments of the present application described above in this specification.

[0167] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0168] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0169] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0170] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0171] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0172] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A data processing method, characterized in that: A human-computer interaction system applied to a mobile device, the method comprising: Determine an audio signal and an image signal of a target sound zone in a cockpit of the movable device; If it is determined based on the audio signal and the image signal that a user is speaking in the target sound zone, then determining the user's speech type, the speech type including human-machine speech and human-human speech; If the speech type of the user is the human-computer speech, determining a control instruction based on the audio signal so that the human-computer interaction system performs an operation matching the control instruction; If the speech type of the user is the human speech, determining the behavior pattern of the user based on the image signal so that the human-computer interaction system performs an operation matching the behavior pattern of the user; The behavior mode includes a cross-audio zone chat mode. After determining the behavior mode of the user, the method further includes: If the behavior mode of the user is the cross-voice zone chat mode, determining the voice zones of other users who are communicating with the user in the cockpit; generating a call function activation instruction between the target audio zone and the audio zone of the other user in the cockpit; The call function activation instruction is sent to the mobile device to activate the call function between the target sound zone and the sound zone of the other user in the cockpit.

2. The data processing method according to claim 1, characterized in that: The step of determining a control instruction based on the audio signal comprises: Determining the speech content and speech scene of the user based on the audio signal and the image signal; Based on the speech content and the speech scene, a control instruction matching the user's speech intention is determined.

3. The data processing method according to claim 1, characterized in that: The determining the behavior pattern of the user based on the image signal includes: Based on the image signal, determining the head posture deflection angle information, eye information and lip shape information of the user; The user's behavior pattern is determined based on the head posture deflection angle information, the eye information and the lip shape information.

4. The data processing method according to any one of claims 1 to 3, characterized in that: After determining the audio signal and the image signal of the target sound zone in the cockpit of the movable device, the method further includes: If it is determined based on the audio signal and the image signal that the target sound zone includes a user but the user does not speak, then determining facial feature information of the user included in the target sound zone based on the image signal; Based on the facial feature information, an acoustic tuning command matching the user included in the target sound zone is determined.

5. The data processing method according to claim 4, characterized in that: The facial feature information includes eye information and head posture deflection angle information, and determining the sound tuning command matching the user included in the target sound zone based on the facial feature information includes: If it is determined based on the user's eye information and head posture deflection angle information that the eyes are in a closed state and the head posture deflection angle falls within a first head posture deflection range, then the audio tuning command is determined to adjust the volume of the target sound zone to a preset rest volume range; If, based on the user's eye information and head posture deflection angle information, it is determined that the eyes are in a closed state and the head posture deflection angle falls within a second head posture deflection range, the audio tuning command is determined to adjust the volume of the target sound zone to a preset sleep volume range.

6. The data processing method according to any one of claims 1 to 3, characterized in that: Before determining that a user is speaking in the target sound zone based on the audio signal and the image signal, the method further includes: Extracting speech features from the audio signal to obtain speech time sequence feature information; Extracting lip shape features from the image signal to obtain lip shape temporal feature information; Based on the speech timing feature information and the lip shape timing feature information, it is determined whether a user speaks in the target sound zone.

7. A data processing device, characterized in that: A human-computer interaction system applied to a mobile device, the device comprising: A first determination module is used to determine the audio signal and the image signal of the target sound zone in the cockpit of the movable device; A second determination module is configured to determine the speech type of the user if it is determined based on the audio signal and the image signal that a user is speaking in the target sound zone, wherein the speech type includes human-machine speech and human-human speech; A third determination module is used to determine a control instruction based on the audio signal if the speech type of the user is the human-computer speech, so that the human-computer interaction system performs an operation matching the control instruction; a fourth determination module, configured to determine the user's behavior pattern based on the image signal if the user's speech type is the human-to-human speech, so that the human-computer interaction system performs an operation matching the user's behavior pattern; A sending module is used to determine the sound zones of other users in the cockpit who are talking and communicating with the user if the behavior mode of the user is a cross-sound zone chat mode; generate a call function activation instruction between the target sound zone and the sound zones of the other users in the cockpit; and send the call function activation instruction to the mobile device so as to activate the call function between the target sound zone and the sound zones of the other users in the cockpit.

8. A human-computer interaction system, characterized in that: Applied to a mobile device, the human-computer interaction system comprises: A data acquisition module, used to determine the audio signal and image signal of the target sound zone in the cockpit of the movable device; A dialogue intention identification module, configured to determine the speech type of the user if it is determined based on the audio signal and the image signal that a user is speaking in the target audio zone, wherein the speech type includes human-machine speech and human-human speech; A semantic engine module, configured to determine a control instruction based on the audio signal if the speech type of the user is the human-computer speech, so that the human-computer interaction system performs an operation matching the control instruction; A user behavior recognition module, configured to determine the user's behavior pattern based on the image signal if the user's speech type is the human-to-human speech, so that the human-computer interaction system performs an operation matching the user's behavior pattern; The sound zone automatic adjustment module is used to determine the sound zones of other users in the cockpit who are talking and communicating with the user if the user's behavior mode is a cross-sound zone chat mode; generate a call function activation instruction between the target sound zone and the sound zones of the other users in the cockpit; and send the call function activation instruction to the mobile device to activate the call function between the target sound zone and the sound zones of the other users in the cockpit.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the data processing method according to any one of claims 1 to 6.

10. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is used to execute the data processing method described in any one of claims 1 to 6 above.

Images