Interaction control method, vehicle-mounted equipment, vehicle and computer program product

Abstract

The invention provides an interaction control method, vehicle-mounted equipment, a vehicle and a computer program product, and the method comprises the steps: obtaining the voice information and visual information of a target user in a voice interaction process; determining an attention position of a target user according to the visual information; if the voice information is used for indicating to execute the operation associated with the content displayed by the display screen, determining an attention area in the display screen according to the attention position; recognizing the voice information and content in the attention area, and determining a target operation object associated with the voice information from the attention area according to a recognition result; and based on an operation action indicated in the voice information, executing an operation corresponding to the operation action on the target operation object.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to an interactive control method, an in-vehicle device, a vehicle, and a computer program product. Background Art

[0002] In-vehicle devices are usually equipped with voice interfaces, and users can control various services that can be provided by in-vehicle devices through voice. For example, when a user is watching a multimedia content displayed on an electronic device, the user can control the playback of the multimedia content through voice. This service can improve the user experience, reduce the user's operations, and improve the user's efficiency. However, as the number of services that in-vehicle devices can provide increases, the display screens of in-vehicle devices are also getting larger. This allows in-vehicle devices to provide users with related content of multiple services at the same time through the display screen, which requires users to provide precise voice commands to control a certain service, causing inconvenience in operation. Summary of the invention

[0003] The embodiments of the present disclosure at least provide an interactive control method, an in-vehicle device, a vehicle, and a computer program product.

[0004] In a first aspect, an embodiment of the present disclosure provides an interactive control method, including:

[0005] Acquire the target user's voice and visual information during voice interaction;

[0006] Determining the attention position of the target user according to the visual information;

[0007] If the voice information is used to instruct execution of an operation associated with the content displayed on the display screen, determining an attention area in the display screen according to the attention position;

[0008] Recognize the voice information and the content in the attention area, and determine the target operation object associated with the voice information from the attention area according to the recognition result;

[0009] Based on the operation action indicated in the voice information, an operation corresponding to the operation action is performed on the target operation object.

[0010] In this way, by combining visual information and voice information, the target user's attention position is determined through visual information, and when voice information is used to indicate operations associated with content displayed on the display screen, the attention area in the display screen is determined according to the attention position, and then the voice information and the content in the attention area are identified, and the target operation object associated with the voice information is determined from the attention area according to the recognition result, and then based on the operation action indicated in the voice information, the operation corresponding to the operation action is performed on the target operation object, thereby realizing the comprehensive judgment of the target user's operation intention and operation object by using visual information and voice information, and converting the control using voice only into an operation based on the combination of visual information and voice, so that the target user can better judge the target user's operation intention in more concise language and in combination with visual information, thereby improving the convenience and accuracy of the operation.

[0011] In an optional implementation manner, before determining the attention area in the display screen, the method further includes:

[0012] If the attention position is located on the display screen, it is determined that the voice information is used to instruct the execution of an operation associated with the content displayed on the display screen. In an optional implementation,

[0013] In an optional implementation manner, determining the target user's attention position according to the visual information includes:

[0014] determining, based on the visual information, whether the target user has taken an action of indicating the display screen;

[0015] In the case where the target user performs an action of indicating the display screen, the attention position of the target user is determined from the display screen according to the indicating direction of the action.

[0016] In an optional implementation manner, determining, based on the visual information, whether the target user performs an action of indicating the display screen includes:

[0017] Determining action information of the target user according to the visual information;

[0018] Determining, according to the action information, whether the action of the target user is an indication action on the display screen;

[0019] and / or,

[0020] Determining the posture information of the target user according to the visual information;

[0021] It is determined, according to the gesture information, whether the gesture of the target user indicates that the target user has an indication action on the display screen.

[0022] In an optional implementation,

[0023] The performing an operation corresponding to the operation action on the target operation object based on the operation action indicated in the voice information includes:

[0024] Based on the authority information of the target user, determining whether the target user has control authority over the target operation object; wherein the authority information corresponding to the target user is determined based on the seat position of the target user and / or the vehicle status of the target vehicle;

[0025] If so, an operation corresponding to the operation action is performed on the target operation object.

[0026] In an optional implementation manner, determining the attention area in the display screen according to the attention position includes:

[0027] According to the attention position, determining display content associated with the attention position from the display screen;

[0028] The attention area is determined according to a display position of the display content in the display screen.

[0029] In an optional implementation,

[0030] The step of determining a target operation object associated with the voice information from the attention area according to the recognition result includes:

[0031] Generate first feature data corresponding to the attention area, and generate second feature data of the semantics carried in the voice information;

[0032] The first feature data and the second feature data are concatenated to obtain concatenated feature data, and the concatenated feature data is input into an attention coding model, and the concatenated feature data is encoded by the attention coding model to output target feature data;

[0033] Based on the target feature data, an operation action of the target user and a target operation object corresponding to the operation action are determined; the operation object belongs to the attention area.

[0034] In a second aspect, an embodiment of the present disclosure provides a vehicle-mounted device that executes the steps of the above-mentioned first aspect, or any possible implementation of the first aspect.

[0035] In a third aspect, an embodiment of the present disclosure provides a vehicle, comprising the vehicle-mounted device as described in the second aspect.

[0036] In a fourth aspect, an optional implementation of the present disclosure further provides a computer program product, including a computer program, which, when executed by a computer device or a processor, executes the steps of the above-mentioned first aspect, or any possible implementation of the first aspect.

[0037] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure.

[0038] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0040] Figure 1 A flowchart of an interactive control method provided by some embodiments of the present disclosure is shown;

[0041] Figure 2 One of the examples of encoders provided by some embodiments of the present disclosure is shown;

[0042] Figure 3 The second example of the encoder provided by some embodiments of the present disclosure is shown;

[0043] Figure 4 A schematic diagram of a computer device provided by some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0045] The present disclosure provides an interactive control method, which combines visual information and voice information, determines the attention position of a target user through visual information, and in the case where the voice information is used to indicate an operation associated with the content displayed on the display screen, determines the attention area in the display screen according to the attention position, then recognizes the voice information and the content in the attention area, and determines the target operation object associated with the voice information from the attention area according to the recognition result, and then based on the operation action indicated in the voice information, performs the operation corresponding to the operation action on the target operation object, thereby realizing the comprehensive judgment of the operation intention and operation object of the target user by using visual information and voice information, converting the control using only voice into the operation based on the combination of visual information and voice, so that the target user can better judge the operation intention of the target user in more concise language and in combination with visual information, thereby improving the convenience and accuracy of the operation.

[0046] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.

[0047] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0048] The solutions described in this specification and in the examples, if they involve the processing of personal information, will be processed on the premise of having a legal basis (such as obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and will only be processed within the scope of regulations or agreements. If a user refuses to process personal information other than the necessary information for basic functions, it will not affect the user's use of basic functions.

[0049] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the type, scope of use, and usage scenarios of the personal information involved in the present disclosure should be informed to the user in an appropriate manner in accordance with relevant laws and regulations, and the user's authorization should be obtained. For example, a prompt message requesting authorization can be sent to the user through a pop-up window on the page, information push, etc., and the above information can be used after the user agrees.

[0050] To facilitate the understanding of the technical solution of the present disclosure, the technical terms in the embodiments of the present disclosure are first explained:

[0051] Computer Vision (CV) refers to the process of analyzing and processing images or videos by computers. It includes theories, methods and technologies for analyzing and processing images or videos by computers.

[0052] User Generated Content (UGC) refers to the content created by users when using products or services, such as text, pictures, videos, audio, etc., created by users when using a product or service.

[0053] To facilitate understanding of the present embodiment, an interactive control method disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the interactive control method provided in the embodiment of the present disclosure is generally a computer device with certain computing capabilities, and the computer device includes, for example: a terminal device or a server or other processing device, and the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the interactive control method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0054] The interactive control method provided by the embodiment of the present disclosure is described below.

[0055] See also Figure 1 FIG. 1 is a flowchart of an interactive control method provided by an embodiment of the present disclosure, wherein the method includes steps S101 to S105, wherein:

[0056] S101: Acquiring voice information and visual information of a target user during voice interaction;

[0057] S102: Determine the attention position of the target user according to the visual information;

[0058] S103: if the voice information is used to instruct execution of an operation associated with the content displayed on the display screen, determining an attention area in the display screen according to the attention position;

[0059] S104: recognizing the voice information and the content in the attention area, and determining a target operation object associated with the voice information from the attention area according to the recognition result;

[0060] S105: Based on the operation action indicated in the voice information, perform an operation corresponding to the operation action on the target operation object.

[0061] The above S101 to S105 are described in detail below.

[0062] Regarding S101 above:

[0063] In a specific implementation, the target user may be, for example, any user riding in the target vehicle, such as a driver and any passenger in the target vehicle.

[0064] For example, a visual sensor for acquiring images or videos of a user may be provided in the target vehicle. The visual sensor can acquire images or videos of the target user in the target vehicle in real time. The visual information may include images or videos of the target user acquired by the visual sensor.

[0065] For different application scenarios of the target vehicle, for example, different target users can be set; for example, in the case where the target vehicle is a self-use vehicle, the target user is, for example, a user who has the right to use the vehicle, such as the owner, the owner's family, friends, etc.; in this case, for example, the facial feature data of users with the right to use the vehicle can be pre-stored; in the process of executing the interactive control method, for example, the facial feature data can be first used to detect whether the user sitting in each seat is a target user with the right to use; if the user in a certain seat is a target user with the right to use, the control permission is opened for the target user, and after the control permission is opened, the voice information and visual information of the target user in the process of voice interaction between the target user and the vehicle are obtained. If a user in a certain seat does not have the right to use, the control permission will not be opened for the user.

[0066] For another example, when the target vehicle is an operating vehicle, such as a taxi or online car-hailing service, the target user may include the vehicle owner or a passenger in the vehicle. For a passenger, he or she may initiate a ride order through a terminal device; after the driver of the target vehicle accepts the order, the vehicle control system communicates with the device of the target vehicle that accepts the order, sets the passenger as the target user, and opens control permissions for the passenger; after the passenger gets on the vehicle, the target user's behavior detection and processing may be performed in real time to obtain the target user's action information. When the order is completed, the control permissions opened for the passenger may be closed.

[0067] After the target user is determined, the visual information of the target user can be obtained by using the visual perception device deployed in the target vehicle during the voice interaction process.

[0068] The voice information of the target user can be collected by a microphone deployed in the target vehicle, for example, to obtain the audio content sent by the target user, and then obtain the voice information based on the audio content.

[0069] Here, it should be noted that the target user may send out voice information before indicating the attention area, at the same time as indicating the attention area, or after indicating the attention area. Therefore, in the embodiment of the present disclosure, when determining the voice information of the target user based on the audio content, for example, the time when the target user indicates the attention area can be used as the time node, and the historical time node and future time node corresponding to the time node can be determined. A time period is determined by the determined historical time node and future time node, and the voice segment corresponding to the time period is intercepted from the voice content, and then the voice segment is content recognized to obtain the voice information.

[0070] Here, in the embodiment of the present disclosure, voice information refers to instructions issued in the form of voice, rather than in a data format.

[0071] When performing content recognition on a voice segment, for example, the voice segment can be first converted into text, and then semantic recognition processing can be performed on the text to obtain the semantic content expressing the voice information in the text to constitute the voice information corresponding to the voice information.

[0072] Alternatively, the converted text may be directly used as the voice information text of the voice information.

[0073] In addition, in many cases, the environment inside the target vehicle is relatively noisy, with a lot of noise, which greatly interferes with the microphone's sound reception; or the target user's voice decibel when sending voice information is low, which is not conducive to obtaining accurate voice information. Furthermore, in another embodiment of the present disclosure, lip reading recognition processing can also be performed on images or videos obtained by visual sensors to obtain voice information.

[0074] The voice information obtained by lip reading recognition can also be used as auxiliary recognition to correct the voice information received by the microphone and improve the recognition accuracy of the voice information.

[0075] Regarding S102 and S103 above:

[0076] The target user's attention position usually indicates the position where the target user's eyes are looking at, or the position indicated by certain body movements of the target user.

[0077] Specifically, the embodiment of the present disclosure provides a specific method for determining the user's attention position according to the visual information, including:

[0078] determining, based on the visual information, whether the target user has taken an action of indicating the display screen;

[0079] In the case where the target user performs an action of indicating the display screen, the attention position of the target user is determined from the display screen according to the indicating direction of the action.

[0080] Specifically, when determining whether the target user performs an action of indicating the display screen according to the visual information, for example, the action information of the target user may be determined according to the visual information;

[0081] Determining, according to the action information, whether the action of the target user is an indication action on the display screen;

[0082] and / or,

[0083] Determining the posture information of the target user according to the visual information;

[0084] It is determined, according to the gesture information, whether the gesture of the target user indicates that the target user has an indication action on the display screen.

[0085] In a specific implementation, for example, a visual sensor deployed in a target vehicle may be used to obtain an image or video of a target user in real time, and then a behavior detection process is performed on the target user based on the image or video to obtain the action information of the target user.

[0086] The target user's action information includes, for example, the action type of the target user's action and action information, wherein the action type includes, for example, turning the head, the target limb making a pointing action, and the like.

[0087] If it is determined based on the action information that the target user's action is an indication action on the actual display screen, it indicates the target user's operation intention on the display screen.

[0088] Here, for example, a pre-trained action detection model is used to perform action detection processing on the user image acquired by the visual sensor to determine the action performed by the target user. When it is determined that the target user performs the target action, it is then determined whether the behavior of the target user indicates that the target user indicates an action on the display screen.

[0089] The indication action on the display screen may include, for example, a turning action of the head or an indication action of a target limb.

[0090] A: For the case where the indication action on the display screen includes a turning action of the head:

[0091] a1: For example, the following method may be used to determine whether the behavior of the target user indicates that the target user has an indication action on the display screen:

[0092] The head posture of the user image acquired by the visual sensor is detected and processed to obtain the head posture of the target user; after determining the head posture of the target user, it is determined whether the head posture of the target user indicates that the face of the target user is facing a certain display area in the display screen. If so, it is considered that the behavior of the target user indicates that the target user has an indication action on the display screen.

[0093] Afterwards, the attention position of the target user can be determined from the display screen of the target vehicle according to the head posture.

[0094] Here, the head posture of the target user may include, for example, a pitch angle, a yaw angle, and a roll angle. Pitch represents the angle of the head when it pitches forward and backward; yaw represents the angle of the head when it deflects left and right in a direction perpendicular to the top of the skull; and roll represents the angle of the head when it swings left and right in a direction perpendicular to the face. Through pitch and yaw, the facial vertical line located at the nose tip of the target user can be determined. When determining whether the head posture of the target user indicates that the face of the target user is facing a certain display area in the display screen based on the head posture, determine whether the facial vertical line of the target user intersects with the display screen of the target vehicle, and whether the duration is greater than a preset duration. If so, it can be considered that the face is facing a certain display area in the display screen.

[0095] Afterwards, the target user's attention position can be determined based on the intersection point of the facial vertical line and the display screen. In addition, the target user's attention position can also be determined based on a position determined based on the intersection point and having a certain relative position relationship with the intersection point. This position relationship can be determined based on the tip of the human nose, the relative position relationship between the two eyes, and roll.

[0096] Afterwards, when the voice information is used to instruct the execution of an operation associated with the content displayed on the display screen, the attention position is used as the center position of the attention area, and the attention area is determined from the display screen. The size of the attention area can be related to the distance between the target user's head and the center position. The larger the distance, the larger the size of the attention area can be; the smaller the distance, the smaller the size of the attention area can be. The specific determination method is not limited in the embodiments of this disclosure.

[0097] a2: For example, the following method may be used to determine whether the behavior of the target user indicates that the target user has an indication action on the display screen:

[0098] The face image of the target user is obtained from the user image acquired by the visual sensor, and the eye tracking model is used to perform eye tracking processing on the target user included in the face image to extract the sight projection direction of the target user.

[0099] In addition, the area where the eyes are located can be identified from the face image of the target user; then the area where the eyes are located can be detected to determine the position of the pupil, and based on the position of the pupil, the direction of the target user's line of sight can be determined.

[0100] After determining the line of sight projection direction of the target user, determine whether the line of sight projection direction intersects with the display screen of the target vehicle and there is no other obstruction between the two; if so, it is determined that the target user has an indication action on the display screen. At this time, the intersection of the line of sight projection direction and the display screen can be set as the attention position, and when voice information is used to indicate the execution of an operation associated with the content displayed on the display screen, the attention area is determined in the display screen according to the attention position.

[0101] Similar to a1 above, the size of the attention area may be related to the distance between the target user and the display area; the smaller the distance, the smaller the size of the attention area. The specific determination method is not limited in the embodiments of the present disclosure.

[0102] By adopting the above method, it is possible to more accurately detect whether the target user points to the display screen.

[0103] B: For the case where the indicating action on the display screen includes the indicating action of the target limb:

[0104] Here, the target limbs include, for example, fingers, arms, and other limbs that can perform pointing actions. For example, a pre-trained neural network model is used to perform target limb action detection and three-dimensional posture recovery processing on the user image acquired by the visual sensor to obtain the target limb action detection result and three-dimensional posture. If the action detection result indicates that the target limb of the target user has made a pointing action pointing in a certain direction, the indication direction of the target limb when making the pointing action is determined according to the three-dimensional posture of the target limb. If the indication direction of the target limb is a certain position on the display screen, and there are no other obstacles between the indication direction and the display screen, it is considered that the target user has a pointing action on the display screen.

[0105] Afterwards, the attention position is determined according to the indicated position of the indicated direction on the display screen.

[0106] In addition, multiple types of the above-mentioned action information may be combined to more accurately identify the operation intention of the target user, and determine the attention area indicated by the user on the premise of identifying the operation intention of the target user.

[0107] In addition, when determining the attention area in the display screen according to the attention position, the following method may also be used:

[0108] According to the attention position, display content associated with the attention position is determined from the display screen; and according to the display position of the display content in the display screen, the attention area is determined.

[0109] In a specific implementation, after the attention position is determined, the above-mentioned method A or B can be adopted to determine a display area in the display screen with the attention position as the center, as the attention area of ​​the target user. In addition, since the attention position of the target user usually corresponds to a specific display content; the attention position is usually not in the middle of the display content; but there is a deviation, for example, if the user wants to control the accuracy of the displayed movie, the display content corresponding to the user's attention position is the movie player; its attention position is usually located at the position of the progress bar displayed by the player, which is usually located at the lower edge of the player; if the attention area is simply determined with the attention position as the center, it is likely that the final determined attention area cannot include the complete display content that the target user pays attention to, thereby causing a subsequent misunderstanding of the user's operation intention. Furthermore, in the embodiment of the present disclosure, the display content associated with the attention position is determined from the display screen according to the attention position, and then the attention area is determined according to the display position of the display content in the display screen, so that the display content that the target user pays attention to can be completely included in the attention area, reducing the probability of misunderstanding the user's intention.

[0110] Regarding S104 above:

[0111] In a specific implementation, when determining the target operation object associated with the voice information from the attention area according to the recognition result, for example, the following method can be used:

[0112] Generate first feature data corresponding to the attention area, and generate second feature data of the semantics carried in the voice information;

[0113] The first feature data and the second feature data are concatenated to obtain concatenated feature data, and the concatenated feature data is input into an attention coding model, and the concatenated feature data is encoded by the attention coding model to output target feature data;

[0114] Based on the target feature data, an operation action of the target user and a target operation object corresponding to the operation action are determined; the operation object belongs to the attention area.

[0115] In a specific implementation, when generating the first feature data corresponding to the attention area, for example, the following method may be used:

[0116] Using a pre-trained first feature extraction network, feature extraction processing is performed on the attention area to obtain first feature data of the attention area.

[0117] Here, the pre-trained first feature extraction network may include, for example, a deep residual network (ResNet), LeNet-5, AlexNeT, VGG-16, GoogLeNet, etc. The above first feature extraction network may also be a neural network constructed according to application requirements. The embodiments of the present disclosure are not limited and can be determined according to actual needs.

[0118] When generating the second feature data of the voice information, if the voice information is in audio format, it is necessary to first convert the voice information in audio format into text format, and then use natural language processing technology to encode (embedding) the voice information converted into text format, so as to convert the voice information in text format into the corresponding text vector. Here, for example, the whisper model of OpenAI can be used to convert the voice information in audio format into text format.

[0119] If the voice information has been converted into a text format, such as in the embodiment corresponding to S101 above, the voice information is obtained by converting the voice into text and performing semantic recognition on the converted text, then the voice information can be directly encoded to obtain a text vector corresponding to the voice information.

[0120] Then, the second feature extraction network is used to perform feature extraction processing on the text vector corresponding to the voice information to obtain second feature data.

[0121] Here, the pre-trained second feature extraction network may be, for example, a deep residual network (any one of Deep ResNet, LeNet-5, AlexNeT, VGG-16, GoogLeNet, etc.), or a neural network constructed according to application requirements, which is not limited in the embodiments of the present disclosure.

[0122] After obtaining the first characteristic data and the second characteristic data, the first characteristic data and the second characteristic data may be concatenated to obtain concatenated characteristic data.

[0123] When encoding the concatenated feature data through the attention encoding model and outputting the target feature data, for example, the following method may be used:

[0124] Adding position code to the splicing feature data to obtain intermediate feature data;

[0125] The intermediate feature data is input into a pre-trained encoding network, and the encoding network is used to perform encoding processing on the intermediate feature data based on the attention mechanism to obtain the target feature data; wherein the encoding network includes at least one level of encoder; and each level of the encoder includes an attention layer for performing attention processing.

[0126] In a specific implementation, the first feature data is obtained by extracting features of the target display content. Therefore, the first feature data and different areas of the target display content have a certain position mapping relationship; for the target display content, different areas correspond to different meanings, that is, different areas can be regarded as different "words" with different meanings; and the different positions of the "words" will lead to different meanings of the target display content.

[0127] At the same time, the second feature data is obtained by extracting features from the voice information, so the second feature data and each character in the voice information also have a certain position mapping relationship. Different character positions usually lead to different meanings of the voice information.

[0128] Therefore, in the embodiment of the present disclosure, after the first feature data and the second feature data are spliced ​​to obtain the spliced ​​feature data, a position code is added to the spliced ​​feature data, and the position code is used to represent the "words" corresponding to the target display content and the positions of the characters in the voice information in the spliced ​​feature data. In the subsequent processing process, the position order of the above-mentioned "words" and characters is constrained, so that the target feature data can accurately express the user's intentions and the corresponding operation objects.

[0129] For example, assuming that the character vector of a certain character is [1,2,3], and the position encoding vector of this character in the concatenated feature data is [4,5,6], then after adding the position encoding to the character vector, the corresponding vector of the character in the intermediate feature data is also [5,7,9].

[0130] Then, the intermediate feature data is input into the pre-trained encoding network, and the encoding network is used to encode the intermediate feature data based on the attention mechanism to obtain the target feature data.

[0131] The encoding network includes at least one level of encoder; each level of the encoder includes an attention layer for performing attention processing.

[0132] When there are multiple levels of encoders, for the first-level encoder, the input feature data corresponding to the first-level encoder is the above-mentioned intermediate feature data; the input feature data corresponding to other encoders except the first-level encoder is the output feature data of the previous level encoder of the other encoder, that is, the input feature data corresponding to the i-th level encoder is the output feature data of the (i-1)-th level encoder, where i is an integer greater than 1.

[0133] The output feature data of the last stage encoder is also the target feature data.

[0134] In a possible implementation, Figure 2 In the example shown, a specific example of an encoder is provided, in which the encoder includes:

[0135] Multi-Head Attention layer, first-level fusion and normalization layer (Add&Norm), feed forward processing layer (Feed Forward), and second-level fusion and normalization layer.

[0136] Each level of encoder is used to perform the following processing on the input feature data corresponding to each level of encoder:

[0137] Use the multi-head attention layer to perform attention processing on the input feature data to obtain attention feature data;

[0138] The attention feature data and the input feature data are fused by using the fusion and normalization layer to obtain first fused feature data, and then the first fused feature data is normalized to obtain the normalized first fused feature data.

[0139] Using the feedforward processing layer, performing feedforward feature extraction processing on the normalized first fused feature data to obtain feedforward feature data;

[0140] The feedforward feature data and the normalized first fused feature data are fused using a secondary fusion and normalization layer to obtain second fused feature data, and the second fused feature data is then normalized to obtain output feature data corresponding to the encoder.

[0141] like Figure 3 In the example shown, the embodiment of the present disclosure also provides a specific example of another encoder, in which the encoder includes:

[0142] The first-level normalization processing layer (Norm1), multi-head attention layer (Multi-Head Attension), multi-layer perceptron (Multilayer Perceptron, MLP), the first-level fusion layer (Add1), the second-level normalization processing layer (Norm2), the feed forward processing layer (Feed Forward), and the second-level fusion layer (Add2).

[0143] Each level of encoder performs the following processing on the input feature data of each level of encoder:

[0144] Normalizing the input feature data of each level of encoder;

[0145] Using the multi-head attention layer in each level of the encoder, the normalized input feature data is subjected to attention processing to obtain attention feature data; and using the multi-layer perceptron in each level of the encoder, the normalized input feature data is subjected to multi-layer perceptual processing to obtain multi-layer perceptual data;

[0146] Fusing the attention weight information, the multi-layer perception data, and the input feature data to obtain fused feature data;

[0147] After normalizing the fused feature data, the feedforward neural network in each level of the encoder is used to process the normalized fused feature data to obtain feedforward feature data;

[0148] The feedforward feature data and the fused feature data are fused to obtain output feature data corresponding to each level of encoder.

[0149] In this embodiment, the multi-head attention layer can perform attention processing on the normalized input feature data. In the obtained attention feature data, the information that needs to be paid more attention to will have its weight increased, while the information that does not need to be paid attention to will have its weight reduced. Therefore, the obtained attention feature data will carry more information that represents the user's intention and the information of the operation object that the target user wants to operate, while weakening other information.

[0150] At the same time, using a multi-layer perceptron to perform multi-layer perception processing on the normalized input feature data is actually a deeper feature extraction of the normalized input feature data. The obtained multi-layer perception data also carries more information that represents the user's intentions and the information about the operation object that the user wants to operate.

[0151] The above-mentioned attention feature data and multi-layer perception data are further fused with the input feature data, so that the obtained fused feature data can retain information that highlights the user's intention and the object the user wants to operate, and also retains the information lost in the attention processing and multi-layer perception processing.

[0152] Afterwards, the fused feature data is further normalized and processed using a feedforward neural network. The obtained feedforward feature data can also carry more information representing the user's intention and the information of the operation object that the user wants to operate.

[0153] Afterwards, the fused feature data and the feedforward feature data are fused. In this way, based on the above process, the information representing the user intention and the operation object carried in the input feature data can be better extracted to improve the accuracy of identifying the user intention.

[0154] After being encoded by the above-mentioned multi-stage encoder, the target feature data finally formed can more accurately represent the operation intention of the target user.

[0155] Regarding the above S105:

[0156] The target feature number is used to describe the intention information of the target user and the operation object corresponding to the intention. When performing the operation corresponding to the intention information on the operation object according to the intention information described by the target feature data, for example, the following method can be used: convert the target feature data into a target text; the target text includes a first text corresponding to the operation action to be performed, and a second text of the operation object that the target user is concerned about. Generate an operation instruction based on the target text; the operation instruction includes operation action information and operation object information;

[0157] Execute the operation instruction.

[0158] In a specific implementation, the operation of converting the target feature data into the target text is similar to the inverse operation of encoding the text; that is, according to the correspondence between the text and the text vector, the closest text is matched for the target feature data to obtain the target text. The target text includes the first text corresponding to the operation action to be performed and the second text of the operation object that the target user is concerned about. Then, the instruction information of the operation instruction is generated according to the first text, and the object information in the operation instruction is generated according to the operation object, and the operation instruction corresponding to the operation object instruction is executed, thereby realizing the control of the operation object.

[0159] For example, when the target user indicates the attention area, a voice message is issued: Navigate to XX street, XX city.

[0160] At this time, for example, a desktop page of a target vehicle control system is displayed in the attention area, and an icon of a navigation software is displayed in the desktop page.

[0161] The determined operation objects are: the icon corresponding to the navigation software and the navigation page after the navigation software is started;

[0162] The operation to be performed is: start the image of the navigation software to start the navigation software, and after the navigation software is started, automatically enter the navigation destination "XX City XX Street" in the navigation address input bar of the navigation software, and then start navigation.

[0163] For another example, when the target user indicates the target display area, a voice message is sent: search for information about animals in the video.

[0164] At this time, a play page of a video player software is displayed in the attention area, and a certain video is played, and the current playing screen of the video is a certain animal.

[0165] The determined operation object is: the animal played in the video.

[0166] The operation to be performed is: search for the animal in the search software and display the relevant information of the animal.

[0167] In addition, in another embodiment of the present disclosure, when performing an operation corresponding to the operation action on the target operation object based on the operation action indicated in the voice information, for example, the following method may be used:

[0168] Based on the authority information of the target user, determining whether the target user has control authority over the target operation object; wherein the authority information corresponding to the target user is determined based on the seat position of the target user and / or the vehicle status of the target vehicle;

[0169] If so, an operation corresponding to the operation action is performed on the target operation object.

[0170] In this embodiment, the control authority over different display areas in the display screen can be opened to different users.

[0171] Exemplarily, the display screen of the target vehicle may include multiple display areas, including a first display area for displaying vehicle status information, in which the vehicle dashboard, vehicle status prompt information, warning information, etc., such as the status of the airbag, the status of the seat belt, the status of the handbrake, the status of the lights, etc., may be displayed. This area usually does not have an interactive function or only has a limited interactive function. For example, under the display interactive function, the user cannot interact with the displayed vehicle dashboard in terms of operation, but can interact with the displayed warning information, and can obtain the detailed content of the warning information by clicking the icon of the warning information.

[0172] The multiple display areas may also include a second display area, which is used to display relevant interactive interfaces of in-vehicle applications, such as navigation, leisure and entertainment, and in-vehicle games; or provide control capabilities for some in-vehicle devices in the vehicle; these in-vehicle devices are usually not related to driving control, such as the control of devices such as microphones, windows, ambient lights, etc. in the vehicle.

[0173] Different target users may have different operating permissions for different display areas. For example, for the driver, when the target vehicle is in the driving process, he has the operating permission for the first display area and the restricted operating permission for the second display area; here, since the target vehicle is in the driving process, the driver needs to observe the road conditions at any time to ensure driving safety. Therefore, when the target vehicle is at a high speed, if the driver controls the second display area in a visible and verbal manner, it may cause driving safety hazards; therefore, when the target vehicle is in the driving process, the driver can be given the restricted operating permission for the second display area, that is, the driver will be allowed to control the second display area only when the speed is lower than a certain speed threshold. When the speed is higher than a certain speed threshold, the user is not allowed to control the second display area in a visible and verbal manner, so as to avoid the distraction of the user's attention when using the visible and verbal function to control, thereby causing safety hazards.

[0174] For the passengers, they may only have the operation authority for the second display area, thereby preventing the passengers from controlling the first display area and interfering with the driver's driving.

[0175] In addition, when the target vehicle is in a state other than driving, such as parking state, maintenance state, etc., control of the first display area and the second display area can be opened to the driver, passengers, maintenance personnel, etc. to facilitate user operation.

[0176] In this way, by setting control permissions for different display areas for different target users, the safety of the vehicle during driving is ensured.

[0177] Furthermore, when performing an operation corresponding to the operation action on the target operation object based on the operation action indicated in the voice information, it is first determined whether the target user has control authority over the target object. If the target user has control authority over the target object, the operation corresponding to the operation action will be performed on the target operation object, otherwise no execution response will be made to the control behavior of the target user.

[0178] The disclosed embodiment obtains the target user's voice information and visual information during the voice interaction process; determines the target user's attention position based on the visual information; if the voice information is used to indicate the execution of an operation associated with the content displayed on the display screen, determines the attention area in the display screen based on the attention position; identifies the voice information and the content in the attention area, and determines the target operation object associated with the voice information from the attention area based on the recognition result; and performs the operation corresponding to the operation action on the target operation object based on the operation action indicated in the voice information. Thus, the user's operation intention and operation object are comprehensively judged by combining vision and the user's voice information, and the user's manual operation on the device is converted into an operation based on the combination of user action and voice, without relying on the application to actively register the voice information of each measurable control object, so that it has a wider application scenario and meets user needs.

[0179] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0180] The embodiments of the present disclosure also provide a vehicle-mounted device for executing the interactive control method as described in any embodiment of the present disclosure.

[0181] An embodiment of the present disclosure also provides a vehicle, comprising the interactive control device as described in any embodiment of the present disclosure, or the vehicle-mounted equipment as described in an embodiment of the present disclosure.

[0182] The present disclosure also provides a computer device, such as Figure 4 FIG. 1 is a schematic diagram of a computer device structure provided in an embodiment of the present disclosure, including:

[0183] A processor 41 and a memory 42; the memory 42 stores machine-readable instructions executable by the processor 41, and the processor 41 is used to execute the machine-readable instructions stored in the memory 42. When the machine-readable instructions are executed by the processor 41, the processor 41 performs the following steps:

[0184] Acquire the target user's voice and visual information during voice interaction;

[0185] Determining the attention position of the target user according to the visual information;

[0186] If the voice information is used to instruct execution of an operation associated with the content displayed on the display screen, determining an attention area in the display screen according to the attention position;

[0187] Recognize the voice information and the content in the attention area, and determine the target operation object associated with the voice information from the attention area according to the recognition result;

[0188] Based on the operation action indicated in the voice information, an operation corresponding to the operation action is performed on the target operation object.

[0189] The above-mentioned memory 42 includes internal memory 421 and external memory 422; the memory 421 here is also called internal memory, which is used to temporarily store the calculation data in the processor 41 and the data exchanged with the external memory 422 such as the hard disk. The processor 41 exchanges data with the external memory 422 through the internal memory 421.

[0190] The specific execution process of the above instructions can refer to the steps of the interactive control method described in the embodiment of the present disclosure, which will not be repeated here.

[0191] The embodiment of the present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the interactive control method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0192] The present disclosure also provides a computer program product that carries a program code. The program code includes instructions that can be used to execute the steps of the interactive control method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0193] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0194] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0195] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0196] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0197] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0198] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An interactive control method, characterized in that: include: Acquire the target user's voice and visual information during voice interaction; Determining the attention position of the target user according to the visual information; If the voice information is used to instruct execution of an operation associated with the content displayed on the display screen, determining an attention area in the display screen according to the attention position; Recognize the voice information and the content in the attention area, and determine the target operation object associated with the voice information from the attention area according to the recognition result; Based on the operation action indicated in the voice information, an operation corresponding to the operation action is performed on the target operation object.

2. The method according to claim 1, characterized in that Before determining the attention area in the display screen, the method further includes: If the attention position is located on the display screen, it is determined that the voice information is used to instruct the execution of an operation associated with the content displayed on the display screen.

3. The method according to claim 1 or 2, characterized in that: Determining the target user's attention position according to the visual information includes: determining, based on the visual information, whether the target user has taken an action of indicating the display screen; In the case where the target user performs an action of indicating the display screen, the attention position of the target user is determined from the display screen according to the indicating direction of the action.

4. The method according to claim 3, characterized in that The determining, based on the visual information, whether the target user performs an action of indicating the display screen includes: Determining action information of the target user according to the visual information; Determining, according to the action information, whether the action of the target user is an indication action on the display screen; and / or, Determining the posture information of the target user according to the visual information; It is determined, according to the gesture information, whether the gesture of the target user indicates that the target user has an indication action on the display screen.

5. The method according to any one of claims 1 to 4, characterized in that: The performing an operation corresponding to the operation action on the target operation object based on the operation action indicated in the voice information includes: Based on the authority information of the target user, determining whether the target user has control authority over the target operation object; wherein the authority information corresponding to the target user is determined based on the seat position of the target user and / or the vehicle status of the target vehicle; If so, an operation corresponding to the operation action is performed on the target operation object.

6. The method according to claim 1, characterized in that The step of determining the attention area in the display screen according to the attention position includes: According to the attention position, determining display content associated with the attention position from the display screen; The attention area is determined according to a display position of the display content in the display screen.

7. The method according to any one of claims 1 to 6, characterized in that: The step of determining a target operation object associated with the voice information from the attention area according to the recognition result includes: Generate first feature data corresponding to the attention area, and generate second feature data of the semantics carried in the voice information; The first feature data and the second feature data are concatenated to obtain concatenated feature data, and the concatenated feature data is input into an attention coding model, and the concatenated feature data is encoded by the attention coding model to output target feature data; Based on the target feature data, an operation action of the target user and a target operation object corresponding to the operation action are determined; the operation object belongs to the attention area.

8. A vehicle-mounted device, characterized in that: Execute the interactive control method as described in any one of claims 1 to 7.

9. A vehicle, characterized in that: include: Comprising the vehicle-mounted device as claimed in claim 8.

10. A computer program product, characterized in that It comprises a computer program, which, when executed by a computer device or a processor, executes the steps of the interactive control method according to any one of claims 1 to 7.

Images