Control method and device of in-vehicle equipment, electronic equipment and computer storage medium

Abstract

The invention relates to the field of intelligent control, in particular to an in-vehicle equipment control method and device, electronic equipment and a computer storage medium. The method comprises the following steps: detecting image data and voice data of an in-vehicle environment of a target vehicle; identifying the image data to obtain an indication direction sent by a user; if a plurality of indication directions exist, determining the confidence degree of each indication direction according to the target characteristics; the target feature comprises at least one of the matching degree of the indication direction and the sending position of the voice data, the indication mode of the indication direction and the indication accuracy of the indication direction; the indication mode comprises a hand orientation indication or a face orientation indication; determining a target indication direction from the plurality of indication directions according to the confidence; performing semantic analysis on the voice data to obtain a target state parameter; controlling the target equipment according to the target state parameter; the target equipment is equipment of the target vehicle in the target indication direction. According to the invention, the control precision and convenience of the in-vehicle equipment can be improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent control technology, and in particular to a control method, device, electronic device and computer storage medium for in-vehicle equipment. Background Art

[0002] With the continuous development of vehicle-related technologies, existing vehicles are equipped with voice interaction technology, and users can control in-vehicle devices by speaking corresponding voice commands. However, voice-based control methods generally require users to accurately speak the name of the electronic component they want to control and / or the state they want to control, otherwise the voice cannot be recognized, resulting in low accuracy and convenience of in-vehicle device control based on voice interaction.

[0003] In response to the above problems, the industry has proposed a solution to control in-vehicle devices based on voice and visual perception. For example, visual perception can be used to identify the user's actions, and the detected actions can be combined with voice commands to make an overall decision, so as to achieve the resolution of voice references that do not have clear entity intentions, such as "turn it on" and "turn it off".

[0004] However, the inventors have found that when the existing visual perception system recognizes multiple actions in the vehicle, it is difficult to determine the user's true control intention. Generally, an error message will be given directly, or the user will be required to manually confirm again, thereby affecting the convenience of controlling the in-vehicle equipment.

[0005] Therefore, a more accurate and convenient in-vehicle device control solution is needed. Summary of the invention

[0006] In view of this, the embodiments of the present application provide a control method, device, electronic device and computer storage medium for an in-vehicle device, which can solve the problem of poor control convenience of existing in-vehicle devices.

[0007] In a first aspect, an embodiment of the present application provides a method for controlling an in-vehicle device, comprising: Detecting image data and voice data corresponding to the interior environment of the target vehicle; Recognize the image data to obtain the indicated direction issued by the user included in the image data; When multiple indication directions are identified, the confidence of each indication direction is determined according to a preset target feature; wherein the target feature includes at least one of a matching degree of the indication direction with the position of the voice data, an indication method of the indication direction, and an indication accuracy corresponding to the indication direction; the indication method includes at least one of a hand orientation indication and a face orientation indication; Filtering the plurality of indication directions according to the confidence level to obtain a target indication direction; Performing semantic analysis on the speech data to obtain target state parameters; the target state parameters are used to characterize the target state to which the equipment in the target vehicle is to be controlled; The target device is controlled according to the target state parameter; the target device is the device of the target vehicle in the target indication direction.

[0008] By adopting the above technical scheme, when multiple indication directions are detected, the confidence of each indication direction is determined by at least one of the emission position matching degree between each indication direction and the voice data, the indication mode of the indication direction and the indication accuracy corresponding to the indication direction, wherein the emission position matching degree is based on the fact that users with control intentions are likely to emit voice and action control at the same time, thereby filtering out occasional action noise that does not correspond to the actual control intention, the indication mode is used to optimize user preferences and / or easier to distinguish indication modes, thereby improving the accuracy of the indication direction, and the indication accuracy is used to measure the standardization of the indication action corresponding to the indication direction and the degree of association between the indication direction and the controllable device in the target vehicle, thereby, according to the confidence determined by at least one of the above items, an indication direction that accurately represents the user's control intention and has the strongest executableness can be screened out, so that the control process with multiple indication directions does not require the user to manually confirm again or pause due to an error, which can improve the convenience of controlling the in-vehicle equipment.

[0009] In some embodiments, the method includes: determining the indication accuracy based on at least one of the length of the interval between the indication time point of the indication direction and the current time point, the duration of the indication action corresponding to the indication direction, the similarity between the indication action corresponding to the indication direction and a preset standard indication action, and the hit degree of the indication direction for the controllable device in the target vehicle.

[0010] By adopting the above technical scheme, whether the user actually has the control intention can be represented to a certain extent based on the duration of the indication action, the proximity of the occurrence time, the standard degree of the indication action and whether the indicated direction is aligned with the controllable device. Therefore, the above technical scheme enables the indication accuracy to more accurately represent the user's true control intention, thereby improving the accuracy of the subsequent confidence calculation based on the indication accuracy.

[0011] In some embodiments, when the indication accuracy is determined according to the interval length between the indication time point of the indication direction and the current time point, the indication accuracy is inversely proportional to the interval length; Or, when the indication accuracy is determined according to the duration of the indication action, the indication accuracy is proportional to the duration of the indication action; Or, when the indication accuracy is determined based on the similarity, the indication accuracy is proportional to the similarity; Or, when the indication accuracy is determined according to the hit degree, the indication accuracy is proportional to the hit degree.

[0012] By adopting the above technical scheme, based on the consistency between the user's indication diagram and the indication behavior, the user's indication control intention represented by the longer duration, the most recent occurrence, the most standard indication action and the most accurate alignment with the controllable device is the strongest. Therefore, the indication accuracy determined by the above scheme can more accurately represent the user's control intention.

[0013] In some embodiments, the method comprises: The indication accuracy is obtained by taking a weighted sum of at least two of the interval length between the indication time point of the indication direction and the current time point, the duration of the indication action corresponding to the indication direction, the similarity between the indication action corresponding to the indication direction and a preset standard indication action, and the hit degree of the indication direction for the controllable device in the target vehicle; wherein the weights corresponding to the hit degree, similarity, interval length and duration decrease in turn.

[0014] By adopting the above technical solution and combining the behavioral characteristics of users when expressing control intentions through indicating behaviors, the weights corresponding to the hit degree, similarity, interval length and duration are reduced in turn, thereby improving the accuracy of the indication accuracy obtained by weighted summation.

[0015] In some embodiments, the hit degree is used to characterize the degree of overlap between the position of the controllable device and the path where the indicated direction is located; the process of determining the hit degree includes: when there is overlap with multiple controllable devices on the path where the indicated direction is located, determining the hit degree according to the overlap corresponding to the alternative device; wherein the alternative device is the controllable device with the largest ratio between the corresponding overlap and a preset overlap threshold.

[0016] By adopting the above technical solution, when a user indicates a controllable device at a clear position, the overlap between the indicated direction and the controllable device is generally large, thereby distinguishing it from accidental actions that are unrelated to the user's indication intention. Therefore, the accuracy of the hit is determined based on the overlap, which can better represent the user's actual indication control intention.

[0017] In some embodiments, the method includes: for each of the indication directions, performing a weighted sum based on at least two of the issuance position matching degree, indication method and indication accuracy corresponding to the indication direction, to obtain the confidence corresponding to the indication direction; wherein the weight corresponding to the issuance position matching degree is greater than the weights corresponding to the indication method and the indication accuracy respectively, and / or the weight corresponding to the hand orientation indication is greater than the weight corresponding to the face orientation indication.

[0018] By adopting the above technical scheme, the accuracy of the confidence is improved by taking a weighted sum of at least two of the matching degree, the indication method and the indication accuracy. At the same time, the weight corresponding to the issuance position matching degree is set to be greater than the weights corresponding to the indication method and the indication accuracy respectively, and / or the weight corresponding to the hand orientation indication is set to be greater than the weight corresponding to the face orientation indication, thereby improving the accuracy of the confidence in reflecting the user's actual control intention, so that the probability that the screened target indication direction is the direction corresponding to the user's actual control intention is greater.

[0019] In some embodiments, the method comprises: The voice data is detected according to the target identification information to obtain a device identification detection result; the target identification information is identification information of a controllable device corresponding to the target vehicle; Determine the target device according to the device identification detection result; The target state parameter corresponding to the target device is determined according to an action feature of the indication action corresponding to the indication direction.

[0020] By adopting the above technical solution, considering that the user may correctly say the name of the device he wants to control, the target device can be identified from the voice data, and the user's potential effective control intention can be more likely to be excavated. At the same time, the target state parameters are determined by indicating the direction, so that the user does not need to accurately input the state parameters to be controlled, thereby further improving the convenience of controlling the in-vehicle equipment.

[0021] In a second aspect, an embodiment of the present application further provides a control device for an in-vehicle device, the control device for the in-vehicle device comprising: A detection module, used to detect image data and voice data corresponding to the target vehicle environment; an identification module, used to identify the image data and obtain the indicated direction issued by the user included in the image data; A determination module, configured to determine the confidence of each of the indication directions according to a preset target feature when a plurality of indication directions are identified; wherein the target feature includes at least one of a matching degree of an originating position between the indication direction and the voice data, an indication mode of the indication direction, and an indication accuracy corresponding to the indication direction; and the indication mode includes at least one of a hand orientation indication and a face orientation indication; A screening module, used for screening the plurality of indication directions according to the confidence level to obtain a target indication direction; An analysis module, used for performing semantic analysis on the speech data to obtain a target control parameter; the target state parameter is used to characterize the target state to which the equipment in the target vehicle is to be controlled; A control module is used to control a target device according to the target state parameter; the target device is a device of the target vehicle in the target indication direction.

[0022] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a processor and a memory, wherein the memory is used to store instructions, and the processor is used to call the instructions in the memory so that the electronic device executes the control method of the in-vehicle device as described in the first aspect.

[0023] In a fourth aspect, an embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores computer instructions. When the computer instructions are executed on a processor, the processing device executes the control method of the in-vehicle device as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The present invention is a flowchart of a method for controlling an in-vehicle device according to an embodiment of the present application.

[0025] Figure 2 The present invention is a flowchart of a method for controlling an in-vehicle device according to another embodiment of the present application.

[0026] Figure 3 The present invention is a schematic diagram of the structure of a control device for an in-vehicle device provided according to an embodiment of the present application.

[0027] Figure 4 The figure is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the implementation methods of the present application and the features in the implementation methods can be combined with each other without conflict.

[0029] In the following description, many specific details are set forth to facilitate a full understanding of the present application. The described implementations are only part of the implementations of the present application, rather than all the implementations.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0031] It should be further noted that, in this article, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0032] In this application, "at least one" means one or more, and "more" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0033] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0034] In related technologies, users can send different vehicle body electronic components and their operating parameters to the domain controller by speaking corresponding voice commands, and call the corresponding actuators through the bus to transmit electronic signals, thereby realizing voice control of the operation of vehicle body electronic components. However, the semantic definition of such voice commands often requires a complete sentence with a clear "single entity + operating parameter" verb-object format before natural language understanding and skill domain placement can be performed through the voice model. For example, "open the window", "move the seat back a little", "close the sunshade", etc., the user must accurately say the name of the electronic component, otherwise the voice cannot be recognized or the control command cannot be placed in the corresponding vertical domain, thereby making the voice interaction experience worse. Although voice can improve such problems by adopting model semantic generalization, it still requires users to accurately say a complete sentence voice command.

[0035] Although there are similar technologies that combine voice with visual perception to address the above issues, that is, the in-vehicle occupant behavior monitoring system identifies the user's behavior, maps the monitored user's hand direction to a certain body electronic component through an algorithm, and transmits the signal and combines it with the voice command to make an overall decision, thereby achieving the resolution of voice references such as "open it" and "close it" that do not have clear entity intentions. However, because the visual monitoring system may identify multiple hand directions of multiple people when it meets the threshold of its monitoring algorithm model, when multiple hand directions are monitored concurrently, such as the main driver points to the left window and says "open it", and the co-driver points to the right window, the user is often required to manually confirm the hand direction representing his actual control intention again, or it cannot be done, which results in the user's true intention of issuing the voice command not being executed in a timely and accurate manner, affecting the efficiency and convenience of vehicle voice interaction.

[0036] It can be seen from the above that in existing vehicle interaction scenarios, when there are multiple indication directions, it is impossible to accurately judge the user's control intention, thereby affecting the accuracy and convenience of in-vehicle device control.

[0037] In view of this, the embodiments of the present application provide a control method, device, equipment and computer storage medium for in-vehicle equipment, which can improve the accuracy and convenience of in-vehicle equipment control.

[0038] See also Figure 1 , is a flowchart of the steps of the control method of the in-vehicle device provided in an embodiment of the present application. The control method of the in-vehicle device provided in this embodiment is based on a preset controller, and the controller can be a central control system of the vehicle, etc. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0039] See also Figure 1 As shown, the control method of the in-vehicle device may include the following steps: Step 101: Detect image data and voice data corresponding to the interior environment of a target vehicle.

[0040] The image data of the interior environment of the target vehicle is collected by a preset visual sensor. Specifically, the image data may include images of the front and rear seats of the target vehicle. Correspondingly, the voice data is collected by a preset voice sensor. Specifically, the voice data may include human voice data in the interior environment of the vehicle. The detection of the image data and the voice data may be automatic detection or may be triggered based on a user's instruction.

[0041] Step 102: Identify the image data to obtain the indicated direction issued by the user included in the image data.

[0042] The image data is identified according to a preset target detection algorithm, and the information for indicating the direction sent by the user included in the image data is identified. Considering that there may be objects containing direction information such as arrows and signs in the target vehicle, the action of indicating the direction sent by the human body in the image data can be identified. It is understandable that the human body can indicate the direction through different parts of the body, such as by pointing the hand in a specific direction, turning the head in a specific direction, extending or facing a specific direction, etc. Therefore, in this embodiment, the direction indication includes hand direction indication, facial direction indication, and limb direction, etc.

[0043] Step 103: When multiple indication directions are identified, the confidence of each indication direction is determined respectively according to preset target features; wherein the target features include at least one of the matching degree of the issuance position between the indication direction and the voice data, the indication method of the indication direction and the indication accuracy corresponding to the indication direction; the indication method includes at least one of hand orientation indication and face orientation indication.

[0044] The confidence level of the indicated direction is used to represent the probability that the indicated direction represents the control intention of the user.

[0045] When identifying multiple indication directions, in order to automatically select the indication direction that best represents the user's control intention without the need for the user to confirm again, thereby improving the accuracy and convenience of in-vehicle device control, considering that users with control intentions are likely to issue voice and action controls at the same time, the confidence of the indication direction can be determined based on the matching degree of the issuing position between the indication direction and the voice data. Specifically, the higher the matching degree of the issuing position, the greater the confidence, thereby filtering out occasional action noise that does not correspond to the actual control intention or misidentified environmental noise.

[0046] Optionally, taking into account that the user may have a preferred indication method, such as the user is accustomed to indicating with the hand facing the indication direction rather than the face facing the indication direction, a specific indication method is more likely to correspond to the user's actual control intention than other indication methods. Therefore, the confidence level corresponding to the indication direction can be determined according to the type of indication method corresponding to the indication direction, wherein the confidence level corresponding to the indication direction under different indication methods can be determined according to the user preference level corresponding to the indication method, and the higher the user preference level, the greater the confidence level of the indication direction corresponding to the indication method.

[0047] Optionally, considering that a specific indication method is easier to distinguish the indicated content than other indication methods, such as hand orientation indication is easier to identify direction information based on the detection target than limb orientation indication, therefore, the confidence of the indication direction corresponding to the indication method can also be determined according to the difficulty of extracting the indication direction feature corresponding to the indication method. Specifically, the easier it is to extract the indication direction feature corresponding to the indication method or the greater the distinguishability of the extracted direction feature compared to the features in other methods, the greater the confidence of the indication direction corresponding to the indication method. It can be understood that when the user has no preference for the indication method, the preference is not obvious, or there is a lack of user preference data, the confidence of the indication direction corresponding to the indication method can be determined according to the difficulty of extracting the indication direction feature corresponding to the indication method.

[0048] Optionally, considering that when the user has an actual control intention, he will pay more attention and be more careful about the execution of the indicated behavior, and it can be expected that the user will perform a correct and effective indicated behavior. Therefore, the indicated direction corresponding to the control intention is generally more accurate. Accuracy can include that the indicated direction itself is clearer and easier to identify, and it can also include that the alignment between it and the controllable device in the target vehicle is greater. Therefore, in this embodiment, the confidence corresponding to the indicated direction can also be determined according to the indication accuracy of the indicated direction. The greater the indication accuracy, the greater the corresponding confidence. Specifically, the indication accuracy is used to measure the standard of the indicated action corresponding to the indicated direction and / or the alignment between the indicated direction and the controllable device in the target vehicle. Among them, considering that the indication instruction is ultimately aimed at being executable, therefore, the more accurate the alignment with the controllable device in the target vehicle and the closer the indicated action is to the preset standard action, the greater its accuracy.

[0049] Therefore, based on the confidence level determined by at least one of the above items, an indication direction that accurately represents the user's control intention and has the strongest executableness can be screened out, so that the control process with multiple indication directions does not require the user to manually confirm again or pause due to an error, which can improve the convenience of controlling the in-vehicle equipment.

[0050] In some embodiments, the process of determining the indication accuracy may include the following steps: The indication accuracy is determined based on at least one of the length of the interval between the indication time point of the indication direction and the current time point, the duration of the indication action corresponding to the indication direction, the similarity between the indication action corresponding to the indication direction and a preset standard indication action, and the hit degree of the indication direction for the controllable device in the target vehicle.

[0051] Specifically, based on the characteristics of the user's indication behavior, the user's most recent action is generally most strongly associated with his or her control intention. For example, after a user makes an indication, he or she will generally wait for the instruction to be executed and will not immediately make other similar indication actions. Therefore, the indication accuracy can be determined based on the interval length between the indication time point of the indicated direction and the current time point. Specifically, when determining the indication accuracy based on the interval length between the indication time point of the indicated direction and the current time point, the indication accuracy is inversely proportional to the interval length.

[0052] Optionally, based on the characteristics of the user's indication behavior, when the user expects his indication behavior to correspond to the execution result, the user's behavior generally lasts for a period of time, which is different from other occasional actions of the user such as scratching, touching the head or picking up objects. Therefore, the indication accuracy can be determined according to the duration of the indication action corresponding to the indication direction. Specifically, when the indication accuracy is determined according to the duration of the indication action, the indication accuracy is proportional to the duration of the indication action. It is understandable that, taking into account the user's habitual actions such as supporting the chin and holding objects, in some embodiments, the duration of the action can also be compared with a preset duration threshold interval, and the indication accuracy of the indication direction corresponding to the action duration that is too large or too small can be reduced.

[0053] Optionally, when the user expects that his indication behavior corresponds to the execution result, his action will generally be close to the commonly used standard indication action, such as the standard upward pointing hand direction, which is different from other actions of the user occasionally, such as scratching, touching the head or picking up objects. Therefore, the indication accuracy can be determined according to the similarity between the indication action corresponding to the indication direction and the preset standard indication action. Specifically, when the indication accuracy is determined according to the similarity, the indication accuracy is proportional to the similarity. Among them, the preset standard indication action can be a preset standard hand direction of the indication direction, a standard facial image of the indication direction, etc., such as a standard hand direction pointing to the left, a facial image when the face is facing to the right, etc.

[0054] Optionally, considering that when the user knows the location of the equipment in the vehicle, such as the location of controllable devices such as seats, windows, and sunshades, the indication will generally point to the location of the device he wants to control, and the probability that the user's behavior that does not represent the intention to control the device is aimed at the controllable device is relatively reduced. In other words, a standard and effective indication action is expected to be aimed at the controllable device, so the indication accuracy can be determined based on the hit degree of the indication direction for the controllable device in the target vehicle. Specifically, when the indication accuracy is determined based on the hit degree, the indication accuracy is proportional to the hit degree. It should be noted that the controllable device includes electronic components of the executable instructions corresponding to the target vehicle, and the present invention is not limited to this.

[0055] Based on the characteristic that there is consistency between the user's indication intention and the implementation of the indication behavior, the user's indication control intention represented by the indication action that lasts longer, occurs more recently, is indicated with the most standard indication action, and is most accurately aligned with the controllable device is the strongest. Therefore, the indication accuracy determined by the above scheme can more accurately represent the user's control intention, thereby improving the accuracy of the subsequent confidence calculation based on the indication accuracy.

[0056] In some embodiments, based on at least one of the length of the interval between the indication time point of the indication direction and the current time point, the duration of the indication action corresponding to the indication direction, the similarity between the indication action corresponding to the indication direction and a preset standard indication action, and the hit degree of the indication direction for the controllable device in the target vehicle, the process of determining the indication accuracy may also include: taking a weighted sum of at least two of the length of the interval between the indication time point of the indication direction and the current time point, the duration of the indication action corresponding to the indication direction, the similarity between the indication action corresponding to the indication direction and a preset standard indication action, and the hit degree of the indication direction for the controllable device in the target vehicle to obtain the indication accuracy; wherein the weights corresponding to the hit degree, similarity, interval length, and duration decrease in sequence.

[0057] In order to further improve the accuracy of indication in representing the actual control intention of the user, combined with the behavioral characteristics of the user when expressing the control intention through indication behavior, that is, first, the user should try to point to a clear controllable device, secondly, the action used by the user should be as close as possible to the conventional standard indication action, and thirdly, the user's action should be recent, and finally the user's action should last for a certain period of time. Therefore, in this embodiment, the weights corresponding to the hit degree, similarity, interval length and duration are reduced in turn, and the accuracy of the indication accuracy in representing the actual control intention of the user is improved by taking the weighted sum of at least two of the hit degree, similarity, interval length and duration.

[0058] Among them, the hit degree is used to characterize the overlap between the position of the controllable device and the path where the indicated direction is located; in some embodiments, the process of determining the hit degree includes: when there is overlap with multiple controllable devices on the path where the indicated direction is located, determining the hit degree according to the overlap corresponding to the alternative device; wherein the alternative device is the controllable device with the largest ratio between the corresponding overlap and the preset overlap threshold.

[0059] When a user indicates a controllable device at a clear position, the overlap between the indicated direction and the controllable device is generally large, while the probability of aiming an action unrelated to the user's indicating intention at the controllable device is small. Therefore, when the indicated direction overlaps with multiple controllable devices on the path, the hit degree is determined according to the overlap corresponding to the controllable device with the largest ratio between the overlap and a preset overlap threshold, which can better represent the user's actual indicating control intention through the hit degree.

[0060] Specifically, step 103 also includes: for each of the indication directions, performing a weighted sum according to at least two of the issuance position matching degree, indication method and indication accuracy corresponding to the indication direction, to obtain the confidence corresponding to the indication direction; wherein the weight corresponding to the issuance position matching degree is greater than the weights corresponding to the indication method and the indication accuracy respectively, and / or the weight corresponding to the hand orientation indication is greater than the weight corresponding to the face orientation indication.

[0061] The confidence level is calculated by taking a weighted sum of at least two of the matching degree, indication method and indication accuracy to improve the accuracy of the confidence level. At the same time, the weight corresponding to the matching degree of the emitted position is set to be greater than the weights corresponding to the indication method and the indication accuracy respectively, and / or the weight corresponding to the hand orientation indication is set to be greater than the weight corresponding to the face orientation indication. This improves the accuracy with which the confidence level reflects the actual control intention of the user, making it more likely that the screened target indication direction is the direction corresponding to the actual control intention of the user, thereby improving the accuracy of in-vehicle device control.

[0062] Step 104: Filter the multiple indication directions according to the confidence level to obtain a target indication direction.

[0063] Specifically, the indication direction with the highest confidence may be determined as the target indication direction.

[0064] Optionally, fitting may be performed on a plurality of indication directions with confidences greater than a confidence threshold to obtain a target indication direction.

[0065] Step 105: semantically analyzing the speech data to obtain target state parameters; the target state parameters are used to characterize the target state to which the equipment in the target vehicle is to be controlled.

[0066] Specifically, the voice data can be processed by natural language and converted into text data, and then the text data can be semantically analyzed to obtain the target state parameters. When performing semantic analysis, the text data can be identified according to the preset optional device state database, and the identified device state parameters can be determined as target state parameters. The optional device state database includes a plurality of optional parameters for characterizing the controllable states of the device, such as "open", "close", "large", "small", etc. For example, when the voice data is "open the window" or "open it", the target state parameter is open. Correspondingly, when the voice data is "move it forward 5 cm", the target state parameter is move forward 5 cm.

[0067] Step 106: Control the target device according to the target state parameter; the device of the target vehicle is in the target indication direction.

[0068] The controllable device of the target vehicle in the target indication direction is determined as the target device. Optionally, when there are multiple controllable devices in the target indication direction, a target device can be screened out based on the distance between the multiple controllable devices and the user, the user's historical control data, the user's control preferences and other information. For example, the controllable device closest to the user, the controllable device most recently used by the user, or the controllable device preferred by the user among the multiple controllable devices in the target indication direction can be determined as the target device. Optionally, when there are multiple controllable devices in the target indication direction, the degree of compatibility between the target state parameter and the multiple controllable devices can also be determined, and the controllable device with the highest degree of compatibility can be determined as the target device, wherein the degree of compatibility is used to characterize whether the target state parameter can be applied to the controllable device to control the controllable device.

[0069] See also Figure 2 , is a flowchart of the steps of a control method of an in-vehicle device provided by another embodiment of the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0070] See also Figure 2 As shown, the control method of the in-vehicle device may include the following steps: Step 201: Detect image data and voice data corresponding to a target vehicle.

[0071] Step 201 of the embodiment of the present application is similar to step 101 of the aforementioned embodiment, and will not be described again here to avoid repetition.

[0072] Step 202: Identify the image data to obtain the indicated direction included in the image data.

[0073] Step 202 of the embodiment of the present application is similar to step 102 of the aforementioned embodiment, and will not be described again here to avoid repetition.

[0074] Step 203: Detect the voice data according to the target identification information, and determine the target device according to the device identification detection result; the target identification information is the identification information of the controllable device corresponding to the target vehicle.

[0075] Among them, considering that the user may correctly say the name of the device he wants to control, the voice data can be identified according to the identification information of the controllable device corresponding to the target vehicle, and the device specified by the user through voice control can be identified, so as to better explore the user's potential and effective control intentions based on multimodal data.

[0076] Step 204: Determine the target state parameter corresponding to the target device according to the action characteristics of the indication action corresponding to the indication direction.

[0077] Specifically, the action features include the action trajectory features of the indicated action, and the action position, action change information, etc. of the indicated action are extracted according to the motion trajectory features, and the target state parameters are determined according to the action position and action change information. For example, when the voice data sent by the user is "raise the car window to this high", the height to which the car window needs to be raised is determined according to the height of his hand or elbow. For another example, when the voice data sent by the user is "take a picture of the scene I see", the shooting range can be determined according to the user's facial rotation range, and the image data within the user's field of vision can be captured through a camera, etc., thereby improving the efficiency and convenience of interactive control of in-vehicle devices.

[0078] Considering that users may have a low level of understanding of the control methods of certain in-vehicle devices and are unable to provide accurate target state parameters, such as the degree to which the user wants the device to be controlled or the standard name of the operation to be performed, the target state parameters are mined by indicating the direction in combination with the optional control state of the target device, thereby eliminating the need for users to accurately input the state parameters to be controlled, thereby further improving the convenience of controlling in-vehicle devices.

[0079] Please refer to Figure 3 , is a schematic diagram of the structure of the control device 301 of the in-vehicle device provided in the embodiment of the present application. Figure 3 As shown, the control device 301 of the in-vehicle equipment includes: A detection module 3001 is used to detect image data and voice data corresponding to the interior environment of a target vehicle; The recognition module 3002 is used to recognize the image data and obtain the indicated direction issued by the user included in the image data; The determination module 3003 is used to determine the confidence of each of the indication directions according to a preset target feature when multiple indication directions are identified; wherein the target feature includes at least one of the matching degree of the position of the indication direction and the voice data, the indication mode of the indication direction, and the indication accuracy corresponding to the indication direction; the indication mode includes at least one of the hand orientation indication and the face orientation indication; A screening module 3004 is used to screen the plurality of indication directions according to the confidence level to obtain a target indication direction; The analysis module 3005 is used to perform semantic analysis on the speech data to obtain target control parameters; the target state parameters are used to characterize the target state to which the equipment in the target vehicle is to be controlled; The control module 3006 is used to control the target device according to the target state parameter; the target device is the device of the target vehicle in the target indication direction.

[0080] Please refer to Figure 4 , is a schematic diagram of the hardware structure of the electronic device 401 provided in the embodiment of the present application. Figure 4 As shown, the electronic device 401 may include a processor 4001 and a memory 4002. The memory 4002 is used to store one or more computer programs 4003. The one or more computer programs 4003 are configured to be executed by the processor 4001. The one or more computer programs 4003 include instructions, and the above instructions can be used to implement the above-mentioned control method of the in-vehicle device in the electronic device 401.

[0081] It is understandable that the structure shown in this embodiment does not constitute a specific limitation on the electronic device 401. In other embodiments, the electronic device 401 may include more or fewer components than shown, or combine or separate some components, or arrange the components differently.

[0082] The processor 4001 may include one or more processing units, for example, the processor 4001 may include an application processor (AP), a modem, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0083] The processor 4001 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 4001 is a cache memory. The memory may store instructions or data that the processor 4001 has just used or circulated. If the processor 4001 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 4001, and thus improves the efficiency of the system.

[0084] In some embodiments, the processor 4001 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and / or a USB interface, etc.

[0085] In some embodiments, processor 4001 is used to execute acceleration schemes such as single instruction multiple data (SIMD) and very long instruction word (VLIW).

[0086] In some embodiments, memory 4002 may include high-speed random access memory and may also include non-volatile memory, such as a hard disk, an internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0087] This embodiment also provides a computer storage medium, in which computer instructions are stored. When the instructions are executed on a processor, the processor executes the above-mentioned related method steps to implement the control method of the in-vehicle device in the above-mentioned embodiment.

[0088] Among them, the electronic device, device, and computer storage medium provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0089] In practical applications, the above functions can be distributed to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0090] In several embodiments provided in the present application, the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are schematic. For example, the division of the modules or units is a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0091] The unit described as a separate component may or may not be physically separated, and the component shown as a unit may be one physical unit or multiple physical units, that is, it may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment.

[0092] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0093] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

[0094] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application.

Claims

1. A method for controlling an in-vehicle device, characterized in that: The control method of the in-vehicle device comprises: Detecting image data and voice data corresponding to the interior environment of the target vehicle; Recognize the image data to obtain the indicated direction issued by the user included in the image data; When multiple indication directions are identified, the confidence of each indication direction is determined according to a preset target feature; wherein the target feature includes at least one of a matching degree of the indication direction with the position of the voice data, an indication method of the indication direction, and an indication accuracy corresponding to the indication direction; the indication method includes at least one of a hand orientation indication and a face orientation indication; Filtering the plurality of indication directions according to the confidence level to obtain a target indication direction; Performing semantic analysis on the speech data to obtain target state parameters; the target state parameters are used to characterize the target state to which the equipment in the target vehicle is to be controlled; The target device is controlled according to the target state parameter; the target device is the device of the target vehicle in the target indication direction.

2. The method according to claim 1, characterized in that The process of determining the confidence level includes: The indication accuracy is determined based on at least one of the length of the interval between the indication time point of the indication direction and the current time point, the duration of the indication action corresponding to the indication direction, the similarity between the indication action corresponding to the indication direction and a preset standard indication action, and the hit degree of the indication direction for the controllable device in the target vehicle.

3. The method according to claim 2, characterized in that The process of determining the indication accuracy further includes: When the indication accuracy is determined according to the interval length between the indication time point of the indication direction and the current time point, the indication accuracy is inversely proportional to the interval length; Or, when the indication accuracy is determined according to the duration of the indication action, the indication accuracy is proportional to the duration of the indication action; Or, when the indication accuracy is determined based on the similarity, the indication accuracy is proportional to the similarity; Or, when the indication accuracy is determined according to the hit degree, the indication accuracy is proportional to the hit degree.

4. The method according to claim 2, characterized in that: The process of determining the indication accuracy includes: The indication accuracy is obtained by taking a weighted sum of at least two of the interval length between the indication time point of the indication direction and the current time point, the duration of the indication action corresponding to the indication direction, the similarity between the indication action corresponding to the indication direction and a preset standard indication action, and the hit degree of the indication direction for the controllable device in the target vehicle; wherein the weights corresponding to the hit degree, similarity, interval length and duration decrease in turn.

5. The method according to claim 2, characterized in that: The hit degree is used to characterize the degree of overlap between the position of the controllable device and the path where the indicated direction is located; The process of determining the hit degree includes: when there is overlap with multiple controllable devices on the path where the indicated direction is located, determining the hit degree according to the overlap corresponding to the alternative device; wherein the alternative device is the controllable device with the largest ratio between the corresponding overlap and a preset overlap threshold.

6. The method for controlling an in-vehicle device according to claim 1, characterized in that: The process of determining the confidence level includes: For each of the indication directions, a weighted sum is performed based on at least two of the issuance position matching degree, indication method and indication accuracy corresponding to the indication direction to obtain the confidence corresponding to the indication direction; wherein the weight corresponding to the issuance position matching degree is greater than the weights corresponding to the indication method and the indication accuracy respectively, and / or the weight corresponding to the hand orientation indication is greater than the weight corresponding to the face orientation indication.

7. The method for controlling an in-vehicle device according to claim 1, characterized in that: The method further comprises: The voice data is detected according to the target identification information, and the target device is determined according to the device identification detection result; the target identification information is the identification information of the controllable device corresponding to the target vehicle; The target state parameter corresponding to the target device is determined according to an action feature of the indication action corresponding to the indication direction.

8. A control device for in-vehicle equipment, characterized in that: The control device of the in-vehicle equipment comprises: A detection module, used to detect image data and voice data corresponding to the interior environment of the target vehicle; an identification module, used to identify the image data and obtain the indicated direction issued by the user included in the image data; A determination module, configured to determine the confidence of each of the indication directions according to a preset target feature when a plurality of indication directions are identified; wherein the target feature includes at least one of a matching degree of an originating position between the indication direction and the voice data, an indication mode of the indication direction, and an indication accuracy corresponding to the indication direction; and the indication mode includes at least one of a hand orientation indication and a face orientation indication; A screening module, used for screening the plurality of indication directions according to the confidence level to obtain a target indication direction; An analysis module, used for performing semantic analysis on the speech data to obtain a target control parameter; the target state parameter is used to characterize the target state to which the equipment in the target vehicle is to be controlled; A control module is used to control a target device according to the target state parameter; the target device is a device of the target vehicle in the target indication direction.

9. An electronic device, comprising a processor and a memory, characterized in that: The memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the control method of the in-vehicle device according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The computer storage medium stores computer instructions, and when the computer instructions are executed on a processor, the processor is caused to execute the control method of the in-vehicle device according to any one of claims 1 to 7.

Images