Vehicle control method and device, storage medium, controller and vehicle

Abstract

The invention relates to a vehicle control method and device, a storage medium, a controller and a vehicle, and the method comprises the steps: obtaining voice information in the vehicle, carrying out the semantic recognition processing of the voice information, determining an initial semantic instruction, carrying out the lip movement detection of a target user in the vehicle, and carrying out the lip movement detection. Obtaining a first detection result used for judging the validity of the initial semantic instruction, under the condition that the first detection result judges that the initial semantic instruction is valid, performing sight line detection on the target user to obtain a second detection result, and correcting the initial semantic instruction based on the second detection result to obtain a target semantic instruction; and finally executing the target semantic instruction to control the vehicle. Therefore, the voice control accuracy of the vehicle can be improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle control technology, and in particular to a vehicle control method, device, storage medium, controller and vehicle. Background Art

[0002] At present, the in-vehicle voice system usually has a long voice recognition function. The system can continuously recognize the user's voice for a certain period of time and execute operations corresponding to the voice commands to control the vehicle.

[0003] However, when the user is talking to others, the system is likely to misjudge the conversation content as a valid voice command, thereby mistakenly executing the voice command operation. In addition, it is difficult for the user to fully and accurately describe the control command for the vehicle through voice, so the accuracy of existing voice control of the vehicle is low. Summary of the invention

[0004] An embodiment of the present application provides a vehicle control method that can improve the accuracy of vehicle voice control to at least partially solve the above-mentioned technical problems.

[0005] In order to achieve the above object, according to a first aspect of the present application, a vehicle control method is provided, comprising: Acquiring voice information in the vehicle, and performing semantic recognition processing on the voice information to determine an initial semantic instruction; Performing lip movement detection on the target user in the vehicle to obtain a first detection result for determining the validity of the initial semantic instruction; When the first detection result determines that the initial semantic instruction is valid, performing a line of sight detection on the target user to obtain a second detection result, and modifying the initial semantic instruction based on the second detection result to obtain a target semantic instruction; The target semantic instruction is executed to control the vehicle.

[0006] Optionally, performing semantic recognition processing on the voice information to determine an initial semantic instruction includes: Performing noise reduction processing on the voice information to obtain first voice data after noise reduction; Performing endpoint detection processing on the first voice data to determine second voice data including a start time point and an end time point; Inputting the second voice data into the first processing model to perform text recognition on the second voice data to obtain text information; The text information is input into a second processing model for semantic recognition to obtain the initial semantic instruction.

[0007] Optionally, performing sight line detection on the target user to obtain a second detection result includes: Acquire the sight line information of the target user within a preset time range before and after the starting time point; The gaze target of the target user is determined based on the line of sight information to obtain the second detection result.

[0008] Optionally, determining the gaze target of the target user based on the line of sight information to obtain the second detection result includes: When the target user continues to gaze at the same vehicle-machine interaction object for more than a preset time within the preset time range, the target is determined to be a valid gaze target; When there is only one valid gaze target, determining the valid gaze target as the only gaze target of the target user; When all the vehicle-machine interaction objects that the target user continuously gazes at within the preset time range do not exceed the preset duration, or there are multiple valid gaze targets, the vehicle-machine interaction object or valid gaze target that the target user continuously gazes at for the longest duration is determined as the only gaze target of the target user.

[0009] Optionally, the method further comprises: When the target user continues to gaze at the same area where the vehicle-machine interaction object does not exist within the preset time range for more than the preset time length, determining whether there are other users in the area; If there are other users in the area, the second detection result is to determine that the initial semantic instruction is invalid; If there are no other users in the area, the vehicle-machine interaction object that the target user has been gazing at for the longest time is determined as the only gaze target of the target user to obtain the second detection result.

[0010] Optionally, the modifying the initial semantic instruction based on the second detection result to obtain a target semantic instruction includes: Acquire the unique gaze target determined based on the second detection result; The control object of the initial semantic instruction is set to the unique fixation target, and the target semantic instruction for controlling the unique fixation target is obtained.

[0011] Optionally, the performing lip movement detection on the target user in the vehicle to obtain a first detection result for determining the validity of the initial semantic instruction includes: Collecting image information containing the target user through a vehicle-mounted image acquisition device; Performing target recognition on the image information to obtain a lip image of the target user in the image information; Whether lip movement information exists in the image information of the target user is detected according to the lip area image of the target user, and a first detection result for determining the validity of the initial semantic instruction is obtained.

[0012] Optionally, the detecting whether the lip movement information exists in the image information of the target user according to the lip part image of the target user to obtain a first detection result for determining the validity of the initial semantic instruction further includes: Acquire at least two images of the lip area in the image information, and generate the first detection result according to the at least two images of the lip area in the image information.

[0013] Optionally, the performing lip movement detection on the target user in the vehicle to obtain a first detection result for determining the validity of the initial semantic instruction further includes: If the lip movement information exists in the image information of the target user, the first detection result determines that the initial semantic instruction is valid; If the lip movement information does not exist in the image information of the target user, the first detection result determines that the initial semantic instruction is invalid.

[0014] According to a second aspect of the present application, a vehicle control device is provided, comprising: A semantic acquisition module, used to acquire voice information in the vehicle and perform semantic recognition processing on the voice information to determine an initial semantic instruction; a lip movement detection module, configured to perform lip movement detection on a target user in the vehicle to obtain a first detection result for determining the validity of the initial semantic instruction; a semantic correction module, configured to, when the first detection result determines that the initial semantic instruction is valid, perform a sight line detection on the target user to obtain a second detection result, and correct the initial semantic instruction based on the second detection result to obtain a target semantic instruction; A vehicle control module is used to execute the target semantic instruction to control the vehicle.

[0015] According to a third aspect of the present application, a computer-readable storage medium is also provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described above are implemented.

[0016] According to a fourth aspect of the present application, a controller is also provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described above are implemented.

[0017] According to a fifth aspect of the present application, a vehicle is also provided, comprising the controller as described above.

[0018] According to a sixth aspect of the present application, a computer program product is also provided, comprising a computer program or instructions, which implement the steps of the method described above when executed by a processor.

[0019] In summary, the embodiment of the present application, through the above technical solution, first uses lip movement detection to determine whether the semantic instruction is issued by the target user, avoiding receiving and executing invalid semantic instructions issued by other users. After confirming that a valid semantic instruction has been received, the line of sight of the target user can be detected to accurately correct the actual execution target of the voice instruction, which can not only improve the accuracy and completeness of the semantic instruction description, but also improve the accuracy and safety of voice control of the vehicle based on accurate semantic instructions.

[0020] Other features and advantages of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0022] In order to more completely understand the present application and its beneficial effects, the following description will be given in conjunction with the accompanying drawings, wherein the same figure numbers represent the same parts in the following description.

[0023] Figure 1 is a flowchart of a vehicle control method provided in an exemplary embodiment of the present disclosure; Figure 2 is a flowchart of generating a target semantic instruction provided in an exemplary embodiment of the present disclosure; Figure 3 is a schematic diagram of a flow chart of determining a unique fixation target provided in an exemplary embodiment of the present disclosure; Figure 4 is a schematic diagram of a target user gazing at a target provided in an exemplary embodiment of the present disclosure; Figure 5 is a schematic diagram of a vehicle control device provided in an exemplary embodiment of the present disclosure; Figure 6 Schematic diagram of the architecture of a vehicle provided in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0025] As people pay more and more attention to environmental protection and low carbon, the pace of development of new energy vehicles has also accelerated significantly. The integration of automobiles with related technologies in the fields of energy, transportation, information and communications has accelerated, and electrification, networking and intelligence have become the development trend of the automobile industry. New technologies for new energy vehicles have sprung up, for example: The application number is CN202410658157.7, the publication number is CN118238797B, and the invention name is New Energy Vehicle Energy Intelligent Management System, Control Method and Related Equipment; The application number is CN202410672579.X, the publication number is CN118597091A, and the invention name is "New Energy Vehicle Energy Intelligent Management Method, System and Related Equipment"; The application number is 202410669449.0, and the invention name is New Energy Vehicle Energy Intelligent Management System, Control Method and Related Equipment; They all describe hybrid technology that is mainly electric, and has multiple advantages such as being fast, economical, quiet, smooth, and green.

[0026] The application number is CN202211678720.4, the publication number is CN117382629B, and the invention name is vehicle power control method, device, medium, vehicle controller and vehicle; The application number is CN202311164098.X, the publication number is CN116890770B, and the invention name is vehicle control system, method and vehicle; The application number is CN202311170393.6, the publication number is CN117533292B, and the invention name is vehicle control system, control method, controller and vehicle; Both describe a new energy power system with four wheel-side motors independently driven as the core, which greatly improves the safety and power of new energy vehicles.

[0027] Based on the problems mentioned in the above background technology, in the related technology, the application of intelligent voice systems in the vehicle environment has gradually become popular. Common intelligent voice systems have a continuous dialogue function, that is, after the user wakes up the voice system, the system will continue to monitor the user's voice and perform semantic analysis. If it is recognized that the content spoken by the user meets a preset instruction, the system will immediately respond and execute the command. This design can enhance the user experience in many scenarios, especially during driving, when users can control various functions of the vehicle system by voice without having to do anything, such as navigation, music playback, and in-car settings. Through continuous voice recognition, users can maintain a smooth conversation with the voice system without having to manually wake up the voice system every time.

[0028] However, in actual use, the voice recognition system is prone to misjudging the user's voice commands. Especially in the in-vehicle environment, users often have conversations with the co-pilot or rear passengers, or make phone calls. The voice recognition system may mistakenly recognize these voices that are not used for vehicle control as commands to the voice system, and thus perform irrelevant operations. Such misjudgments not only affect the user's driving experience, but may also cause safety hazards. Especially during driving, misoperation may cause unnecessary interference or accidents. In addition, in some scenarios, such as control functions such as voice navigation or automatic driving, it is difficult for users to fully and accurately express their control commands and intentions for the vehicle. For example, when describing the surrounding environment and orientation, traditional voice input methods are difficult to meet precise control requirements. Therefore, the accuracy of existing voice control of vehicles is low.

[0029] This application provides a vehicle control method, please refer to Figure 1 The vehicle control method provided in the embodiment of the present application includes steps 101 to 104, which are described in detail below.

[0030] Step 101: Acquire voice information in the vehicle, and perform semantic recognition processing on the voice information to determine initial semantic instructions.

[0031] In some embodiments, voice information emitted by users in the vehicle can be collected by an in-vehicle voice device or component. For example, a voice message "Please switch to night mode" emitted by a user to the central control screen can be captured.

[0032] In some embodiments, after acquiring the voice information, the voice information needs to be pre-processed by noise reduction, speech-to-text, semantic recognition, etc., so as to convert the voice information into semantic instructions that can interact with the vehicle, and then accurately control the vehicle based on the semantic instructions.

[0033] It should also be noted that in the vehicle voice control scenario of the present application, one type of scenario is that the user directly issues clear voice control to a certain in-vehicle interactive device. For example, the user issues a voice command "switch the central control screen to night mode" and directly obtains a semantic instruction to switch the central control screen to night mode.

[0034] The other type is a scenario where voice recognition and vehicle control are performed only for users with vehicle voice control permissions. For example, it is preset that only the driver can voice control the vehicle. When capturing the in-vehicle voice information and subsequently generating semantic instructions, it is necessary to determine whether the generated semantic instructions are issued by authorized users. And after determining that the semantic instruction is issued by permission, if the semantic instruction does not clearly control the object, for example, the semantic instruction is only "switch to night mode", it is necessary to further analyze which specific vehicle-mounted interactive device the semantic instruction is issued to, in order to ensure the accuracy of the semantic instruction itself and the accuracy of vehicle voice control. Therefore, the vehicle control method of the present application can be applied to the above two scenarios.

[0035] In some embodiments, step 101 may include: First, the voice information is subjected to noise reduction processing to obtain first voice data after noise reduction; Next, endpoint detection processing is performed on the first voice data to determine second voice data including a start time point and an end time point; Then, the second voice data is input into the first processing model to perform text recognition on the second voice data to obtain text information; Finally, the text information is input into the second processing model for semantic recognition to obtain initial semantic instructions.

[0036] like Figure 2 As shown, specifically, the received voice signal is first subjected to noise reduction processing to remove background noise and irrelevant sounds, such as background sound in the car, wind noise, traffic noise, etc. Through the noise reduction processing, the first voice data after noise reduction can be obtained, so that the subsequent voice recognition process is clearer and more accurate.

[0037] In some embodiments, the first voice data is processed using endpoint detection technology to identify the start time point and end time point of the voice and determine the valid voice segment. The purpose of endpoint detection is to identify the valid voice part in the voice signal and exclude the silent or meaningless part. Through endpoint detection processing, the second voice data obtained contains the start and end time points of the user's actual voice, and the second voice data is used as a more accurate voice segment for subsequent semantic analysis.

[0038] In some embodiments, the second voice data is input into the first processing model, and the first processing model may be an ASR model, that is, a speech recognition model. The ASR model may analyze the audio features in the second voice data, and convert the audio features into corresponding text information through text recognition processing.

[0039] In some embodiments, the extracted text information is input into a second processing model, which may be an NLU model, i.e., a natural language understanding model. The second processing model is then used to further process the text information, extract the intent and instructions in the text information, and understand the user's semantics. The purpose is to identify the intent that the user wants to express from the plain text and obtain the initial semantic instruction, which contains the specific vehicle control operation or request that the user wants the system to perform.

[0040] Through the above method, the present application can accurately recognize and understand the voice information sent by the user, and pre-process the voice information to obtain initial semantic instructions, providing accurate semantic support for subsequent voice command execution.

[0041] Step 102: Perform lip movement detection on the target user in the vehicle to obtain a first detection result for determining the validity of the initial semantic instruction.

[0042] The target user refers to a user who has the voice control authority for the vehicle, for example, the target user may be the driver of the vehicle. It can be understood that in the above-mentioned scenario where it is necessary to determine whether the semantic instruction is issued by the authorized user, after obtaining the initial semantic instruction, the instruction to perform lip movement detection on the target user is triggered to detect whether the initial semantic instruction comes from the voice information issued by the target user.

[0043] In some embodiments, step 102 may include: First, the image information containing the target user is collected by the vehicle-mounted image acquisition device; Next, target recognition is performed on the image information to obtain the lip area image of the target user in the image information; Finally, whether there is lip movement information in the image information of the target user is detected based on the lip area image of the target user, and a first detection result for determining the validity of the initial semantic instruction is obtained.

[0044] In some embodiments, the vehicle-mounted image acquisition device may include a camera or sensor installed in the vehicle, which is used to capture images of the vehicle environment in real time, and is mainly used to capture the facial images of users. The vehicle-mounted image acquisition device continuously monitors the faces of each user, and especially collects image information for the target user. The collected image information may include image data of the entire face or a part (such as the lip area), providing a basis for subsequent lip reading recognition and motion analysis. By collecting clear facial image data through the vehicle-mounted image acquisition device, it is possible to ensure that the image quality is sufficient to support lip movement detection and subsequent analysis.

[0045] In some embodiments, facial recognition and local feature recognition are performed on the target user. For example, through a face detection algorithm, the system can identify the facial area of ​​the target user from the entire image and further accurately extract the lip area.

[0046] Specifically, the face area can be located (such as using a Haar cascade classifier, a deep learning algorithm, etc.), and then the specific position of the lips can be further identified through a specific facial key point detection method. This can efficiently identify and track the lip area of ​​the target user in real-time images, especially when the target user's face is obstructed or the light changes, thereby providing key data for subsequent lip movement detection.

[0047] During the detection process, the lip movement detection algorithm can analyze the dynamic changes in the target user's lip area image, especially the movement of the lips. For example, the opening and closing, movement or shape changes of the lips are closely related to the process of pronunciation or vocalization. By extracting the movement features of the lips, the algorithm can determine whether there is an obvious lip movement pattern, indicating that the target user is making a sound or speaking, and further determine whether the initial semantic instruction issued by the target user is valid. Correspondingly, if lip movement information is detected, it means that the target user is speaking, and the system continues to perform speech recognition and semantic analysis; if no lip movement information is detected, it is considered that the initial semantic instruction is misidentified and the invalid initial semantic instruction is discarded.

[0048] In some embodiments, lip area images in at least two pieces of image information may be acquired, and a first detection result may be generated based on the lip area images in the at least two pieces of image information.

[0049] Specifically, after obtaining the lip position information in multiple images, these images can be dynamically analyzed. Each image represents the state of the lips at a point in time. By comparing the changes in the lips in these images, it is determined whether the user is speaking or issuing commands. For example, if the lips in two images show obvious opening and closing changes, the system infers that the target user may be speaking. On the contrary, if the lips remain still between the two images, it may indicate that the target user is not speaking.

[0050] By comparing the dynamic features of the lips in these images (such as opening width, lip shape changes, speed, etc.), a first detection result is generated, and the first detection result is used to determine the effectiveness of the voice command. For example, if the lip movements are shown in multiple images, it is considered that the target user is speaking and the voice command may be valid; if the lip movements are not obvious, it is considered that the target user is not speaking and the voice command may be invalid.

[0051] By using the lip movement changes in a time series, the user's actual voice activity can be captured more accurately, reducing misjudgments caused by small lip movements or facial occlusion in a single image.

[0052] Step 103: When the first detection result determines that the initial semantic instruction is valid, perform a line of sight detection on the target user to obtain a second detection result, and modify the initial semantic instruction based on the second detection result to obtain a target semantic instruction.

[0053] It can be understood that after detecting that the initial semantic instruction is issued by the target user, the initial semantic instruction is determined to be valid. However, if the initial semantic instruction does not clearly specify the in-vehicle interactive device to be controlled, for example, the initial semantic instruction is only "switch to night mode", it is necessary to further correct the initial semantic instruction to obtain a more accurate corrected semantic instruction.

[0054] In some embodiments, step 103 may include: Obtain the sight information of the target user within a preset time range before and after the starting time point; The gaze target of the target user is determined based on the line of sight information to obtain a second detection result.

[0055] like Figure 3 As shown, at the starting time point of the voice command, such as the moment when the target user sends a voice message, the line of sight information of the target user is traced back and obtained. The line of sight information refers to the target area where the target user's eyes are fixed within a preset time range before and after the voice command.

[0056] As an example only, the preset time range can be 2 seconds before and after the starting time point, that is, the target user's eye movement data can be collected within 2 seconds before and after the target user issues a voice command. In this time window, the system is ensured to capture the changes in the target user's eye gaze and determine the target of the target user's gaze when the command is issued.

[0057] In some embodiments, the line of sight information may include the gaze direction of the target user's eyes, the relationship between the eyes and the gaze object, such as Figure 4The target user's gaze is focused on the central control screen, instrument panel, HUD and other devices, as well as the duration of the gaze, etc. This information is captured in real time by an eye tracking device in the car (such as a DMS camera) to obtain the second detection result.

[0058] Through the above methods, the judgment method based on line of sight information in this application can significantly improve the accuracy of the vehicle voice control system of the vehicle in a complex driving environment, and enhance user experience and driving safety.

[0059] In some embodiments, the second detection result may be determined by: When the target user continuously gazes at the same vehicle-machine interaction object for more than a preset time within a preset time range, the target is determined to be a valid gaze target; When there is only one valid gaze target, the valid gaze target is determined as the only gaze target of the target user; When all vehicle-machine interaction objects that the target user continuously gazes at within a preset time range do not exceed a preset duration, or there are multiple valid gaze targets, the vehicle-machine interaction object or valid gaze target with the longest continuous gaze duration of the target user is determined as the only gaze target of the target user.

[0060] Among them, the vehicle-machine interaction object can be a vehicle-mounted device that can execute vehicle voice control commands, such as a central control screen, an instrument panel, and a HUD device. If the target user continues to stare at the same vehicle-machine interaction object within a preset time range for more than a preset time, for example, the preset time is 1 second), the system considers the gaze target to be valid and marks it as a valid gaze target. The valid gaze target is used for subsequent voice commands or operation judgments.

[0061] In some cases, the target user may only look at one vehicle-machine interaction object, and the system can directly identify the only valid gaze target. If the system detects only one vehicle-machine interaction object and the gaze time of the object exceeds the preset time (for example, the target user continuously looks at the central control screen for more than 2 seconds), the system will consider the object as the only gaze target of the target user.

[0062] In this case, the valid gaze target is directly determined as the only gaze target, and subsequent command execution will be performed around this target. For example, if the target user gazes at the central control screen and issues a voice command of "switch to night mode", the system automatically executes the command with the central control screen as the operation object, switching the central control screen to night mode.

[0063] It can be understood that if the target user does not gaze at a single target device for more than the preset time within the preset time range, for example, the target user disperses the gaze time on different vehicle-mounted devices and does not exceed the preset threshold, or there are multiple valid gaze targets, for example, the target user gazes at the central control screen and the instrument panel for more than the preset time, the system will make further judgments.

[0064] In this case, the system compares the duration of the gaze of multiple gaze targets and selects the target with the longest gaze duration as the only gaze target. For example, if the target user gazes at the central control screen for 2 seconds and the instrument panel for 1 second, the system can detect that the central control screen is the only gaze target of the target user.

[0065] If the gaze duration of multiple targets is similar or none of them significantly exceeds the others, the system can select the most suitable gaze target based on other logic (such as priority, device type, etc.). For example, the system can prioritize more important or more frequently interacted devices (such as the central control screen) as gaze targets.

[0066] In some embodiments, the method of the present application may further include: When the target user continues to gaze at the same area without the vehicle-machine interaction object for a preset time period, it is determined whether there are other users in the area; If there are other users in the area, the second detection result is to determine that the initial semantic instruction is invalid; If there are no other users in the area, the vehicle-machine interaction object with which the target user continuously gazes for the longest time is determined as the only gaze target of the target user, and a second detection result is obtained.

[0067] In some embodiments, the target user's line of sight within a preset time range can be monitored to confirm whether the target user is looking at an area in the vehicle without a vehicle-machine interaction object, such as the passenger seat, the back seat area, etc. These areas usually do not involve direct interaction with the vehicle-mounted device, so if the target user continues to look at these areas, it may be that he is talking with other passengers or looking at non-vehicle-machine related areas such as the back seat.

[0068] If it is detected that the target user continues to gaze at these non-interactive areas where there are no vehicle-machine interactive objects for longer than a preset time (for example, more than 2 seconds), it is necessary to further determine whether there are other users in the area, for example, the other users may be passengers sitting in the co-pilot seat or the back seat of the vehicle.

[0069] Specifically, the vehicle's cabin seat sensing information can be obtained first. The cabin seat sensing information can be obtained through sensors installed on the cabin seats, such as pressure sensors, capacitive sensors, etc. When someone sits on a seat, the weight of the human body will exert pressure on the seat surface. The pressure sensor can sense the change in pressure and convert it into an electrical signal. For example, in the co-pilot seat of a vehicle, the pressure sensors can be distributed on the seat cushion and backrest. When the pressure value exceeds a certain threshold, it can be determined that there is someone in the co-pilot seat. The capacitive sensor uses the change in capacitance between the human body and the sensor to detect whether there is someone. When the human body approaches the sensor, it will change the electric field distribution around the sensor, thereby causing the capacitance to change, and it can be determined whether there is someone in the seat area.

[0070] Therefore, if the cockpit seat sensing information is combined with the presence of other users in the area, it means that the target user is most likely talking to other users rather than interacting with the vehicle system, and it is necessary to determine whether to execute the voice command based on this information.

[0071] It can be understood that if it is detected that the target user is looking at the co-pilot area or the back seat area, and there are other passengers in the area (such as the co-pilot or the back seat passengers), it is inferred that the target user may not be interacting with the vehicle system, but communicating with other passengers. In this case, the voice command in the gaze area is judged to be invalid and discarded, because the target user's attention is not focused on the vehicle interaction device, but on the conversation with other passengers, avoiding the wrong execution of the command. For example, if the target user speaks to the co-pilot, the voice command will not be recognized as a command, but will be regarded as an instruction or conversation to the co-pilot. Therefore, the second detection result of the system is to determine that the initial semantic command is invalid to avoid misresponding to the voice from other passengers.

[0072] Correspondingly, if the system detects that the target user is looking at the co-pilot or rear seat area, but there are no other passengers in that area (for example, there is no one in the co-pilot seat), it indicates that the target user's gaze behavior is more likely to be unrelated to the interaction with the vehicle system, but the voice information it sends may still have other important factors affecting the command judgment, and the voice recognition judgment logic for the entire vehicle cabin will be further executed.

[0073] In this case, the target user's true intention will be judged based on other vehicle-machine interaction objects that the target user is looking at, such as the central control screen, instrument panel, HUD, etc. If the target user looks at these interactive targets for a long time, the system will determine that the gaze target is the only gaze target of the target user. By comparing the continuous gaze duration of all vehicle-machine interaction objects, the vehicle-machine interaction target with the longest gaze duration is selected, that is, the interactive device that the target user pays the most attention to. For example, if the target user looks at the central control screen for 2 seconds and the instrument panel for 1 second, the central control screen can be determined as the only gaze target and the relevant commands can be executed.

[0074] In this case, the second detection result selects the vehicle-machine interaction object with the longest gaze time as the only gaze target, ensuring that the voice command is only executed when the target user explicitly gazes at the relevant device, so as to achieve the purpose of semantic correction of the initial semantic command.

[0075] In some embodiments, the target semantic instruction is obtained by: Acquire a unique fixation target determined based on the second detection result; The control object of the initial semantic instruction is set as the only fixation target, and the target semantic instruction for controlling the only fixation target is obtained.

[0076] Specifically, the control object of the initial semantic instruction is associated with the gaze target. For example, if the target user says "switch to night mode" and is looking at the central control screen, the system will combine the "switch to night mode" instruction with the central control screen and determine that the operation object is the central control screen.

[0077] Through this association, a more contextually meaningful target semantic instruction is generated, which not only contains the original operation intention, such as "switch to night mode", but also clarifies the only gaze target to be controlled, such as the central control screen. The target semantic instruction is the final control command, and the system will perform specific operations according to this instruction to correct the initial semantic instruction and obtain a more accurate target semantic instruction.

[0078] Step 104: Execute the target semantic instruction to control the vehicle.

[0079] Specifically, after generating the target semantic instruction, the system transmits the target semantic instruction to the vehicle control system to perform the corresponding operation. For example, if the target semantic instruction is "switch the central control screen to night mode", the vehicle control system controls the display mode of the central control screen to switch to night mode, so as to achieve high-precision voice control of the vehicle based on the semantically accurate target semantic instruction.

[0080] In summary, the embodiment of the present application, through the above technical solution, first uses lip movement detection to determine whether the semantic instruction is issued by the target user, so as to avoid receiving and executing invalid semantic instructions issued by other users. After confirming that a valid semantic instruction has been received, the line of sight of the target user is detected to accurately correct the actual execution target of the voice instruction, thereby not only improving the accuracy and completeness of the semantic instruction description, but also improving the accuracy and safety of voice control of the vehicle based on accurate semantic instructions.

[0081] Figure 5 is a schematic diagram of the structure of a vehicle control device provided in an embodiment of the present application. Figure 5 , the vehicle control device may include a semantic acquisition module 201, a lip movement detection module 202, a semantic correction module 203 and a vehicle control module 204. The semantic acquisition module 201 is used to acquire voice information in the vehicle, and perform semantic recognition processing on the voice information to determine the initial semantic instruction; the lip movement detection module 202 is used to perform lip movement detection on the target user in the vehicle to obtain a first detection result for determining the validity of the initial semantic instruction; the semantic correction module 203 is used to perform line of sight detection on the target user to obtain a second detection result when the first detection result determines that the initial semantic instruction is valid, and correct the initial semantic instruction based on the second detection result to obtain a target semantic instruction; the vehicle control module 204 is used to execute the target semantic instruction to control the vehicle.

[0082] Among them, the semantic acquisition module 201, the lip movement detection module 202, the semantic correction module 203 and the vehicle control module 204 can be respectively used to execute steps 101-104 in the embodiments corresponding to the above-mentioned vehicle control method. For the specific implementation methods of these modules and more details, please refer to the corresponding method part, which will not be repeated here.

[0083] An embodiment of the present application also provides a computer-readable storage medium, on which instructions are stored. When the instructions are executed by a processor, the processor is configured to execute the above-mentioned vehicle control method.

[0084] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0085] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0086] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0087] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0088] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0089] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0090] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated communication signals and carrier waves.

[0091] like Figure 6 , which is a schematic diagram of the architecture of a vehicle provided in an embodiment of the present application. In this embodiment, the vehicle 400 may include a controller 300, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above vehicle control method are implemented. In this embodiment, the vehicle may be a fuel vehicle, a plug-in hybrid vehicle, or a new energy vehicle, etc., and this disclosure does not specifically limit this.

[0092] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0093] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0094] The embodiments, implementation methods and related technical features of the present application can be combined and replaced with each other without conflict.

[0095] The above are only preferred embodiments of the present application and do not constitute any form of limitation to the present application. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application are still within the scope of the technical solution of the present application.

Claims

1. A vehicle control method, characterized in that: include: Acquiring voice information in the vehicle, and performing semantic recognition processing on the voice information to determine an initial semantic instruction; Performing lip movement detection on the target user in the vehicle to obtain a first detection result for determining the validity of the initial semantic instruction; When the first detection result determines that the initial semantic instruction is valid, performing a line of sight detection on the target user to obtain a second detection result, and modifying the initial semantic instruction based on the second detection result to obtain a target semantic instruction; The target semantic instruction is executed to control the vehicle.

2. The method according to claim 1, characterized in that The performing semantic recognition processing on the voice information to determine the initial semantic instruction includes: Performing noise reduction processing on the voice information to obtain first voice data after noise reduction; Performing endpoint detection processing on the first voice data to determine second voice data including a start time point and an end time point; Inputting the second voice data into the first processing model to perform text recognition on the second voice data to obtain text information; The text information is input into a second processing model for semantic recognition to obtain the initial semantic instruction.

3. The method according to claim 2, characterized in that Performing line of sight detection on the target user to obtain a second detection result includes: Acquire the sight line information of the target user within a preset time range before and after the starting time point; The gaze target of the target user is determined based on the line of sight information to obtain the second detection result.

4. The method according to claim 3, characterized in that The determining the gaze target of the target user based on the line of sight information to obtain the second detection result includes: When the target user continues to gaze at the same vehicle-machine interaction object for more than a preset time within the preset time range, the target is determined to be a valid gaze target; When there is only one valid gaze target, determining the valid gaze target as the only gaze target of the target user; When all the vehicle-machine interaction objects that the target user continuously gazes at within the preset time range do not exceed the preset duration, or there are multiple valid gaze targets, the vehicle-machine interaction object or valid gaze target that the target user continuously gazes at for the longest duration is determined as the only gaze target of the target user.

5. The method according to claim 4, characterized in that The method further comprises: When the target user continues to gaze at the same area where the vehicle-machine interaction object does not exist within the preset time range for more than the preset time length, determining whether there are other users in the area; If there are other users in the area, the second detection result is to determine that the initial semantic instruction is invalid; If there are no other users in the area, the vehicle-machine interaction object that the target user has been gazing at for the longest time is determined as the only gaze target of the target user to obtain the second detection result.

6. The method according to any one of claims 1 to 5, characterized in that The step of modifying the initial semantic instruction based on the second detection result to obtain a target semantic instruction includes: Acquire the unique gaze target determined based on the second detection result; The control object of the initial semantic instruction is set to the unique fixation target, and the target semantic instruction for controlling the unique fixation target is obtained.

7. The method according to claim 1, characterized in that The performing lip movement detection on the target user in the vehicle to obtain a first detection result for determining the validity of the initial semantic instruction includes: Collecting image information containing the target user through a vehicle-mounted image acquisition device; Performing target recognition on the image information to obtain a lip image of the target user in the image information; Whether the lip movement information exists in the image information of the target user is detected according to the lip area image of the target user, and a first detection result for determining the validity of the initial semantic instruction is obtained.

8. The method according to claim 7, characterized in that The step of detecting whether the lip movement information exists in the image information of the target user according to the lip part image of the target user, and obtaining a first detection result for determining the validity of the initial semantic instruction, further includes: Acquire at least two images of the lip area in the image information, and generate the first detection result according to the at least two images of the lip area in the image information.

9. The method according to claim 7, characterized in that: The performing lip movement detection on the target user in the vehicle to obtain a first detection result for determining the validity of the initial semantic instruction also includes: If the lip movement information exists in the image information of the target user, the first detection result determines that the initial semantic instruction is valid; If the lip movement information does not exist in the image information of the target user, the first detection result determines that the initial semantic instruction is invalid.

10. A vehicle control device, characterized in that: include: A semantic acquisition module, used to acquire voice information in the vehicle and perform semantic recognition processing on the voice information to determine an initial semantic instruction; a lip movement detection module, configured to perform lip movement detection on a target user in the vehicle to obtain a first detection result for determining the validity of the initial semantic instruction; a semantic correction module, configured to, when the first detection result determines that the initial semantic instruction is valid, perform a sight line detection on the target user to obtain a second detection result, and correct the initial semantic instruction based on the second detection result to obtain a target semantic instruction; A vehicle control module is used to execute the target semantic instruction to control the vehicle.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A controller having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A vehicle, characterized in that: Comprising the controller as claimed in claim 12.

14. A computer program product, characterized in that The method comprises a computer program or instructions, which implement the steps of the method according to any one of claims 1 to 9 when executed by a processor.

Images