Semantic recognition-based cabin control method, apparatus and device, and storage medium

Abstract

The invention relates to the technical field of vehicle control, and discloses a cockpit control method, device and equipment based on semantic recognition and a storage medium, and the method comprises the steps: determining lip language recognition information corresponding to a user according to a target cockpit image; determining a target voice control text based on the lip language recognition information and the voice information of the user; determining a target control intention of the user based on the target voice control text and a preset cabin knowledge graph; and determining a to-be-executed operation of the vehicle cabin according to the target control intention, and controlling the vehicle cabin to execute the to-be-executed operation. The target voice control text is determined based on the lip language recognition information and the sound information of the user, the target control intention of the user is determined based on the target voice control text and the preset cabin knowledge graph, and then the vehicle cabin is controlled to execute the to-be-executed operation determined according to the target control intention. The problem that in the prior art, an intelligent cabin cannot accurately recognize the intention of a user, and consequently the cabin control precision is low is solved.

Description

Technical Field

[0001] The present invention relates to the field of vehicle control technology, and in particular to a cockpit control method, device, equipment and storage medium based on semantic recognition. Background Art

[0002] The vehicle cockpit is the part where the vehicle and the driver interact most closely. With the continuous development of Internet of Vehicles technology, smart cockpits have gradually become a part of people's diversified living space. Among them, voice interaction is an important way of interaction in the current smart cockpit scenario.

[0003] In existing smart cockpit scenarios, voice interaction usually involves the smart cockpit recognizing the user's intention based on the voice commands issued by the user. However, due to the complexity and diversity of the control targets of the smart cockpit, recognizing the user's intention through voice commands when the user's speech is generalized can easily lead to inaccurate recognition of the user's intention, which in turn leads to low cockpit control accuracy and affects the user experience.

[0004] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of the present invention is to provide a cockpit control method, device, equipment and storage medium based on semantic recognition, aiming to solve the technical problem in the prior art that recognizing user intentions through voice commands easily leads to inaccurate recognition of user intentions, which in turn leads to low cockpit control accuracy.

[0006] To achieve the above object, the present invention provides a cockpit control method based on semantic recognition, the method comprising:

[0007] Acquire a target cockpit image, and determine lip reading recognition information corresponding to the user according to the target cockpit image;

[0008] Determining a target voice control text based on the lip reading recognition information and the voice information of the user;

[0009] Determining the target control intention of the user based on the target voice control text and a preset cockpit knowledge graph;

[0010] An operation to be performed of a vehicle cockpit is determined according to the target control intention, and the vehicle cockpit is controlled to perform the operation to be performed.

[0011] Optionally, the step of determining lip reading recognition information corresponding to the user according to the target cockpit image includes:

[0012] Acquire a user's lip image based on the target cockpit image;

[0013] Performing feature extraction on the user's lip image to obtain lip image features corresponding to the user's lip image;

[0014] Obtaining a lip feature vector based on the lip image features, and determining a matching probability between the user lip image and preset semantic information according to the lip feature vector;

[0015] The target semantic information corresponding to the user's lip image is determined according to the matching probability to determine the lip reading recognition information corresponding to the user.

[0016] Optionally, the step of determining a target voice control text based on the lip reading recognition information and the voice information of the user includes:

[0017] Determine the user voice text based on the lip reading recognition information and the user's voice information;

[0018] Determining the target text category corresponding to the user voice text according to a preset vocabulary database, wherein the preset vocabulary database is a database storing a plurality of text categories and voice control vocabulary corresponding to each of the text categories;

[0019] A target voice control text is determined based on the target text category and the user voice text.

[0020] Optionally, the step of determining the user's target control intention based on the target voice control text and a preset cockpit knowledge graph includes:

[0021] Determine a current control scenario based on the target voice control text and a preset cockpit knowledge graph;

[0022] The target control intention of the user is determined based on the target voice control text and the current control scenario.

[0023] Optionally, the step of determining the target control intention of the user based on the target voice control text and the current control scenario includes:

[0024] Acquire all control intentions corresponding to the current control scenario according to the preset cockpit knowledge graph;

[0025] Performing text matching between the target voice control text and the text corresponding to each control intention to obtain the intention matching degree corresponding to each control intention;

[0026] Determining the intention weight corresponding to each control intention based on the intention matching degree and the trigger frequency corresponding to each control intention;

[0027] The target control intention of the user is determined based on the intention weight.

[0028] Optionally, before the step of determining the operation to be performed of the vehicle cockpit according to the target control intention and controlling the vehicle cockpit to perform the operation to be performed, the method further includes:

[0029] Determining target voiceprint information corresponding to the sound information;

[0030] Comparing the target voiceprint information with all voiceprint information in a preset voiceprint database to obtain a comparison result, wherein the preset voiceprint database stores system voiceprint information of the vehicle-mounted system;

[0031] Determining whether the sound information is sent by the vehicle-mounted system according to the comparison result;

[0032] If not, determining the target sound zone to which the sound information belongs;

[0033] The step of determining the operation to be performed of the vehicle cockpit according to the target control intention, and controlling the vehicle cockpit to perform the operation to be performed includes:

[0034] An operation to be performed of a vehicle cockpit is determined based on the target control intention and the target sound zone, and the vehicle cockpit is controlled to perform the operation to be performed.

[0035] Optionally, the step of determining the operation to be performed of the vehicle cockpit based on the target control intention and the target sound zone, and controlling the vehicle cockpit to perform the operation to be performed includes:

[0036] Determining a to-be-controlled area of ​​the vehicle cabin according to the target sound zone;

[0037] An operation to be performed in the vehicle cockpit is determined based on the target control intention, and the area to be controlled performs the operation to be performed.

[0038] In addition, to achieve the above-mentioned purpose, the present invention also proposes a cockpit control device based on semantic recognition, the device comprising:

[0039] An image acquisition module, used to acquire a target cockpit image and determine lip reading recognition information corresponding to the user based on the target cockpit image;

[0040] A text determination module, used for determining a target voice control text based on the lip reading recognition information and the user's voice information;

[0041] An intention determination module, used to determine the user's target control intention based on the target voice control text and a preset cockpit knowledge graph;

[0042] The operation execution module is used to determine the operation to be executed of the vehicle cockpit according to the target control intention, and control the vehicle cockpit to execute the operation to be executed.

[0043] In addition, to achieve the above-mentioned purpose, the present invention also proposes a cockpit control device based on semantic recognition, the device comprising: a memory, a processor, and a cockpit control program based on semantic recognition stored in the memory and executable on the processor, the cockpit control program based on semantic recognition being configured to implement the steps of the cockpit control method based on semantic recognition as described above.

[0044] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a cockpit control program based on semantic recognition is stored. When the cockpit control program based on semantic recognition is executed by a processor, the steps of the cockpit control method based on semantic recognition as described above are implemented.

[0045] The present invention discloses the steps of collecting a target cockpit image, and determining lip reading recognition information corresponding to the user based on the target cockpit image; determining a target voice control text based on the lip reading recognition information and the user's voice information; determining the user's target control intention based on the target voice control text and a preset cockpit knowledge graph; determining the to-be-executed operation of the vehicle cockpit based on the target control intention, and controlling the vehicle cockpit to execute the to-be-executed operation; compared with the prior art in which the smart cockpit recognizes the user's intention based on the voice command issued by the user, which is prone to inaccurate recognition when the user's speech is generalized, the present invention determines the target voice control text based on the lip reading recognition information corresponding to the user and the user's voice information, and determines the user's target control intention based on the target voice control text and the preset cockpit knowledge graph, and then controls the vehicle cockpit to execute the to-be-executed operation determined according to the target control intention, thereby solving the technical problem in the prior art that recognizing the user's intention through voice commands is prone to inaccurate recognition of the user's intention, thereby leading to low cockpit control accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a structural diagram of a cockpit control device based on semantic recognition in a hardware operating environment involved in an embodiment of the present invention;

[0047] Figure 2 It is a flowchart of a first embodiment of a cockpit control method based on semantic recognition according to the present invention;

[0048] Figure 3 It is a flow chart of a second embodiment of a cockpit control method based on semantic recognition according to the present invention;

[0049] Figure 4 It is a schematic diagram of a cockpit knowledge graph in the second embodiment of the cockpit control method based on semantic recognition of the present invention;

[0050] Figure 5 It is a flowchart of a third embodiment of a cockpit control method based on semantic recognition according to the present invention;

[0051] Figure 6 This is a structural block diagram of the first embodiment of the cockpit control device based on semantic recognition of the present invention.

[0052] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0054] Reference Figure 1 , Figure 1 It is a structural diagram of a cockpit control device based on semantic recognition in the hardware operating environment involved in an embodiment of the present invention.

[0055] like Figure 1 As shown, the cockpit control device based on semantic recognition may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0056] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the cockpit control device based on semantic recognition, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0057] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a cockpit control program based on semantic recognition.

[0058] exist Figure 1In the cockpit control device based on semantic recognition shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the cockpit control device based on semantic recognition of the present invention can be set in the cockpit control device based on semantic recognition, and the cockpit control device based on semantic recognition calls the cockpit control program based on semantic recognition stored in the memory 1005 through the processor 1001, and executes the cockpit control method based on semantic recognition provided by the embodiment of the present invention.

[0059] The embodiment of the present invention provides a cockpit control method based on semantic recognition, referring to Figure 2 , Figure 2 It is a flowchart of the first embodiment of the cockpit control method based on semantic recognition of the present invention.

[0060] In this embodiment, the cockpit control method based on semantic recognition includes the following steps:

[0061] Step S10: collecting a target cockpit image, and determining lip reading recognition information corresponding to the user according to the target cockpit image.

[0062] It should be noted that the execution subject of the method of this embodiment can be a cockpit control device that controls the vehicle cockpit by recognizing the user's semantics, or other cockpit control systems that can achieve the same or similar functions and include the cockpit control device. Here, the cockpit control system based on semantic recognition (hereinafter referred to as the system) is used to specifically illustrate the cockpit control method based on semantic recognition provided in this embodiment and the following embodiments.

[0063] It should be understood that the target cockpit image may be an image of the vehicle cockpit area. In this embodiment, the cockpit video information may be collected by an image acquisition device in the vehicle, and the target cockpit image may be acquired based on the cockpit video information.

[0064] It is understandable that the lip reading recognition information may be semantic information of the user recognized from the image. In this embodiment, the user's speech content may be recognized through the lip features of the user in the target cockpit image to obtain the semantic information.

[0065] Furthermore, in order to obtain accurate lip reading recognition information and improve the recognition accuracy of the user's intention, the step of determining the lip reading recognition information corresponding to the user according to the target cockpit image includes:

[0066] S101: Acquire a user's lip image based on the target cockpit image.

[0067] It should be understood that the user lip image may be an image of the user's lip region. In this embodiment, the system may perform image recognition on the target cockpit image, thereby extracting the image of the user's lip region from the target cockpit image.

[0068] S102: Extract features of the user's lip image to obtain lip image features corresponding to the user's lip image.

[0069] It should be noted that the above-mentioned lip image features are the characteristics of the lip image, such as edge features, texture features, etc., which are not limited in this embodiment. In this embodiment, CNN (Convolutional Neural Networks) can be used to extract features of the user's lip image to obtain the lip image features in the user's lip image.

[0070] S103: Obtain a lip feature vector based on the lip image features, and determine a matching probability between the user lip image and preset semantic information according to the lip feature vector.

[0071] In this embodiment, the lip feature vector may be a vector used to characterize the lip features of the user. The system may encode the lip image features through a BLSTM (Bi-directional Long Short-Term Memor) network or a feature encoder to obtain a lip feature vector corresponding to the lip feature image.

[0072] It should be noted that the above-mentioned preset semantic information may be pre-set semantic information for controlling the cockpit, such as: making a call, lowering the media volume, adjusting the temperature inside the car, etc., and this embodiment does not limit this.

[0073] It can be understood that the above matching probability may be the probability of matching between the user's lip image and the preset semantic information.

[0074] S104: Determine target semantic information corresponding to the user's lip image according to the matching probability, so as to determine lip reading recognition information corresponding to the user.

[0075] In this embodiment, the target semantic information may be the semantic information with the highest matching degree with the user's lip features among the preset semantic information.

[0076] In practical applications, the system can first perform image recognition on the target cockpit image to obtain the image of the user's lip area and the user's lip image in the target cockpit image, and perform feature extraction on the user's lip image through a CNN network to obtain the lip image features in the user's lip image. The lip feature image can then be further feature encoded through a feature encoder to obtain a lip feature vector corresponding to the lip image features, and the matching probability between the user's lip image and all semantic information used to control the cockpit is calculated based on the lip feature vector. The semantic information with the highest matching degree with the user's lip features is then determined as the target semantic information based on the matching probability, and the lip reading recognition information corresponding to the user is then obtained.

[0077] Step S20: determining a target voice control text based on the lip reading recognition information and the user's voice information.

[0078] It should be noted that the above-mentioned sound information can be obtained by collecting sensors installed in the vehicle. In practical applications, the sensors in the vehicle can collect the audio signals of the user speaking in the vehicle in real time, and obtain the user's sound information based on the audio signals. Among them, this embodiment does not limit the language (such as English, Chinese, dialect, etc.) spoken by the user.

[0079] It should be noted that the above-mentioned target voice control text can be a voice text issued by the user for controlling the vehicle cabin, wherein the format of the target voice control text can be TXT format (Text, plain text), JSON format (JavaScript Object Notation, JS object notation), etc., and this embodiment does not impose any restrictions on this.

[0080] In a specific implementation, since the user may use a dialect to control the vehicle cabin through voice, if the corresponding voice control text is directly obtained through the user's voice information, it may cause inaccurate semantic recognition of the user. Therefore, in this embodiment, the user's voice information can be adjusted through the user's lip reading recognition information, and the target voice control text can be obtained through the adjusted voice information. At this time, the target voice control text obtained is more accurate, thereby improving the accuracy of user voice recognition.

[0081] Furthermore, in order to further improve the recognition accuracy of the user voice control text, the step S20 includes:

[0082] Step S201: Determine the user speech text based on the lip reading recognition information and the user's voice information.

[0083] It should be noted that the above-mentioned user voice text may be a voice text for controlling the vehicle cockpit obtained by preliminary recognition based on the user's lip reading recognition information and sound information.

[0084] Step S202: determining the target text category corresponding to the user speech text according to a preset vocabulary database, wherein the preset vocabulary database is a database storing a plurality of text categories and speech control vocabulary corresponding to each of the text categories.

[0085] It should be understood that the above-mentioned target text categories can be text categories used to characterize user control of different areas of the cabin, such as: text categories for controlling seats, text categories for controlling the temperature in the car, text categories for controlling media, etc. This embodiment does not limit this.

[0086] It can be understood that the above-mentioned preset vocabulary database can be a pre-constructed database storing the corresponding relationship between each text category and the voice control vocabulary, wherein the voice control vocabulary can be the vocabulary used to control the vehicle cabin.

[0087] Step S203: determining a target voice control text based on the target text category and the user voice text.

[0088] In a specific implementation, different control words can be set in advance for different areas in the vehicle cabin, such as the control words for seats can be adjusting seats, adjusting seats to X°, etc.; the control words for the temperature in the car can be hot, cold, too high temperature, etc.; the control words for media can be turning on the radio, playing music, etc. Then different text categories can be divided for different control areas in the cabin, and then a preset vocabulary database can be constructed based on the text categories corresponding to each control area and the corresponding voice control words. Since the user voice text determined by the system based on lip recognition information and the user's voice information may have semantic generalization, resulting in inaccurate voice text, in order to improve the accuracy of text recognition, the words in the user voice text can be matched with the preset vocabulary database to determine the target text category corresponding to the user voice text, thereby determining the cabin area that the user currently needs to control, and then the user voice text can be screened according to the voice control vocabulary corresponding to the target text category, and useless words in the user voice text can be eliminated to obtain the target voice control text.

[0089] Step S30: Determine the target control intention of the user based on the target voice control text and a preset cockpit knowledge graph.

[0090] It should be noted that the above-mentioned preset cockpit knowledge graph may be a knowledge map constructed based on control scenarios, user intentions and semantic slots, wherein the semantic slots may be word slots corresponding to user intentions.

[0091] It should be understood that the above-mentioned target control intention may be the user's intention to control the vehicle cabin, for example: turning on the air conditioner / ventilation, adjusting the seat, playing music, etc.

[0092] In this embodiment, several vehicle cockpit control scenarios, possible control intentions of the user in the current control scenario, and the control semantics corresponding to the control intentions can be pre-determined to construct a preset cockpit knowledge graph. Subsequently, after determining the user's target voice control text, the preset cockpit knowledge graph can be used to search for the target control intention corresponding to the target voice control text.

[0093] Step S40: determining the operation to be performed of the vehicle cockpit according to the target control intention, and controlling the vehicle cockpit to perform the operation to be performed.

[0094] It should be noted that the above-mentioned operations to be executed may be operations performed by the vehicle cockpit according to the user's intention. For example, if the user's intention is to play music, the operations to be executed in the vehicle cockpit may be to open the music player and automatically play the system playlist; if the user's intention is to turn on the air conditioner, the operations to be executed in the vehicle cockpit may be to turn on the air conditioner.

[0095] In a specific implementation, if the system determines that the user's target control intention is to play music, the vehicle-side system can compile the target control intention according to the vehicle-side system to obtain the action instruction program, and determine the application end as a music player according to the action execution program, so that the action execution program can be sent to the music player. After receiving the action execution program, the music player can execute the action execution program to play music.

[0096] The present embodiment discloses collecting a target cockpit image, and determining the lip reading recognition information corresponding to the user according to the target cockpit image; determining a target voice control text based on the lip reading recognition information and the user's voice information; determining the user's target control intention based on the target voice control text and a preset cockpit knowledge graph; determining the to-be-executed operation of the vehicle cockpit according to the target control intention, and controlling the vehicle cockpit to execute the to-be-executed operation; compared with the prior art in which the smart cockpit recognizes the user's intention according to the voice command issued by the user, which is prone to inaccurate recognition when the user's speech is generalized, the present embodiment determines the target voice control text based on the lip reading recognition information corresponding to the user and the user's voice information, and determines the user's target control intention based on the target voice control text and the preset cockpit knowledge graph, and then controls the vehicle cockpit to execute the to-be-executed operation determined according to the target control intention, thereby solving the technical problem in the prior art that recognizing the user's intention through voice commands easily leads to inaccurate recognition of the user's intention, thereby leading to low cockpit control accuracy.

[0097] refer to Figure 3 , Figure 3 It is a flow chart of the second embodiment of the cockpit control method based on semantic recognition of the present invention.

[0098] Based on the above first embodiment, in order to improve the recognition accuracy of the user's intention and thus realize the precise control of the vehicle cabin, in this embodiment, the step S30 includes:

[0099] Step S301: Determine the current control scenario based on the target voice control text and the preset cockpit knowledge graph.

[0100] It should be noted that the above current control scene may be the cockpit scene controlled by the user this time. In this embodiment, there are multiple control scenes in the cockpit of the vehicle, such as seats, air conditioning, media, etc., which are not limited in this embodiment.

[0101] Step S302: Determine the target control intention of the user based on the target voice control text and the current control scenario.

[0102] In this embodiment, after the system determines the current control scenario through the preset cockpit knowledge graph, it can match the target voice control text with all semantic slots (texts) under the current control scenario, and determine the user's target control intention based on the matching results.

[0103] Furthermore, the step S302 includes: obtaining all control intentions corresponding to the current control scenario according to the preset cockpit knowledge graph; performing text matching on the target voice control text with the text corresponding to each control intention to obtain the intention matching degree corresponding to each control intention; determining the intention weight corresponding to each control intention based on the intention matching degree and the trigger frequency corresponding to each control intention; and determining the target control intention of the user based on the intention weight.

[0104] It should be noted that the above intention matching degree may be the matching degree between the target voice control text and the text corresponding to each control intention.

[0105] It can be understood that the above trigger frequency may be the frequency at which the user triggers the cockpit control under various control intentions.

[0106] In this embodiment, the system can determine the intention weight corresponding to each control intention according to the intention matching degree and trigger frequency corresponding to each control intention. Specifically, the higher the intention matching degree and trigger frequency corresponding to the control intention, the higher the intention weight corresponding to the control intention. The system can pre-set corresponding weight coefficients for the intention matching degree and trigger frequency. After determining the intention matching degree and trigger frequency corresponding to each control intention, the intention weight corresponding to each control intention can be calculated based on the intention matching degree, trigger frequency, the first weight coefficient corresponding to the intention matching degree, and the second weight coefficient corresponding to the trigger frequency. Then, the system can determine the control intention with the highest intention weight as the target control intention.

[0107] In the specific implementation, refer to Figure 4 , Figure 4 FIG. 1 is a schematic diagram of a cockpit knowledge graph in the second embodiment of the cockpit control method based on semantic recognition of the present invention. Figure 4 As shown in Figure 1, the cockpit knowledge graph can include control scenarios (seats, air conditioning, media, etc.), user intent (ventilation, heating, playing music, turning on the radio, etc.) and semantic slots ( Figure 4 ). When determining the user's target control intention, first, all control intentions in the current control scenario can be obtained from the cockpit knowledge graph. If the current control scenario is media, the control intentions in the media scenario can be obtained: play music, turn on the radio, etc., and then the text corresponding to each control intention can be obtained (such as listening to songs, listening to music, turning up the volume, turning down the volume, turning off music, etc.), and the target voice control text can be matched with the text corresponding to each control intention to obtain the intention matching degree between the target voice control text and each control intention, and then the intention weight corresponding to each control intention can be determined in combination with the trigger frequency corresponding to each control intention. Finally, the control intention with the highest intention weight can be determined as the target control intention.

[0108] This embodiment determines the current control scenario based on the target voice control text and the preset cockpit knowledge graph, obtains all control intentions corresponding to the current control scenario according to the preset cockpit knowledge graph, and then performs text matching on the target voice control text with the text corresponding to each control intention to obtain the intention matching degree corresponding to each control intention, determines the intention weight corresponding to each control intention based on the intention matching degree and the trigger frequency corresponding to each control intention, and determines the user's target intention based on the intention weight, thereby improving the recognition accuracy of the user's intention and further improving the control accuracy of the vehicle cockpit.

[0109] refer to Figure 5 , Figure 5 It is a flowchart of the third embodiment of the cockpit control method based on semantic recognition of the present invention.

[0110] Based on the above embodiments, in order to accurately distinguish the source of the sound information and thus ensure accurate execution of the vehicle cabin control, in this embodiment, before step S40, the method further includes:

[0111] Step S40a: Determine the target voiceprint information corresponding to the sound information.

[0112] It should be noted that, in this embodiment, the target voiceprint information corresponding to the user's voice information can be obtained by performing voiceprint recognition on the user's voice information.

[0113] Step S40b: Compare the target voiceprint information with all voiceprint information in a preset voiceprint database to obtain a comparison result, wherein the preset voiceprint database stores system voiceprint information of the vehicle-mounted system.

[0114] It should be understood that the above-mentioned preset voiceprint database may be a database storing system voiceprint information of the vehicle-mounted system, wherein the system voiceprint information may be voiceprint information corresponding to the system sound of the vehicle-mounted system.

[0115] Step S40c: determining whether the sound information is issued by the vehicle-mounted system according to the comparison result.

[0116] Step S40d: If not, determining the target sound zone to which the sound information belongs.

[0117] It can be understood that the target sound zone may be a sound zone where voice control (or sound) is emitted.

[0118] In this embodiment, the vehicle cabin can be divided into four sound zones, including: the main driver's seat sound zone, the co-driver's seat sound zone, the main driver's seat rear sound zone, and the co-driver's seat rear sound zone. In practical applications, in order to prevent the sound emitted by the vehicle system from interfering with the control of the vehicle cabin, after obtaining the target voiceprint information corresponding to the sound information emitted by the user, the system can compare it with the voiceprint information of all vehicle systems in the preset voiceprint database to determine whether the current target voiceprint information is the voiceprint information corresponding to the vehicle system sound, and then determine whether the sound information is emitted by the vehicle system. If not, the target sound zone that emits the sound information can be determined.

[0119] In actual applications, when the vehicle-mounted system emits a system sound, the system sound may interfere with the control of the vehicle cabin. Therefore, in this embodiment, the voiceprint information corresponding to the sound information collected by the system can be compared with the system voiceprint information stored in the preset voiceprint database, and based on the comparison result, it can be determined whether the target voiceprint information belongs to the system voiceprint information. If not, it can be determined that the sound information is not the sound emitted by the system. At this time, the target sound zone to which the sound information belongs can be determined, thereby achieving accurate identification of the source of the sound information, thereby ensuring the accurate execution of subsequent vehicle cabin control.

[0120] Accordingly, step S40 includes:

[0121] Step S40 ′: determining an operation to be performed in the vehicle cockpit based on the target control intention and the target sound zone, and controlling the vehicle cockpit to perform the operation to be performed.

[0122] It can be understood that after determining the target sound zone corresponding to the sound information, the system can determine the operation to be performed in the vehicle cockpit based on the target control intention and the target sound zone.

[0123] Specifically, the step S40' includes: determining a to-be-controlled area of ​​the vehicle cockpit according to the target sound zone; determining an operation to be performed of the vehicle cockpit based on the target control intention, and executing the operation to be performed in the to-be-controlled area.

[0124] In a specific implementation, the system can determine the area to be controlled in the vehicle cabin according to the target sound zone, and determine the operation to be performed corresponding to the area to be controlled based on the target control intention, and then control the vehicle cabin to perform the operation to be performed in the area to be controlled. For example: if the target sound zone is the passenger seat sound zone, and the target control intention is seat heating, the area to be controlled can be determined as the passenger seat, and the operation to be performed corresponding to the passenger seat can be seat heating. Then the system can perform the operation to be performed on the passenger seat as seat heating, thereby achieving precise control of the vehicle cabin.

[0125] This embodiment compares the target voiceprint information corresponding to the sound information with all the voiceprint information of the vehicle system stored in a preset voiceprint database, and determines whether the sound information is issued by the vehicle system based on the comparison result. If not, the target sound zone to which the sound information belongs is determined, and then the to-be-executed operation of the vehicle cockpit is determined based on the target control intention and the target sound zone to control the vehicle cockpit to execute the to-be-executed operation, so as to accurately distinguish the source of the sound information and ensure the accurate execution of the vehicle cockpit control.

[0126] In addition, an embodiment of the present invention further proposes a storage medium, on which a cockpit control program based on semantic recognition is stored. When the cockpit control program based on semantic recognition is executed by a processor, the steps of the cockpit control method based on semantic recognition as described above are implemented.

[0127] Reference Figure 6 , Figure 6 This is a structural block diagram of the first embodiment of the cockpit control device based on semantic recognition of the present invention.

[0128] like Figure 6 As shown, the cockpit control device based on semantic recognition proposed in an embodiment of the present invention includes:

[0129] An image acquisition module 601 is used to acquire a target cockpit image and determine lip reading recognition information corresponding to the user according to the target cockpit image;

[0130] A text determination module 602, configured to determine a target voice control text based on the lip reading recognition information and the user's voice information;

[0131] An intention determination module 603 is used to determine the target control intention of the user based on the target voice control text and a preset cockpit knowledge graph;

[0132] The operation execution module 604 is used to determine the operation to be executed of the vehicle cockpit according to the target control intention, and control the vehicle cockpit to execute the operation to be executed.

[0133] Furthermore, the image acquisition module 601 is also used to obtain a user lip image based on the target cockpit image; perform feature extraction on the user lip image to obtain lip image features corresponding to the user lip image; obtain a lip feature vector based on the lip image features, and determine a matching probability between the user lip image and preset semantic information based on the lip feature vector; determine the target semantic information corresponding to the user lip image based on the matching probability to determine the lip reading recognition information corresponding to the user.

[0134] Furthermore, the text determination module 602 is also used to determine the user voice text based on the lip reading recognition information and the user's voice information; determine the target text category corresponding to the user voice text according to a preset vocabulary database, the preset vocabulary database is a database storing a number of text categories and voice control vocabularies corresponding to each of the text categories; determine the target voice control text based on the target text category and the user voice text.

[0135] The cockpit control device based on semantic recognition of the present embodiment discloses collecting a target cockpit image, and determining the lip reading recognition information corresponding to the user according to the target cockpit image; determining a target voice control text based on the lip reading recognition information and the user's voice information; determining the user's target control intention based on the target voice control text and a preset cockpit knowledge graph; determining the to-be-executed operation of the vehicle cockpit according to the target control intention, and controlling the vehicle cockpit to execute the to-be-executed operation; compared with the prior art in which the smart cockpit recognizes the user's intention according to the voice command issued by the user, which is prone to inaccurate recognition when the user's speech is generalized, the present embodiment determines the target voice control text based on the lip reading recognition information corresponding to the user and the user's voice information, and determines the user's target control intention based on the target voice control text and the preset cockpit knowledge graph, and then controls the vehicle cockpit to execute the to-be-executed operation determined according to the target control intention, thereby solving the technical problem in the prior art that recognizing the user's intention through voice commands easily leads to inaccurate recognition of the user's intention, thereby leading to low cockpit control accuracy.

[0136] Based on the above-mentioned first embodiment of the cockpit control device based on semantic recognition of the present invention, a second embodiment of the cockpit control device based on semantic recognition of the present invention is proposed.

[0137] In this embodiment, the intention determination module 603 is also used to determine the current control scenario based on the target voice control text and the preset cockpit knowledge graph; and determine the user's target control intention based on the target voice control text and the current control scenario.

[0138] Furthermore, the intention determination module 603 is also used to obtain all control intentions corresponding to the current control scenario according to the preset cockpit knowledge graph; perform text matching on the target voice control text with the text corresponding to each control intention to obtain the intention matching degree corresponding to each control intention; determine the intention weight corresponding to each control intention based on the intention matching degree and the trigger frequency corresponding to each control intention; and determine the user's target control intention based on the intention weight.

[0139] This embodiment determines the current control scenario based on the target voice control text and the preset cockpit knowledge graph, obtains all control intentions corresponding to the current control scenario according to the preset cockpit knowledge graph, and then performs text matching on the target voice control text with the text corresponding to each control intention to obtain the intention matching degree corresponding to each control intention, determines the intention weight corresponding to each control intention based on the intention matching degree and the trigger frequency corresponding to each control intention, and determines the user's target intention based on the intention weight, thereby improving the recognition accuracy of the user's intention and further improving the control accuracy of the vehicle cockpit.

[0140] Based on the above-mentioned device embodiments, a third embodiment of a cockpit control device based on semantic recognition of the present invention is proposed.

[0141] In this embodiment, the operation execution module 604 is also used to determine the target voiceprint information corresponding to the sound information; compare the target voiceprint information with all the voiceprint information in a preset voiceprint database to obtain a comparison result, wherein the preset voiceprint database stores the system voiceprint information of the vehicle-mounted system; determine whether the sound information is issued by the vehicle-mounted system according to the comparison result; if not, determine the target sound zone to which the sound information belongs; determine the operation to be executed in the vehicle cabin based on the target control intention and the target sound zone, and control the vehicle cabin to execute the operation to be executed.

[0142] Furthermore, the operation execution module 604 is further used to determine the area to be controlled of the vehicle cockpit according to the target sound zone; determine the operation to be executed of the vehicle cockpit based on the target control intention, and execute the operation to be executed in the area to be controlled.

[0143] This embodiment compares the target voiceprint information corresponding to the sound information with all the voiceprint information of the vehicle system stored in a preset voiceprint database, and determines whether the target voiceprint information has a matching relationship with these voiceprint information based on the comparison result. If not, the target user to which the target voiceprint information belongs is determined, and the sound zone corresponding to the audio signal emitted by the target user is determined as the target sound zone. Then, based on the target control intention and the target sound zone, the to-be-executed operation of the vehicle cockpit is determined to control the vehicle cockpit to execute the to-be-executed operation, so as to accurately distinguish the source of the sound information, thereby ensuring the accurate execution of the vehicle cockpit control.

[0144] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0145] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0147] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A cockpit control method based on semantic recognition, characterized in that: The method comprises: Acquire a target cockpit image, and determine lip reading recognition information corresponding to the user according to the target cockpit image; Determining a target voice control text based on the lip reading recognition information and the voice information of the user; Determining the target control intention of the user based on the target voice control text and a preset cockpit knowledge graph; An operation to be performed of a vehicle cockpit is determined according to the target control intention, and the vehicle cockpit is controlled to perform the operation to be performed.

2. The cockpit control method based on semantic recognition according to claim 1, characterized in that: The step of determining the lip reading recognition information corresponding to the user according to the target cockpit image comprises: Acquire a user's lip image based on the target cockpit image; Performing feature extraction on the user's lip image to obtain lip image features corresponding to the user's lip image; Obtaining a lip feature vector based on the lip image features, and determining a matching probability between the user lip image and preset semantic information according to the lip feature vector; The target semantic information corresponding to the user's lip image is determined according to the matching probability to determine the lip reading recognition information corresponding to the user.

3. The cockpit control method based on semantic recognition according to claim 1, characterized in that: The step of determining the target voice control text based on the lip reading recognition information and the user's voice information comprises: Determine the user voice text based on the lip reading recognition information and the user's voice information; Determining the target text category corresponding to the user voice text according to a preset vocabulary database, wherein the preset vocabulary database is a database storing a plurality of text categories and voice control vocabulary corresponding to each of the text categories; A target voice control text is determined based on the target text category and the user voice text.

4. The cockpit control method based on semantic recognition according to claim 1, characterized in that: The step of determining the user's target control intention based on the target voice control text and a preset cockpit knowledge graph includes: Determine a current control scenario based on the target voice control text and a preset cockpit knowledge graph; The target control intention of the user is determined based on the target voice control text and the current control scenario.

5. The cockpit control method based on semantic recognition according to claim 4, characterized in that: The step of determining the target control intention of the user based on the target voice control text and the current control scenario includes: Acquire all control intentions corresponding to the current control scenario according to the preset cockpit knowledge graph; Performing text matching between the target voice control text and the text corresponding to each control intention to obtain the intention matching degree corresponding to each control intention; Determining the intention weight corresponding to each control intention based on the intention matching degree and the trigger frequency corresponding to each control intention; The target control intention of the user is determined based on the intention weight.

6. The cockpit control method based on semantic recognition according to claim 1, characterized in that: Before the step of determining the operation to be performed of the vehicle cockpit according to the target control intention and controlling the vehicle cockpit to perform the operation to be performed, the method further includes: Determining target voiceprint information corresponding to the sound information; Comparing the target voiceprint information with all voiceprint information in a preset voiceprint database to obtain a comparison result, wherein the preset voiceprint database stores system voiceprint information of the vehicle-mounted system; Determining whether the sound information is sent by the vehicle-mounted system according to the comparison result; If not, determining the target sound zone to which the sound information belongs; The step of determining the operation to be performed of the vehicle cockpit according to the target control intention, and controlling the vehicle cockpit to perform the operation to be performed includes: An operation to be performed of a vehicle cockpit is determined based on the target control intention and the target sound zone, and the vehicle cockpit is controlled to perform the operation to be performed.

7. The cockpit control method based on semantic recognition according to claim 6, characterized in that: The step of determining the operation to be performed of the vehicle cockpit based on the target control intention and the target sound zone, and controlling the vehicle cockpit to perform the operation to be performed includes: Determining a to-be-controlled area of ​​the vehicle cabin according to the target sound zone; An operation to be performed in the vehicle cockpit is determined based on the target control intention, and the area to be controlled performs the operation to be performed.

8. A cockpit control device based on semantic recognition, characterized in that: The device comprises: An image acquisition module, used to acquire a target cockpit image and determine lip reading recognition information corresponding to the user based on the target cockpit image; A text determination module, used for determining a target voice control text based on the lip reading recognition information and the user's voice information; An intention determination module, used to determine the user's target control intention based on the target voice control text and a preset cockpit knowledge graph; The operation execution module is used to determine the operation to be executed of the vehicle cockpit according to the target control intention, and control the vehicle cockpit to execute the operation to be executed.

9. A cockpit control device based on semantic recognition, characterized in that: The device comprises: a memory, a processor, and a cockpit control program based on semantic recognition stored in the memory and executable on the processor, wherein the cockpit control program based on semantic recognition is configured to implement the steps of the cockpit control method based on semantic recognition as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a cockpit control program based on semantic recognition, and when the cockpit control program based on semantic recognition is executed by the processor, the steps of the cockpit control method based on semantic recognition as described in any one of claims 1 to 7 are implemented.

Images