Vehicle-machine interaction method and device, vehicle, electronic equipment and storage medium

Abstract

The invention provides a vehicle-machine interaction method and device, a vehicle, electronic equipment and a storage medium, and relates to the technical field of vehicles, and the main technical scheme comprises the following steps: obtaining a sign language image of vehicle control by a user; identifying the sign language image to obtain corresponding vehicle control information; the corresponding vehicle control is executed according to the vehicle control information, compared with the related technology, the sign language image is recognized, the corresponding vehicle control information is obtained, the corresponding vehicle control is executed, and people with hearing and speaking obstacles can conduct vehicle interaction in a voice interaction mode.

Description

Technical Field

[0001] The present disclosure relates to the field of vehicle technology, and in particular to a vehicle-computer interaction method and device, a vehicle, an electronic device, and a storage medium. Background Art

[0002] With the development of technology, more and more functions are provided on vehicles and other equipment, which not only facilitate driving but also entertainment. In the development of vehicles, vehicle interaction is increasingly valued. Among them, voice interaction, as one of the convenient interaction methods, plays a vital role in vehicle interaction.

[0003] However, the voice interaction method in the related technology cannot be used by people with hearing and speech impairments. How to enable people with hearing and speech impairments to interact with vehicles through voice interaction is a problem that needs to be solved urgently. Summary of the invention

[0004] The present disclosure provides a vehicle-machine interaction method and device, a vehicle, an electronic device and a storage medium, the main purpose of which is to enable people with hearing and speech impairments to interact with the vehicle through voice interaction.

[0005] According to a first aspect of the present disclosure, a vehicle-computer interaction method is provided, comprising:

[0006] Acquire the user's sign language image of the vehicle operation;

[0007] Recognizing the sign language image to obtain corresponding vehicle control information;

[0008] Execute corresponding vehicle manipulation according to the vehicle manipulation information.

[0009] Optionally, obtaining a sign language image of the user controlling the vehicle includes:

[0010] Extract features from the collected images inside the vehicle;

[0011] If the key features of the sign language image are extracted, the sign language image is obtained from the collected in-vehicle image.

[0012] Optionally, the step of recognizing the sign language image to obtain corresponding vehicle control information includes:

[0013] Recognize the sign language image as language text by a sign language recognition algorithm, wherein the sign language recognition algorithm is trained based on sign language data;

[0014] The language text is semantically analyzed to obtain corresponding vehicle control information.

[0015] Optionally, the training method of the sign language recognition algorithm includes:

[0016] Acquire a preset number of training sign language images and training language texts corresponding to the training sign language images, wherein the sign language data includes the training sign language images and the corresponding training language texts;

[0017] Inputting the training sign language image and the training language text into the sign language recognition algorithm to obtain training recognition information;

[0018] Based on the training recognition information, a loss value of the sign language recognition algorithm is calculated by a preset loss function, wherein the loss value is a value that measures the degree of difference between the predicted translation information and the actual translation information of the sign language recognition algorithm;

[0019] Based on the loss value, the sign language recognition algorithm is optimized by a preset optimization algorithm to obtain a trained sign language recognition algorithm.

[0020] Optionally, performing corresponding vehicle manipulation according to the vehicle manipulation information includes:

[0021] Acquiring seat information of the user in the vehicle;

[0022] Determining whether the vehicle control information exceeds a preset control authority according to the seat information;

[0023] If the vehicle control information exceeds the preset control authority, corresponding vehicle control is performed according to the vehicle control information.

[0024] Optionally, the step of recognizing the sign language image to obtain corresponding vehicle control information includes:

[0025] Comparing the sign language image with a preset gesture code to determine corresponding gesture code information;

[0026] Determining corresponding vehicle control information according to the gesture coding information;

[0027] Wherein, each of the gesture codes corresponds to a type of vehicle control information.

[0028] According to a second aspect of the present disclosure, a vehicle-machine interaction device is provided, comprising:

[0029] A first acquisition unit is used to acquire a sign language image of the user controlling the vehicle;

[0030] A recognition unit, used to recognize the sign language image to obtain corresponding vehicle control information;

[0031] An execution unit is used to execute corresponding vehicle manipulation according to the vehicle manipulation information.

[0032] Optionally, the acquiring unit includes:

[0033] An extraction module, used for extracting features from the collected vehicle interior images;

[0034] An acquisition module is used to acquire the sign language image from the collected in-vehicle image when the key features of the sign language image are extracted.

[0035] Optionally, the identification unit includes:

[0036] A recognition module, used to recognize the sign language image as language text by using a sign language recognition algorithm, wherein the sign language recognition algorithm is trained based on sign language data;

[0037] The analysis module is used to perform semantic analysis on the language text to obtain corresponding vehicle control information.

[0038] Optionally, the device further comprises:

[0039] A second acquisition unit is used to acquire a preset number of training sign language images and training language texts corresponding to the training sign language images, wherein the sign language data includes the training sign language images and the corresponding training language texts;

[0040] An input unit, used for inputting the training sign language image and the training language text into the sign language recognition algorithm to obtain training recognition information;

[0041] A calculation unit, configured to calculate a loss value of the sign language recognition algorithm based on the training recognition information by using a preset loss function, wherein the loss value is a value for measuring the degree of difference between the predicted translation information and the real translation information of the sign language recognition algorithm;

[0042] The optimization unit is used to optimize the sign language recognition algorithm based on the loss value through a preset optimization algorithm to obtain a trained sign language recognition algorithm.

[0043] Optionally, the execution unit includes:

[0044] An acquisition module, used to acquire seat information of the user in the vehicle;

[0045] A determination module, configured to determine whether the vehicle control information exceeds a preset control authority according to the seat information;

[0046] An execution module is used to execute corresponding vehicle control according to the vehicle control information when the vehicle control information exceeds the preset control authority.

[0047] Optionally, the identification unit further includes:

[0048] A comparison module, used to compare the sign language image with a preset gesture code to determine corresponding gesture code information;

[0049] A determination module, which determines corresponding vehicle control information according to the gesture coding information;

[0050] Wherein, each of the gesture codes corresponds to a type of vehicle control information.

[0051] According to a third aspect of the present disclosure, a vehicle is provided, wherein the vehicle includes the vehicle-computer interaction device as described in the second aspect of the present disclosure.

[0052] According to a fourth aspect of the present disclosure, there is provided an electronic device, including:

[0053] at least one processor; and

[0054] a memory communicatively connected to the at least one processor; wherein,

[0055] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0056] According to a fifth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0057] According to a sixth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0058] The vehicle-machine interaction method and device, vehicle, electronic device and storage medium provided in the present disclosure obtain sign language images of the user controlling the vehicle; recognize the sign language images to obtain corresponding vehicle control information; and perform corresponding vehicle control according to the vehicle control information. Compared with the related art, the embodiments of the present disclosure recognize sign language images to obtain corresponding vehicle control information and perform corresponding vehicle control, which allows people with hearing and hearing impairments to perform vehicle interaction through voice interaction.

[0059] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0061] Figure 1 A flow chart of a vehicle-machine interaction method provided by an embodiment of the present disclosure;

[0062] Figure 2 A flowchart of another vehicle-machine interaction method provided by an embodiment of the present disclosure;

[0063] Figure 3 A schematic diagram of the structure of a vehicle-machine interaction device provided by an embodiment of the present disclosure;

[0064] Figure 4 A schematic diagram of the structure of another vehicle-machine interaction device provided by an embodiment of the present disclosure;

[0065] Figure 5 A schematic block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0066] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0067] The following describes the vehicle-machine interaction method and device, vehicle, electronic device and storage medium of the embodiments of the present disclosure with reference to the accompanying drawings.

[0068] Figure 1 The following is a flow chart of a vehicle-machine interaction method provided by an embodiment of the present disclosure. Figure 1 As shown, the method is applied to a vehicle, and the method comprises the following steps:

[0069] Step 101, obtaining a sign language image of a user operating a vehicle.

[0070] The embodiments of the present disclosure may be applied to any type of vehicle. Specifically, the embodiments of the present disclosure are not limited to the type of vehicle.

[0071] In the embodiments of the present disclosure, a sign language image of a user in a vehicle is captured by an image capture device in the vehicle to obtain a sign language image, wherein the image capture device in the vehicle includes but is not limited to: an ordinary camera or a depth camera. The image capture device in the vehicle can be an image capture device that comes with the vehicle, or it can be an image capture device that has a data interaction relationship with the vehicle. Specifically, the embodiments of the present application do not limit the type of the image capture device in the vehicle.

[0072] It should be noted here that the types of the collected sign language images include but are not limited to: picture types, video types, etc. The image content of the sign language images includes but is not limited to: gestures, sign language, actions, etc.

[0073] Step 102: Recognize the sign language image to obtain corresponding vehicle control information.

[0074] In the disclosed embodiment, the vehicle control information is divided into: control information for controlling the vehicle, and control information for controlling the application of the vehicle system. Among them, the control information for controlling the vehicle, for example, opening the window, turning on the air conditioner, adjusting the air conditioner temperature and other control information. The control information for controlling the application of the vehicle system, for example, opening the map for search and navigation, opening the video player and other control information.

[0075] Step 103: Execute corresponding vehicle control according to the vehicle control information.

[0076] In the disclosed embodiment, the execution of vehicle control information is divided into: executing the control information of controlling the vehicle and executing the control information of the application of controlling the vehicle system. When executing the control information of controlling the vehicle, it is executed by directly controlling the vehicle through the vehicle system. For example, when the vehicle control information is to open the window, the vehicle system will directly execute the operation of opening the window according to the vehicle control information. When executing the control information of the application of controlling the vehicle system, it is necessary to call the application through the application program interface of the application to realize the corresponding vehicle control. For example, when the vehicle control information is to open the map for search and navigation, the vehicle system will call the map application through the application program interface of the map application so that the map application can execute the vehicle control information.

[0077] The application program interface is an interface that comes with the application and is defined by the application creator. Each application has a corresponding application program interface, and the corresponding application can be called through the application program interface to implement the application function. Specifically, the definition mode and method of the application program interface can be referred to the detailed description in the relevant technology, so it will not be repeated here.

[0078] The vehicle-machine interaction method provided by the present disclosure obtains a sign language image of a user controlling a vehicle; recognizes the sign language image to obtain corresponding vehicle control information; and performs corresponding vehicle control according to the vehicle control information. Compared with the related art, the embodiment of the present disclosure recognizes the sign language image to obtain the corresponding vehicle control information and performs the corresponding vehicle control, so that people with hearing and hearing impairments can perform vehicle interaction through voice interaction.

[0079] In one possible implementation of the embodiment of the present disclosure, the collected vehicle interior image includes not only a sign language image, but also images in various forms. If the sign language image of the user controlling the vehicle is not obtained from the collected vehicle interior image, the sign language image recognition will fail. Therefore, in order to obtain the sign language image contained in the vehicle interior image, the embodiment of the present disclosure provides a flow chart of another vehicle-machine interaction method, such as Figure 2 As shown:

[0080] Step 201 : extracting features from the collected vehicle interior image.

[0081] In the disclosed embodiment, the collected images inside the vehicle need to be input into an image recognition model, which performs feature extraction on the collected images inside the vehicle and outputs a sign language image.

[0082] In order to improve the recognition accuracy of the image recognition model, before the image recognition model is used to perform feature extraction on the collected vehicle images, the image recognition model needs to be trained. When training the image recognition model, a preset number of training sign language images need to be obtained and the training sign language images are input into the image recognition model to train the image recognition model. When the image recognition model reaches the preset convergence condition, a trained image recognition model is obtained.

[0083] The preset number refers to a pre-defined number value, the larger the preset number value, the higher the recognition rate of the trained image recognition model, and the smaller the preset number value, the lower the recognition rate of the trained image recognition model. Specifically, the preset number value is set according to actual conditions, and the embodiment of the present disclosure does not limit it.

[0084] The preset convergence condition is a custom set condition. When the image recognition model reaches the condition, it is considered that the image recognition model has been trained. For example, the preset convergence condition is: the loss value is less than the preset threshold. In each training, the size of the loss value is calculated. When the loss value is less than the preset threshold, the model is considered to have converged and the training ends. The preset threshold is a custom set value. The larger the preset threshold, the lower the recognition rate of the trained image recognition model. The smaller the preset threshold, the higher the recognition rate of the trained image recognition model.

[0085] It should be noted here that in order for the image recognition model to meet the preset convergence conditions, it is necessary to implement it through the loss function and the optimization algorithm. Specifically, for the loss function and the optimization algorithm, please refer to the detailed description in the relevant technology, so I will not go into details here.

[0086] Step 202: If the key features of the sign language image are extracted, the sign language image is obtained from the collected in-vehicle image.

[0087] In the disclosed embodiment, the key features of the sign language image refer to: features that can determine that the collected vehicle interior image is a sign language image. The features of the collected vehicle interior image can be extracted through a trained image recognition model, wherein the collection of the vehicle interior image is continuously performed through an image acquisition device, and the form of the collected vehicle interior image includes but is not limited to: video, etc. If the trained image recognition model extracts the key features of the sign language image, then a segment of the image containing the key features of the sign language image is directly output from the collected vehicle interior image, and the image is the sign language image. If the key features of the sign language image are not extracted, then the extraction from the collected vehicle interior image continues, and the feature extraction of the collected vehicle interior image is also continuously performed. After the vehicle is powered on, the collection of vehicle interior images and feature extraction of the collected vehicle interior images can begin.

[0088] In one implementable manner of the disclosed embodiment, after the captured image inside the vehicle is determined as a sign language image through the image recognition model in the above embodiment, the sign language image needs to be recognized to obtain corresponding vehicle control information. The recognition of the sign language image can be implemented in but not limited to the following manner: recognizing the sign language image as language text through a sign language recognition algorithm, wherein the sign language recognition algorithm is trained based on sign language data; and performing semantic analysis on the language text to obtain corresponding vehicle control information.

[0089] The language text is a textual expression of the content of the sign language image. Performing semantic analysis on the language text means processing the language text into a form that can be understood and processed by a computer, for example, processing the language text into a program language, etc. The method of performing semantic analysis on the language text may be, but is not limited to, performing semantic analysis on the language text using a natural language processing system (NLP). Specifically, the method of performing semantic analysis on the language text is not limited in the embodiments of the present disclosure.

[0090] In order to complete the recognition of sign language images, the sign language recognition model needs to be trained. The training of the sign language recognition model can also be adopted but not limited to the following method: training the sign language data to obtain the sign language recognition algorithm. The sign language recognition algorithm exists in the sign language recognition model, and the training of the sign language recognition algorithm is the training of the sign language recognition model.

[0091] As a refinement of the above embodiment, the sign language recognition algorithm is obtained by training the sign language data, which can be implemented in the following manner but is not limited to: obtaining a preset number of training sign language images, and training language texts corresponding to the training sign language images, wherein the sign language data includes the training sign language images and the corresponding training language texts; inputting the training sign language images and the training language texts into the sign language recognition algorithm to obtain training recognition information; based on the training recognition information, calculating the loss value of the sign language recognition algorithm by using a preset loss function, wherein the loss value is a value for measuring the degree of difference between the predicted translation information and the actual translation information of the sign language recognition algorithm; based on the loss value, optimizing the sign language recognition algorithm by using a preset optimization algorithm to obtain a trained sign language recognition algorithm.

[0092] The preset loss function refers to a pre-defined loss function, such as a mean square error function, a mean absolute error function, a quantile loss function, etc. The preset optimization algorithm refers to a pre-defined optimization algorithm, such as a gradient descent method, a Newton method, a quasi-Newton method, an improved iterative scaling method, etc. Specifically, the preset optimization algorithm and the preset loss function can be selected according to actual conditions, and the embodiments of the present disclosure do not limit them.

[0093] In one implementable method of the disclosed embodiment, since the permissions of persons of different identities and positions in the vehicle are inconsistent, the vehicle controls corresponding to the vehicle control information are also different. Therefore, it is necessary to identify the permissions of the vehicle control information to determine whether the controls of the vehicle by persons of different identities and positions comply with the permissions to ensure driving safety. The identification of the permissions of the vehicle control information can be implemented in the following ways but is not limited to: obtaining the seat information of the user in the vehicle; determining whether the vehicle control information exceeds the preset control permissions based on the seat information; if the vehicle control information exceeds the preset control permissions, performing corresponding vehicle controls based on the vehicle control information.

[0094] Among them, the method of obtaining seat information includes but is not limited to: directly determining the user's seat information through an image acquisition device. The preset control authority is a kind of control authority set by custom settings. The preset control authority is mainly divided into two types: driving control authority and passenger control authority. The driving control authority is the control authority of the user when in the main driving position, and the passenger control authority is the control authority of the user in the co-pilot and back seat positions. The driving control authority includes the passenger control authority, and on this basis, it adds control authorities including but not limited to: adjusting the main driver's seat, adjusting the suspension, adjusting the HUD (Head-up-Display, head-up display system), etc., which are control authorities that only the driver can operate and affect driving safety.

[0095] In actual applications, you can also customize gesture codes in the vehicle system and use specific gestures to quickly control the vehicle. In this case, there is no need to use a sign language recognition algorithm to recognize the sign language image. When it is determined that the gesture of the sign language image corresponds to the custom gesture code, the vehicle can be directly controlled. Regarding the quick control of the vehicle through specific gestures, it can be implemented in the following ways, but not limited to: compare the sign language image with the preset gesture code to determine the corresponding gesture code information; determine the corresponding vehicle control information based on the gesture code information; wherein each gesture code corresponds to a type of vehicle control information. For the encoding method of the preset gesture code, please refer to the detailed description in the relevant technology, so it will not be repeated here.

[0096] In summary, the embodiments of the present disclosure can achieve the following effects:

[0097] 1. By recognizing sign language images, obtaining corresponding vehicle control information and executing corresponding vehicle controls, people with hearing and hearing impairments can interact with the vehicle through voice interaction.

[0098] 2. By recognizing sign language images, the corresponding vehicle control information is obtained and the corresponding vehicle control is performed, which solves the problem of not being able to touch the touch screen in some scenarios and provides a new way of long-distance information input.

[0099] 3. By comparing the sign language image with the preset gesture code, the corresponding gesture code information is determined to obtain the vehicle control information and perform the corresponding vehicle control, so that the vehicle control can be realized conveniently and quickly.

[0100] Corresponding to the above-mentioned vehicle-machine interaction method, the present invention also provides a vehicle-machine interaction device. Since the device embodiment of the present invention corresponds to the above-mentioned method embodiment, the details not disclosed in the device embodiment can be referred to the above-mentioned method embodiment, and will not be repeated in the present invention.

[0101] Figure 3 A schematic diagram of a vehicle-machine interaction device provided in an embodiment of the present disclosure is shown in FIG. Figure 3 As shown, including:

[0102] A first acquisition unit 31 is used to acquire a sign language image of a user controlling a vehicle;

[0103] A recognition unit 32, configured to recognize the sign language image to obtain corresponding vehicle control information;

[0104] The execution unit 33 is used to execute corresponding vehicle manipulation according to the vehicle manipulation information.

[0105] The vehicle-machine interaction device provided by the present disclosure obtains the sign language image of the user controlling the vehicle; recognizes the sign language image to obtain the corresponding vehicle control information; and performs the corresponding vehicle control according to the vehicle control information. Compared with the related art, the embodiment of the present disclosure recognizes the sign language image, obtains the corresponding vehicle control information and performs the corresponding vehicle control, which allows people with hearing and hearing impairments to perform vehicle interaction through voice interaction.

[0106] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 4 As shown, the first acquisition unit 31 includes:

[0107] An extraction module 311 is used to extract features from the collected vehicle interior image;

[0108] The determination module 312 is used to obtain the sign language image from the collected in-vehicle image when the key features of the sign language image are extracted.

[0109] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 4 As shown, the identification unit 32 includes:

[0110] A recognition module 321, configured to recognize the sign language image as language text by using a sign language recognition algorithm, wherein the sign language recognition algorithm is trained based on sign language data;

[0111] The analysis module 322 is used to perform semantic analysis on the language text to obtain corresponding vehicle control information.

[0112] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 4 As shown, the device also includes:

[0113] A second acquisition unit 34 is used to acquire a preset number of training sign language images and training language texts corresponding to the training sign language images, wherein the sign language data includes the training sign language images and the corresponding training language texts;

[0114] An input unit 35, used to input the training sign language image and the training language text into the sign language recognition algorithm to obtain training recognition information;

[0115] A calculation unit 36, configured to calculate a loss value of the sign language recognition algorithm based on the training recognition information by using a preset loss function, wherein the loss value is a value for measuring the degree of difference between the predicted translation information and the actual translation information of the sign language recognition algorithm;

[0116] The optimization unit 37 is used to optimize the sign language recognition algorithm based on the loss value by using a preset optimization algorithm to obtain a trained sign language recognition algorithm.

[0117] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 4 As shown, the execution unit 33 includes:

[0118] An acquisition module 331 is used to acquire seat information of the user in the vehicle;

[0119] A determination module 332, configured to determine whether the vehicle control information exceeds a preset control authority according to the seat information;

[0120] The execution module 333 is used to execute corresponding vehicle control according to the vehicle control information when the vehicle control information exceeds the preset control authority.

[0121] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 4 As shown, the identification unit 32 also includes:

[0122] A comparison module 323 is used to compare the sign language image with a preset gesture code to determine corresponding gesture code information;

[0123] A determination module 324 determines corresponding vehicle control information according to the gesture coding information;

[0124] Wherein, each of the gesture codes corresponds to a type of vehicle control information.

[0125] In an embodiment of the present disclosure, a vehicle is also provided, wherein the vehicle is equipped with a vehicle-computer interaction device.

[0126] It should be noted that the above explanation of the method embodiment is also applicable to the device of the embodiment of the present disclosure, and the principle is the same, which is no longer limited in the embodiment of the present disclosure.

[0127] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0128] Figure 5A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0129] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 502 or a computer program loaded from a storage unit 508 to a RAM (Random Access Memory) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An I / O (Input / Output) interface 505 is also connected to the bus 504.

[0130] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0131] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the method of vehicle-machine interaction. For example, in some embodiments, the method of vehicle-machine interaction may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the aforementioned vehicle-computer interaction method in any other appropriate manner (for example, by means of firmware).

[0132] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0133] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0134] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a RAM, a ROM, an EPROM (Electrically Programmable Read-Only-Memory) or a flash memory, an optical fiber, a CD-ROM (Compact Dis sc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0135] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0136] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0137] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0138] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0139] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0140] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A vehicle-machine interaction method, characterized in that: include: Acquire the user's sign language image of the vehicle operation; Recognizing the sign language image to obtain corresponding vehicle control information; Execute corresponding vehicle manipulation according to the vehicle manipulation information.

2. The method according to claim 1, characterized in that: The step of obtaining the sign language image of the user controlling the vehicle includes: Extract features from the collected images inside the vehicle; If the key features of the sign language image are extracted, the sign language image is obtained from the collected in-vehicle image.

3. The method according to claim 2, characterized in that The step of recognizing the sign language image to obtain corresponding vehicle control information includes: Recognize the sign language image as language text by a sign language recognition algorithm, wherein the sign language recognition algorithm is trained based on sign language data; The language text is semantically analyzed to obtain corresponding vehicle control information.

4. The method according to claim 3, characterized in that: The training method of the sign language recognition algorithm includes: Acquire a preset number of training sign language images and training language texts corresponding to the training sign language images, wherein the sign language data includes the training sign language images and the corresponding training language texts; Inputting the training sign language image and the training language text into the sign language recognition algorithm to obtain training recognition information; Based on the training recognition information, a loss value of the sign language recognition algorithm is calculated by a preset loss function, wherein the loss value is a value for measuring the degree of difference between the predicted translation information and the actual translation information of the sign language recognition algorithm; Based on the loss value, the sign language recognition algorithm is optimized by a preset optimization algorithm to obtain a trained sign language recognition algorithm.

5. The method according to claim 1, characterized in that The performing corresponding vehicle manipulation according to the vehicle manipulation information comprises: Acquiring seat information of the user in the vehicle; Determining whether the vehicle control information exceeds a preset control authority according to the seat information; If the vehicle control information exceeds the preset control authority, corresponding vehicle control is performed according to the vehicle control information.

6. The method according to claim 2, characterized in that The step of recognizing the sign language image to obtain corresponding vehicle control information includes: Comparing the sign language image with a preset gesture code to determine corresponding gesture code information; Determining corresponding vehicle control information according to the gesture coding information; Wherein, each of the gesture codes corresponds to a type of vehicle control information.

7. A vehicle-machine interaction device, characterized in that: include: A first acquisition unit is used to acquire a sign language image of the user controlling the vehicle; A recognition unit, used to recognize the sign language image to obtain corresponding vehicle control information; An execution unit is used to execute corresponding vehicle manipulation according to the vehicle manipulation information.

8. A vehicle, characterized in that: The vehicle includes the vehicle-machine interaction device as described in claim 7.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

Images