Automatic driving remote control method giving consideration to voice control and related device

Abstract

The invention discloses an automatic driving remote control method giving consideration to voice control and a related device, and the method comprises the steps: a remote control terminal builds a communication connection line between the remote control terminal and all intelligent automobiles in a control range of the remote control terminal, and transmits an information collection instruction; the intelligent automobile obtains real-time state information data based on a sensor carried by the intelligent automobile and sends the real-time state information data back to the remote control terminal; the remote control terminal analyzes the real-time state information data and generates a real-time state information processing response; the remote control terminal receives the voice instruction, generates a remote control instruction matched with the voice instruction and sends the remote control instruction to the intelligent automobile; and the intelligent automobile performs control operation based on the remote control instruction. According to the automatic driving remote control method giving consideration to voice control and the related device, remote control over the automatic driving intelligent automobile based on the voice instruction is achieved, manual operation is reduced, the remote control cost is reduced, and the processing efficiency of the automatic driving intelligent automobile in case of emergencies is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of unmanned automatic driving control technology, and in particular to an automatic driving remote control method and related devices taking voice control into consideration. Background Art

[0002] In recent years, with the development of science and technology and the increase in demand, the production and sales of smart cars using new energy have continued to rise, and the market share of smart cars with assisted driving and even automatic driving has also been increasing. Smart cars continue to develop in a positive direction. The most representative of self-driving smart cars is the "Carrot Run" taxi, which was a great success in its trial run in Wuhan, causing a sensation across the country and even the world. The successful trial run of the "Carrot Run" taxi proves that the technology of smart self-driving cars has become relatively mature and is only one step away from large-scale practical application. Smart cars using new energy have already occupied the first half, while self-driving smart cars will naturally occupy the second half.

[0003] However, although the self-driving smart cars are currently running well overall, in complex environments, the self-driving smart cars appear to have limited capabilities and cannot make timely and most accurate responses to emergencies. Therefore, the self-driving smart cars currently in operation basically have remote control functions, that is, a driving safety officer is arranged at the remote control terminal to deal with emergencies in the self-driving smart cars. Usually, a driving safety officer needs to monitor multiple self-driving smart cars in real time, and its remote control operation is relatively complex. In the event of an emergency, if the driving safety officer fails to remotely control the self-driving smart car in time, it will cause the smart car to be delayed for too long, which will affect the passengers' riding experience at the least, and affect the operation of other vehicles at the worst, disrupting traffic order; in addition, if a relatively simple emergency is encountered and the driving safety officer does not need to perform simulated driving control, the driving safety officer still needs to process it on the simulated driving control terminal, consuming unnecessary human resources. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art. The present invention provides an automatic driving remote control method and related devices that take voice control into consideration, thereby realizing remote control of an automatic driving smart car based on voice commands, reducing manual operations, reducing remote control costs, and effectively improving the processing efficiency of the automatic driving smart car when encountering emergencies.

[0005] The present invention provides an automatic driving remote control method taking voice control into consideration, the method comprising:

[0006] The remote control terminal establishes a communication connection line with all smart cars within its control range and sends information collection instructions to the smart cars;

[0007] The smart car acquires real-time status information data based on the sensors carried by the smart car, and sends the real-time status information data back to the remote control terminal;

[0008] The remote control terminal analyzes the real-time status information data and generates a real-time status information processing response;

[0009] The remote control terminal receives the voice command, generates a remote control command matching the voice command, and sends the remote control command to the smart car;

[0010] The smart car performs control operations based on the remote control instructions.

[0011] Furthermore, the remote control terminal establishes a communication connection line with all smart cars within its control range, and sends an information collection instruction to the smart cars, including:

[0012] The remote control terminal opens the remote control connection main port, searches for all working smart cars within its control range, and sends a communication connection configuration signal to the smart cars;

[0013] The smart car receives the communication connection configuration signal and returns a response signal;

[0014] The remote control terminal configures a remote control connection sub-port and a corresponding IP address for the smart car according to the order in which the response signals are returned;

[0015] After the configuration is completed, the remote control terminal establishes a communication connection line with the smart car based on the connection sub-port and IP address of the smart car;

[0016] The remote control terminal sends an information collection instruction to the smart car based on the communication connection line.

[0017] Furthermore, the smart car acquires real-time status information data based on the sensors carried by the smart car, and sends the real-time status information data back to the remote control terminal, including:

[0018] The smart car obtains radar sensing information around the smart car based on the radar component;

[0019] The smart car acquires visual image sensing information around the smart car based on a binocular visual camera;

[0020] The smart car obtains the posture sensing information of the smart car itself based on the inertial measurement unit;

[0021] The smart car sends the radar sensing information, visual image sensing information and posture sensing information to the remote control terminal based on the communication connection line.

[0022] Further, the remote control terminal analyzes the real-time status information data to generate a real-time status information processing response, including:

[0023] The remote control terminal analyzes the real-time status information data and compares it with the data in a preset database to determine whether the smart car corresponding to the real-time status information data requires manual intervention;

[0024] When the remote control terminal determines that the smart car corresponding to the real-time status information data requires human intervention, the real-time status information data is fed back to the remote control terminal of the corresponding driving safety officer.

[0025] Furthermore, when the remote control terminal determines that the smart car corresponding to the real-time status information data requires manual intervention, feeding back the real-time status information data to the remote control terminal of the corresponding driving safety officer includes:

[0026] The remote control terminal determines the processing level of the smart car corresponding to the real-time status information data that requires human intervention, and based on the processing level, feeds back the real-time status information data to the voice control terminal or simulated driving control terminal of the corresponding driving safety officer.

[0027] Furthermore, the remote control terminal receives the voice command, generates a remote control command matching the voice command, and sends the remote control command to the smart car, including:

[0028] The remote control terminal receives the voice command signal of the driving safety officer based on the voice pickup device;

[0029] Preprocessing the voice command signal;

[0030] Extracting an acoustic feature signal of the preprocessed voice command signal, processing the acoustic feature signal based on an acoustic matching model, and acquiring phoneme information in the preprocessed voice command signal;

[0031] Extracting semantic feature information from the phoneme information in the preprocessed voice command signal, processing the semantic feature information based on a semantic matching model, and generating a voice control command;

[0032] Decoding the voice control command and matching it in a remote control command database to the corresponding remote control command;

[0033] The matched remote control instruction is sent to the smart car based on the communication connection line.

[0034] Furthermore, after the smart car performs control operations based on the remote control instructions for a period of time, the smart car obtains status information data after the remote control operation based on its own sensors and returns it to the remote control terminal, and the remote control terminal determines the remote control operation effect of the smart car based on the status information data after the remote control operation.

[0035] The present invention also provides an automatic driving remote control system taking into account voice control, the automatic driving remote control system taking into account voice control is used to implement the automatic driving remote control method taking into account voice control, the system comprises:

[0036] A communication connection establishment module, wherein the communication connection establishment module is used for the remote control terminal to establish a communication connection line with all smart cars within its control range and send information collection instructions to the smart cars;

[0037] A status information acquisition module, which is used for the smart car to acquire real-time status information data based on its own sensors and send the real-time status information data back to the remote control terminal;

[0038] A status information analysis module, which is used by the remote control terminal to analyze the real-time status information data and generate a real-time status information processing response;

[0039] A voice command matching module, which is used for the remote control terminal to receive voice commands, generate remote control commands that match the voice commands, and send the remote control commands to the smart car;

[0040] A remote control operation module is used for the smart car to perform control operations based on the remote control instructions.

[0041] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned autonomous driving remote control method taking voice control into consideration.

[0042] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned automatic driving remote control method taking voice control into consideration is implemented.

[0043] The present invention provides an automatic driving remote control method and related devices taking voice control into consideration, so as to realize remote control of an automatic driving intelligent vehicle based on voice commands. Compared with remote control by a driving safety officer based on a simulated driving control terminal, the remote control method based on voice command control has a faster response speed and a higher response efficiency. Meanwhile, in handling relatively simple emergencies where it is unnecessary for the driving safety officer to perform simulated driving control, the remote control method based on voice command control consumes less human resources of the driving safety officer, does not require more manual simulated driving operation commands, and has a simpler control process and a shorter consumption time, so that the automatic driving intelligent vehicle can get rid of emergencies more quickly. In addition, the voice control command hardware can further reduce the cost of the remote control terminal, effectively improve the handling efficiency of the automatic driving intelligent vehicle when encountering emergencies, and has certain application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 This is a flow chart of the remote control method for autonomous driving taking voice control into consideration in the first embodiment of the present invention;

[0046] Figure 2 This is a flow chart of the remote control terminal in the first embodiment of the present invention establishing a communication connection line and sending an information collection instruction;

[0047] Figure 3 This is a control schematic diagram of the remote control method for autonomous driving taking voice control into consideration in the first embodiment of the present invention;

[0048] Figure 4 This is a flow chart of the smart car in the first embodiment of the present invention acquiring real-time status information data and sending it back to the remote control terminal;

[0049] Figure 5 is a flow chart of a remote control terminal generating a real-time status information processing response in Embodiment 1 of the present invention;

[0050] Figure 6 This is a flow chart of the remote control terminal generating a remote control command and sending it to the smart car in the first embodiment of the present invention;

[0051] Figure 7 This is an architecture diagram of an autonomous driving remote control system taking voice control into consideration in the second embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] In the present invention, it should be understood that terms such as "include" or "have" are intended to indicate the existence of features, numbers, steps, behaviors, components, parts or a combination thereof disclosed in the specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, behaviors, components, parts or a combination thereof exist or are added.

[0054] It should also be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0055] Embodiment 1

[0056] Embodiment 1 of the present invention provides an automatic driving remote control method taking voice control into consideration, the method comprising: a remote control terminal establishes a communication connection line with all smart cars within its control range, and sends information collection instructions to the smart cars; the smart cars obtain real-time status information data based on their own sensors, and send the real-time status information data back to the remote control terminal; the remote control terminal analyzes the real-time status information data and generates a real-time status information processing response; the remote control terminal receives voice instructions, generates a remote control instruction matching the voice instructions, and sends the remote control instruction to the smart cars; the smart cars perform control operations based on the remote control instructions.

[0057] In an optional implementation of this embodiment, Figure 1 As shown, Figure 1 The flowchart of the remote control method for autonomous driving taking voice control into consideration in the first embodiment of the present invention is shown, and includes the following steps:

[0058] S101, the remote control terminal establishes a communication connection line with all smart cars within its control range, and sends an information collection instruction to the smart cars;

[0059] In an optional implementation of this embodiment, Figure 2 As shown, Figure 2 The flowchart of the remote control terminal establishing a communication connection line and sending an information collection instruction in the first embodiment of the present invention is shown, including the following steps:

[0060] S201, the remote control terminal opens the remote control connection main port, searches for all working smart cars within its control range, and sends a communication connection configuration signal to the smart cars;

[0061] In an optional implementation of this embodiment, Figure 3 As shown, Figure 3 A control schematic diagram of an automatic driving remote control method taking voice control into consideration in Embodiment 1 of the present invention is shown, including a remote control terminal, several smart cars, and a driving safety officer arranged in the remote control terminal.

[0062] In an optional implementation of this embodiment, the remote control terminal opens a remote control connection master port set on the remote control terminal and generates a remote control connection master IP address.

[0063] In an optional implementation of this embodiment, the remote control terminal sends a communication connection configuration signal indiscriminately to all working smart cars within its control range based on the remote control connection total port and the remote control connection total IP address.

[0064] Specifically, the remote control terminal sends the next communication connection configuration signal after a certain period of time after sending the previous communication connection configuration signal.

[0065] S202, the smart car receives the communication connection configuration signal and returns a response signal;

[0066] In an optional implementation of this embodiment, after receiving the communication connection configuration signal sent by the remote control terminal, the working smart car returns a communication connection configuration response signal to the remote control terminal.

[0067] S203, the remote control terminal configures a remote control connection sub-port and a corresponding IP address for the smart car according to the order in which the response signals are returned;

[0068] In an optional implementation of this embodiment, the remote control terminal sequentially configures serial numbers for the corresponding smart cars according to the order in which the smart cars return response signals, and allocates remote control connection sub-ports and remote control connection sub-IP addresses according to the serial numbers.

[0069] S204, after the configuration is completed, the remote control terminal establishes a communication connection line with the smart car based on the connection sub-port and IP address of the smart car;

[0070] In an optional implementation of this embodiment, the remote control terminal establishes communication connection lines with the corresponding smart car based on its own remote control connection total port and remote control connection total IP address, and the remote control connection sub-port and remote control connection sub-IP address of the smart car.

[0071] S205. The remote control terminal sends an information collection instruction to the smart car based on the communication connection line.

[0072] In an optional implementation of this embodiment, after the communication connection line is established, the remote control terminal sends an information collection instruction to the smart car based on the communication connection line.

[0073] S102, the smart car acquires real-time status information data based on its own sensors, and sends the real-time status information data back to the remote control terminal;

[0074] In an optional implementation of this embodiment, Figure 4 As shown, Figure 4 The flowchart of the smart car in the first embodiment of the present invention acquiring real-time status information data and sending it back to the remote control terminal is shown, including the following steps:

[0075] S401, the smart car obtains radar sensing information around the smart car based on a radar component;

[0076] In an optional implementation of this embodiment, the smart car is equipped with a radar component, and the radar component is arranged around the smart car. The smart car obtains radar sensing information around the smart car based on the radar component.

[0077] Specifically, the radar component includes a millimeter wave radar component, a laser radar component and an ultrasonic radar component.

[0078] Furthermore, the radar component is mainly used to obtain location information of obstacles around the smart car.

[0079] S402, the smart car obtains visual image sensing information around the smart car based on a binocular visual camera;

[0080] In an optional implementation of this embodiment, the smart car is equipped with binocular vision cameras, which are arranged around and on the roof of the smart car. The binocular vision cameras are mainly used to obtain visual image sensing information around the smart car.

[0081] Specifically, the binocular vision camera arranged on the roof of the smart car can rotate 360°.

[0082] S403, the smart car obtains the posture sensing information of the smart car itself based on the inertial measurement unit;

[0083] In an optional implementation of this embodiment, the smart car is equipped with an inertial measurement unit IMU, and the inertial measurement unit IMU is arranged around the body, four wheels, roof and chassis of the smart car. The inertial measurement unit is used to obtain the smart car's own posture sensing information.

[0084] S404: The smart car sends the radar sensing information, visual image sensing information and posture sensing information to the remote control terminal based on the communication connection line.

[0085] In an optional implementation of this embodiment, after collecting its own radar sensor information, visual image sensor information and posture sensor information, the smart car will send the radar sensor information, visual image sensor information and posture sensor information to the remote control terminal based on the communication connection line.

[0086] Radar sensor information and visual image sensor information are collected here, mainly used to judge the environment in which the smart car is located and the general situation of the emergency. Posture sensor information is collected mainly to judge whether the running posture of the smart car is intact, whether the operation is normal, and whether the body and related parts are damaged.

[0087] S103, the remote control terminal analyzes the real-time status information data and generates a real-time status information processing response;

[0088] In an optional implementation of this embodiment, Figure 5 As shown, Figure 5 The flowchart of the remote control terminal generating a real-time status information processing response in the first embodiment of the present invention is shown, including the following steps:

[0089] S501, the remote control terminal analyzes the real-time status information data and compares it with the data in a preset database to determine whether the smart car corresponding to the real-time status information data needs manual intervention;

[0090] In an optional implementation of this embodiment, the remote control terminal analyzes the radar sensor information, visual image sensor information and posture sensor information sent by the smart car, compares it with the relevant data of the normal operation of the smart car in the preset database, and determines whether the smart car corresponding to the real-time status information data requires human intervention.

[0091] Specifically, when the information reflected by the radar sensor information, visual image sensor information and posture sensor information sent by the smart car exceeds a certain threshold of the range of normal related data in the preset database, it is determined that an emergency has occurred in the smart car and manual intervention is required.

[0092] S502, feeding back the real-time status information data to the remote control terminal of the corresponding driving safety officer;

[0093] In an optional implementation of this embodiment, when the remote control terminal determines in step S501 that the smart car corresponding to the real-time status information data requires human intervention, the real-time status information data is fed back to the remote control terminal of the driving safety officer corresponding to the smart car.

[0094] S503, the remote control terminal processes the smart car based on the real-time status information data;

[0095] In an optional implementation of this embodiment, when the remote control terminal determines in step S501 that the smart car corresponding to the real-time status information data does not require human intervention, the remote control terminal processes the smart car independently based on the real-time status information data.

[0096] S504, the remote control terminal determines the processing level of the smart car corresponding to the real-time status information data that requires manual intervention, and feeds back the real-time status information data to the voice control terminal or simulated driving control terminal of the corresponding driving safety officer based on the processing level;

[0097] In an optional implementation of this embodiment, if the judgment in step S502 requires manual processing, the remote control terminal further determines the processing level of the smart car corresponding to the real-time status information data that requires manual intervention, or the processing level is determined by the driving safety officer, and based on the processing level, the real-time status information data is fed back to the voice control terminal or simulated driving control terminal of the corresponding driving safety officer.

[0098] Specifically, the processing levels include simple and severe, wherein the real-time status information data of the simple processing level is fed back to the voice control terminal of the driving safety officer, and the real-time status information of the severe processing level is fed back to the simulated driving control terminal of the driving safety officer.

[0099] It should be noted that if the remote control terminal sets the processing level to simple and assigns it to the driving safety officer's voice control terminal, if the driving safety officer judges that the processing level is serious, it can be assigned back to the driving safety officer's simulated driving control terminal; similarly, if the remote control terminal sets the processing level to serious and assigns it to the driving safety officer's simulated driving control terminal, if the driving safety officer judges that the processing level is simple, it can be assigned back to the driving safety officer's voice control terminal.

[0100] S505, feeding back the real-time status information data to the voice control terminal of the corresponding driving safety officer;

[0101] In an optional implementation of this embodiment, when it is determined that the processing level of the emergency event corresponding to the real-time status information data is simple, the remote control terminal feeds back the real-time status information data to the voice control terminal of the corresponding driving safety officer.

[0102] S506: Feedback the real-time status information data to the simulated driving control terminal of the corresponding driving safety officer.

[0103] In an optional implementation of this embodiment, when it is determined that the processing level of the emergency event corresponding to the real-time status information data is serious, the remote control terminal feeds back the real-time status information data to the simulated driving control terminal of the corresponding driving safety officer.

[0104] It should be noted that the voice control terminal only includes a voice control device, and the simulated driving control terminal includes all the operating hardware of the driving seat of the smart car, including steering wheel, brake control, brake control and other operating hardware, which can independently control the actual operation control of the smart car such as driving, braking, steering, etc.

[0105] S104, the remote control terminal receives the voice command, generates a remote control command matching the voice command, and sends the remote control command to the smart car;

[0106] In an optional implementation of this embodiment, Figure 6 As shown, Figure 6 The flowchart of the remote control terminal generating a remote control instruction and sending it to the smart car in the first embodiment of the present invention is shown, including the following steps:

[0107] S601, the remote control terminal receives a voice command signal from a driver safety officer based on a voice pickup device;

[0108] In an optional implementation of this embodiment, the remote control terminal receives the voice command signal of the driving safety officer based on a voice pickup device provided on the voice control terminal of the driving safety officer.

[0109] S602, preprocessing the voice command signal;

[0110] In an optional implementation of this embodiment, the preprocessing of the voice command signal includes endpoint detection, pre-emphasis, framing, windowing, enhancement, denoising, etc., which are mainly used to ensure the quality of the voice command signal and ensure the uniformity and smoothness of the voice command signal.

[0111] S603, extracting an acoustic feature signal of the preprocessed voice command signal, processing the acoustic feature signal based on an acoustic matching model, and acquiring phoneme information in the preprocessed voice command signal;

[0112] In an optional implementation of this embodiment, an acoustic feature signal of the preprocessed voice command signal is extracted, and the acoustic feature signal is a voice acoustic feature vector in the preprocessed voice command signal that can be processed by a computer.

[0113] In an optional implementation of this embodiment, the acoustic feature signal is processed based on a trained acoustic matching model to obtain phoneme information in the preprocessed voice command signal.

[0114] Specifically, the acoustic matching model trained based on acoustic feature signals in a large number of speech signals processes the acoustic feature signals, and processes the acoustic feature signals as input feature parameters to obtain the phoneme information of the smallest unit in the preprocessed speech command signal.

[0115] Here, the phoneme information in the preprocessed voice command signal is obtained, and homophones in the voice command signal that are similar to the suspected acoustic pronunciation of the existing command can be identified.

[0116] S604, extracting semantic feature information from the phoneme information in the preprocessed voice command signal, processing the semantic feature information based on a semantic matching model, and generating a voice control command;

[0117] In an optional implementation of this embodiment, semantic feature information in the phoneme information in the preprocessed voice command signal is extracted, and the semantic feature information is processed based on a trained semantic matching model to generate a voice control command.

[0118] Specifically, the semantic matching model trained based on a large amount of semantic feature information processes the semantic feature information, identifies the semantic feature information with contextual association or similar semantics in the phoneme information in step S603, and combines the separate phoneme information into sentences with certain logic to form voice control instructions.

[0119] S605, decoding the voice control instruction and matching it in a remote control instruction database to match the corresponding remote control instruction;

[0120] In an optional implementation of this embodiment, a speech decoding and recognition network model is established, the voice control instruction is recognized based on the speech decoding and recognition network model, and a best matching tag is generated. The best matching tag is matched in a remote control instruction database to match the corresponding remote control instruction.

[0121] Specifically, a speech decoding and recognition network model is established based on Mel-Frequency Cepstral Coefficients (MFCC) and Hidden Markov Model (HMM). The recognition network model recognizes the voice control instruction and generates an optimal path with the maximum probability of outputting a valid voice control instruction, which corresponds to the best matching label.

[0122] S606: Send the matched remote control instruction to the smart car based on the communication connection line.

[0123] In an optional implementation of this embodiment, the remote control instruction matched in step S605 is sent to the smart car based on the communication connection line.

[0124] S105: The smart car performs control operations based on the remote control instructions.

[0125] In an optional implementation of this embodiment, the smart car performs a control operation based on the remote control instruction sent by the remote control terminal in step S104.

[0126] In an optional implementation of this embodiment, the comparison relationship between commonly used voice control commands and control operations performed by the smart car based on the matched remote control commands is as shown in Table 1 below:

[0127] Table 1

[0128]

[0129] In an optional implementation of the present embodiment, after the smart car performs control operations based on the remote control instructions for a period of time, the smart car obtains status information data after the remote control operation based on its own sensors and returns the data to the remote control terminal, and the remote control terminal determines the remote control operation effect of the smart car based on the status information data after the remote control operation.

[0130] Specifically, when the remote control terminal determines that the remote control operation effect of the smart car is not good based on the state information data after the remote control operation, the state information data collected at this time is directly fed back to the simulated driving control terminal of the driving safety officer, and the driving safety officer processes it again based on the simulated driving control terminal;

[0131] When the remote control terminal determines that the remote control operation of the smart car is effective based on the status information data after the remote control operation, the smart car returns to the autonomous driving state.

[0132] In summary, Embodiment 1 of the present invention provides an automatic driving remote control method taking voice control into consideration, which realizes remote control of an automatic driving smart car based on voice commands. Compared with remote control by a driving safety officer based on a simulated driving control terminal, the remote control method based on voice command control has a faster response speed and higher response efficiency. At the same time, in handling relatively simple emergencies where it is unnecessary for the driving safety officer to perform simulated driving control, the remote control method based on voice command control consumes less human resources of the driving safety officer, does not require more manual simulated driving operation instructions, and has a simpler control process and consumes less time, so the automatic driving smart car can get rid of emergencies more quickly. In addition, equipping with voice control command hardware can further reduce the cost of the remote control terminal, effectively improve the handling efficiency of the automatic driving smart car when encountering emergencies, and has certain application value.

[0133] Embodiment 2

[0134] Embodiment 2 of the present invention provides an autonomous driving remote control system that takes voice control into consideration. The autonomous driving remote control system that takes voice control into consideration is used to implement the autonomous driving remote control method that takes voice control into consideration in Embodiment 1. The system includes a communication connection establishment module, a status information acquisition module, a status information analysis module, a voice command matching module, and a remote control operation module.

[0135] In an optional implementation of this embodiment, Figure 7 As shown, Figure 7 The architecture diagram of the autonomous driving remote control system taking voice control into consideration in the second embodiment of the present invention is shown, and the system includes:

[0136] A communication connection establishing module 10, wherein the communication connection establishing module 10 is used for the remote control terminal to establish a communication connection line with all smart cars within its control range, and to send information collection instructions to the smart cars;

[0137] In an optional implementation of this embodiment, the remote control terminal establishes a communication connection line with all smart cars within its control range, and sends an information collection instruction to the smart car, including:

[0138] The remote control terminal opens the remote control connection main port, searches for all working smart cars within its control range, and sends a communication connection configuration signal to the smart cars;

[0139] The smart car receives the communication connection configuration signal and returns a response signal;

[0140] The remote control terminal configures a remote control connection sub-port and a corresponding IP address for the smart car according to the order in which the response signals are returned;

[0141] After the configuration is completed, the remote control terminal establishes a communication connection line with the smart car based on the connection sub-port and IP address of the smart car;

[0142] The remote control terminal sends an information collection instruction to the smart car based on the communication connection line.

[0143] A status information acquisition module 20, which is used for the smart car to acquire real-time status information data based on its own sensors and send the real-time status information data back to the remote control terminal;

[0144] In an optional implementation of this embodiment, the smart car acquires real-time status information data based on its own sensors, and sends the real-time status information data back to the remote control terminal, including:

[0145] The smart car obtains radar sensing information around the smart car based on the radar component;

[0146] The smart car acquires visual image sensing information around the smart car based on a binocular visual camera;

[0147] The smart car obtains the posture sensing information of the smart car itself based on the inertial measurement unit;

[0148] The smart car sends the radar sensing information, visual image sensing information and posture sensing information to the remote control terminal based on the communication connection line.

[0149] A status information analysis module 30, wherein the status information analysis module 30 is used by the remote control terminal to analyze the real-time status information data and generate a real-time status information processing response;

[0150] In an optional implementation of this embodiment, the remote control terminal analyzes the real-time status information data to generate a real-time status information processing response, including:

[0151] The remote control terminal analyzes the real-time status information data and compares it with the data in a preset database to determine whether the smart car corresponding to the real-time status information data requires manual intervention;

[0152] When the remote control terminal determines that the smart car corresponding to the real-time status information data requires human intervention, the real-time status information data is fed back to the remote control terminal of the driving safety officer corresponding to the smart car;

[0153] When the remote control terminal determines that the smart car corresponding to the real-time status information data does not require human intervention, the remote control terminal processes the smart car based on the real-time status information data;

[0154] If the judgment is that manual processing is required, the remote control terminal further determines the processing level of the smart car corresponding to the real-time status information data that requires manual intervention, or the driver safety officer determines the processing level, and based on the processing level, the real-time status information data is fed back to the voice control terminal or simulated driving control terminal of the corresponding driver safety officer;

[0155] If the remote control terminal sets the processing level as simple and allocates it to the voice control terminal of the driving safety officer, if the driving safety officer determines that the processing level is serious, it can be allocated back to the simulated driving control terminal of the driving safety officer; similarly, if the remote control terminal sets the processing level as serious and allocates it to the simulated driving control terminal of the driving safety officer, if the driving safety officer determines that the processing level is simple, it can be allocated back to the voice control terminal of the driving safety officer;

[0156] When it is determined that the processing level of the emergency event corresponding to the real-time status information data is simple, the remote control terminal feeds back the real-time status information data to the voice control terminal corresponding to the driving safety officer;

[0157] When it is determined that the processing level of the emergency event corresponding to the real-time status information data is serious, the remote control terminal feeds back the real-time status information data to the simulated driving control terminal of the corresponding driving safety officer.

[0158] A voice command matching module 40, wherein the voice command matching module 40 is used for the remote control terminal to receive the voice command, generate a remote control command matching the voice command, and send the remote control command to the smart car;

[0159] In an optional implementation of this embodiment, the remote control terminal receives the voice command, generates a remote control command matching the voice command, and sends the remote control command to the smart car, including:

[0160] The remote control terminal receives the voice command signal of the driving safety officer based on the voice pickup device;

[0161] Preprocessing the voice command signal;

[0162] Extracting an acoustic feature signal of the preprocessed voice command signal, processing the acoustic feature signal based on an acoustic matching model, and acquiring phoneme information in the preprocessed voice command signal;

[0163] Extracting semantic feature information from the phoneme information in the preprocessed voice command signal, processing the semantic feature information based on a semantic matching model, and generating a voice control command;

[0164] Decoding the voice control command and matching it in a remote control command database to the corresponding remote control command;

[0165] The matched remote control instruction is sent to the smart car based on the communication connection line.

[0166] A remote control operation module 50 is used for the smart car to perform control operations based on the remote control instructions.

[0167] In an optional implementation of the present embodiment, after the smart car performs control operations based on the remote control instructions for a period of time, the smart car obtains status information data after the remote control operation based on its own sensors and returns the data to the remote control terminal, and the remote control terminal determines the remote control operation effect of the smart car based on the status information data after the remote control operation.

[0168] Specifically, when the remote control terminal determines that the remote control operation effect of the smart car is not good based on the state information data after the remote control operation, the state information data collected at this time is directly fed back to the simulated driving control terminal of the driving safety officer, and the driving safety officer processes it again based on the simulated driving control terminal;

[0169] When the remote control terminal determines that the remote control operation of the smart car is effective based on the status information data after the remote control operation, the smart car returns to the autonomous driving state.

[0170] In summary, the second embodiment of the present invention provides an automatic driving remote control system taking voice control into consideration. The automatic driving remote control system taking voice control into consideration is used to implement the automatic driving remote control method taking voice control into consideration in the first embodiment, and realize remote control of the automatic driving smart car based on voice commands. Compared with the remote control performed by the driving safety officer based on the simulated driving control terminal, the remote control method based on voice command control has a faster response speed and higher response efficiency. At the same time, in the handling of relatively simple emergencies where it is unnecessary for the driving safety officer to perform simulated driving control, the remote control method based on voice command control consumes less human resources of the driving safety officer, and does not require more manual simulated driving operation instructions. The control process is simpler and takes less time, so the automatic driving smart car can get rid of emergencies more quickly. In addition, equipping with voice control command hardware can further reduce the cost of the remote control terminal, effectively improve the handling efficiency of the automatic driving smart car when encountering emergencies, and has certain application value.

[0171] Embodiment 3

[0172] Embodiment 3 of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for remote control of autonomous driving taking voice control into consideration as described in Embodiment 1.

[0173] In summary, embodiment three of the present invention provides an electronic device for implementing the automatic driving remote control method taking voice control into consideration in embodiment one, and realizing remote control of automatic driving smart cars based on voice commands. Compared with remote control by a driving safety officer based on a simulated driving control terminal, the remote control method based on voice command control has a faster response speed and higher response efficiency. At the same time, in handling relatively simple emergencies where it is unnecessary for the driving safety officer to perform simulated driving control, the remote control method based on voice command control consumes less human resources of the driving safety officer, and does not require more manual simulated driving operation instructions. The control process is simpler and takes less time, so the automatic driving smart car can get rid of emergencies more quickly. In addition, equipping with voice control command hardware can further reduce the cost of the remote control terminal, effectively improve the handling efficiency of the automatic driving smart car when encountering emergencies, and has certain application value.

[0174] Embodiment 4

[0175] Embodiment 4 of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for remote control of autonomous driving taking into account voice control described in Embodiment 1 is implemented.

[0176] In summary, Embodiment 4 of the present invention provides a computer-readable storage medium for executing the autonomous driving remote control method taking voice control into consideration as described in Embodiment 1, and realizing remote control of an autonomous driving smart car based on voice commands. Compared with remote control by a driving safety officer based on a simulated driving control terminal, the remote control method based on voice command control has a faster response speed and higher response efficiency. At the same time, in handling relatively simple emergencies where it is unnecessary for the driving safety officer to perform simulated driving control, the remote control method based on voice command control consumes less human resources of the driving safety officer, and does not require more manual simulated driving operation instructions. The control process is simpler and takes less time, so the autonomous driving smart car can get rid of emergencies more quickly. In addition, equipping with voice control command hardware can further reduce the cost of the remote control terminal, effectively improving the processing efficiency of the autonomous driving smart car when encountering emergencies, and has certain application value.

[0177] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0178] In addition, the embodiments of the present invention are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A remote control method for autonomous driving taking voice control into consideration, characterized in that: The method comprises: The remote control terminal establishes a communication connection line with all smart cars within its control range and sends information collection instructions to the smart cars; The smart car acquires real-time status information data based on the sensors carried by the smart car, and sends the real-time status information data back to the remote control terminal; The remote control terminal analyzes the real-time status information data and generates a real-time status information processing response; The remote control terminal receives the voice command, generates a remote control command matching the voice command, and sends the remote control command to the smart car; The smart car performs control operations based on the remote control instructions.

2. The automatic driving remote control method taking voice control into consideration as claimed in claim 1, characterized in that: The remote control terminal establishes a communication connection line with all smart cars within its control range, and sends an information collection instruction to the smart car, including: The remote control terminal opens the remote control connection main port, searches for all working smart cars within its control range, and sends a communication connection configuration signal to the smart cars; The smart car receives the communication connection configuration signal and returns a response signal; The remote control terminal configures a remote control connection sub-port and a corresponding IP address for the smart car according to the order in which the response signals are returned; After the configuration is completed, the remote control terminal establishes a communication connection line with the smart car based on the connection sub-port and IP address of the smart car; The remote control terminal sends an information collection instruction to the smart car based on the communication connection line.

3. The automatic driving remote control method taking voice control into consideration as claimed in claim 1, characterized in that: The smart car acquires real-time status information data based on the sensors carried by the smart car, and sends the real-time status information data back to the remote control terminal, including: The smart car obtains radar sensing information around the smart car based on the radar component; The smart car acquires visual image sensing information around the smart car based on a binocular visual camera; The smart car obtains the posture sensing information of the smart car itself based on the inertial measurement unit; The smart car sends the radar sensing information, visual image sensing information and posture sensing information to the remote control terminal based on the communication connection line.

4. The automatic driving remote control method taking voice control into consideration as claimed in claim 1, characterized in that: The remote control terminal analyzes the real-time status information data and generates a real-time status information processing response, including: The remote control terminal analyzes the real-time status information data and compares it with the data in a preset database to determine whether the smart car corresponding to the real-time status information data requires manual intervention; When the remote control terminal determines that the smart car corresponding to the real-time status information data requires human intervention, the real-time status information data is fed back to the remote control terminal of the corresponding driving safety officer.

5. The automatic driving remote control method taking voice control into consideration as claimed in claim 4, characterized in that: When the remote control terminal determines that the smart car corresponding to the real-time status information data requires manual intervention, the real-time status information data is fed back to the remote control terminal corresponding to the driving safety officer, including: The remote control terminal determines the processing level of the smart car corresponding to the real-time status information data that requires human intervention, and based on the processing level, feeds back the real-time status information data to the voice control terminal or simulated driving control terminal of the corresponding driving safety officer.

6. The automatic driving remote control method taking voice control into consideration as claimed in claim 1, characterized in that: The remote control terminal receives the voice command, generates a remote control command matching the voice command, and sends the remote control command to the smart car, comprising: The remote control terminal receives the voice command signal of the driving safety officer based on the voice pickup device; Preprocessing the voice command signal; Extracting an acoustic feature signal of the preprocessed voice command signal, processing the acoustic feature signal based on an acoustic matching model, and acquiring phoneme information in the preprocessed voice command signal; Extracting semantic feature information from the phoneme information in the preprocessed voice command signal, processing the semantic feature information based on a semantic matching model, and generating a voice control command; Decoding the voice control command and matching it in a remote control command database to the corresponding remote control command; The matched remote control instruction is sent to the smart car based on the communication connection line.

7. The automatic driving remote control method taking voice control into consideration as claimed in claim 1, characterized in that: After the smart car performs control operations based on the remote control instructions for a period of time, the smart car obtains status information data after the remote control operation based on its own sensors and returns it to the remote control terminal. The remote control terminal determines the remote control operation effect of the smart car based on the status information data after the remote control operation.

8. An automatic driving remote control system taking into account voice control, characterized in that: The autonomous driving remote control system taking into account voice control is used to implement the autonomous driving remote control method taking into account voice control as described in any one of claims 1 to 7, and the system includes: A communication connection establishment module, wherein the communication connection establishment module is used for the remote control terminal to establish a communication connection line with all smart cars within its control range and send information collection instructions to the smart cars; A status information acquisition module, which is used for the smart car to acquire real-time status information data based on its own sensors and send the real-time status information data back to the remote control terminal; A status information analysis module, which is used by the remote control terminal to analyze the real-time status information data and generate a real-time status information processing response; A voice command matching module, which is used for the remote control terminal to receive voice commands, generate remote control commands that match the voice commands, and send the remote control commands to the smart car; A remote control operation module is used for the smart car to perform control operations based on the remote control instructions.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the autonomous driving remote control method taking voice control into consideration as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for remote control of autonomous driving with voice control as described in any one of claims 1 to 7 is implemented.

Images