Vehicle exterior environment sensing method, vehicle-mounted terminal, server and computer program product

Abstract

The invention relates to an outside-vehicle environment sensing method, a vehicle-mounted terminal, a server and a computer program product, and the method comprises the steps: obtaining first user voice information based on a voice recognition module of a vehicle, the first user voice information comprising azimuth information of a target object and indication information used for obtaining the name of the target object; controlling a camera of the vehicle to acquire an external environment image containing the target object; controlling a positioning module of the vehicle to obtain current position information of the vehicle; determining name information of the target object based on the vehicle exterior environment image, the current position information of the vehicle and the first user voice information; and prompting a user of the vehicle with the name information of the target object. According to the invention, the interaction capability between the vehicle and the user can be enhanced, and more convenient and intelligent driving experience is provided for the user.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and specifically to a method for sensing an external vehicle environment, a vehicle-mounted terminal, a server, and a computer program product. Background Art

[0002] At present, the application of environmental perception technology in vehicles, most environmental perception systems mainly focus on the autonomous driving function, generally using sensors to detect the vehicle's surrounding environment, and judging whether to trigger certain mechanisms based on the detection results, and then automatically controlling the vehicle, such as obstacle detection, lane keeping and adaptive cruise control. Therefore, the current environmental perception technology has certain limitations in the interaction between vehicles and users, focusing on specific functions such as lane keeping or obstacle detection, but not comprehensive enough and unable to dynamically adapt to users' real-time needs and behaviors.

[0003] With the continuous advancement of vehicle networking technology and artificial intelligence (AI) technology, vehicles have indeed evolved from a simple means of transportation to a complex mobile intelligent platform. This transformation has placed higher requirements on the vehicle's environmental perception system. Therefore, technological innovation is urgently needed so that the vehicle's environmental perception system will be able to better support the vehicle's intelligence and provide users with a more convenient driving experience. Summary of the invention

[0004] The purpose of this application is to propose a method for perceiving the external environment of a vehicle, a vehicle-mounted terminal, a server and a computer program product, so as to enhance the interaction capability between the vehicle and the user and provide the user with a more convenient and intelligent driving experience.

[0005] To achieve the above-mentioned purpose, according to a first aspect of the present application, a method for sensing an external environment of a vehicle is provided, the method comprising:

[0006] Acquiring first user voice information based on a voice recognition module of the vehicle, wherein the first user voice information includes position information of a target object and instruction information for acquiring a name of the target object;

[0007] Controlling the camera of the vehicle to acquire an image of the vehicle's exterior environment including the target object;

[0008] Controlling the positioning module of the vehicle to obtain the current position information of the vehicle;

[0009] Determine the name information of the target object based on the external environment image, the current position information of the vehicle and the first user voice information;

[0010] The name information of the target object is prompted to the user of the vehicle.

[0011] According to a second aspect of the present application, another method for sensing an external vehicle environment is provided, the method comprising:

[0012] Acquire an image of an external environment including a target object, current position information of the vehicle, and first user voice information uploaded by the vehicle; the first user voice information includes the position information of the target object and instruction information for acquiring the name of the target object;

[0013] Determine the name information of the target object based on the external environment image, the current position information of the vehicle and the first user voice information;

[0014] The name information of the target object is sent to the vehicle, so that the vehicle prompts the user with the name information of the target object.

[0015] According to a third aspect of the present application, a vehicle-mounted terminal is provided, wherein the vehicle-mounted terminal includes a module for executing the method described in the first aspect of the present application.

[0016] According to a fourth aspect of the present application, a server is provided, wherein the server includes a module for executing the method described in the second aspect of the present application.

[0017] According to a fifth aspect of the present application, a computer program product is provided, comprising computer program instructions, wherein the computer program instructions instruct a computer device to perform operations corresponding to the method according to the first aspect or the second aspect of the present application.

[0018] The above-mentioned vehicle external environment perception method, vehicle terminal, server and computer program product proposed in this application have the following beneficial effects:

[0019] During the use of the vehicle, the on-board terminal captures the user's voice command and parses the voice command to obtain the first user's voice information, and obtains the external environment image and the vehicle's current geographic location information. Based on the first user's voice information, the external environment image and the geographic location information, the vehicle terminal understands the question raised by the user (i.e., querying the name information of the target object, such as what is the high-rise building in the left front), and obtains the name information of the target object based on the question, and then returns it to the on-board terminal. The on-board terminal makes a voice broadcast based on the received perception results (for example, the high-rise building in front of your left is a certain building). In the above manner, the interaction ability between the vehicle and the user can be enhanced, providing the user with a more convenient and intelligent driving experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 This is a flow chart of a method for sensing an external vehicle environment in Embodiment 1 of the present application.

[0022] Figure 2 It is a schematic diagram of the interaction principle between the vehicle-mounted terminal and the server in Example 1 of the present application.

[0023] Figure 3 This is a flow chart of a method for sensing the external environment of a vehicle in Embodiment 2 of the present application. DETAILED DESCRIPTION

[0024] The detailed description of the drawings is intended as an illustration of the current embodiment of the present application, and is not intended to represent the only form in which the present application can be implemented. It should be understood that the same or equivalent functions can be accomplished by different embodiments intended to be included in the spirit and scope of the present application.

[0025] See also Figure 1 Embodiment 1 of the present application provides a method for sensing an external vehicle environment. The method is implemented based on a vehicle-mounted terminal and includes the following steps:

[0026] Step S11: acquiring first user voice information based on a voice recognition module of the vehicle, wherein the first user voice information includes position information of a target object and indication information for acquiring a name of the target object.

[0027] Specifically, in one example, the voice recognition module may be equipped with a built-in microphone, which can collect the voice commands of the user in the car and filter out background noise during the driving of the vehicle to ensure the accurate capture of the user's voice information. Finally, voice recognition technology, such as deep learning algorithm, is used to convert the user's voice commands into text information.

[0028] Specifically, the vehicle user needs to clearly indicate the direction of the target object in the voice command, such as "left front", "right rear", etc., so that the vehicle terminal can locate the target object. The user needs to provide instructions for obtaining the name of the target object in the voice command, such as "what is the name of that building", "what is that sign", etc. This is the key for the vehicle terminal to identify the target object; illustratively, the voice command may be, for example, "please tell me the name of the building in front of me on the left", "identify the meaning of the sign in the right rear", "I want to know the name of the restaurant in front", etc.

[0029] The speech recognition module converts the preprocessed sound signal into text information and extracts the orientation information and indication information of the target object. Through this step, the vehicle terminal can understand the user's query intention, laying the foundation for subsequent environmental image acquisition and target object recognition. This voice-based interaction method not only facilitates the driver to obtain information during driving, but also reflects the vehicle's intelligent and humanized design concept.

[0030] Step S12, controlling the camera of the vehicle to acquire an image of the vehicle's exterior environment including the target object.

[0031] Specifically, for example, if the user's voice command is "Please tell me the name of the building in the left front", then the vehicle exterior environment image captured by the left front camera will be obtained as the vehicle exterior environment image containing the target object. For another example, if the user's voice command is "Recognize the meaning of the sign in the right rear", then the vehicle exterior environment image captured by the right rear camera will be obtained as the vehicle exterior environment image containing the target object.

[0032] Step S13, controlling the positioning module of the vehicle to obtain the current position information of the vehicle.

[0033] Specifically, the vehicle's positioning module can be GPS (Global Positioning System), GLONASS (GLONASS), Beidou navigation system or other satellite navigation systems.

[0034] Step S14: determining the name information of the target object based on the vehicle exterior environment image, the vehicle current position information and the first user voice information.

[0035] Specifically, based on the first user voice information, the position information of the target object and the indication information for obtaining the name of the target object can be obtained, and the characteristics of each object in the image are determined by identifying the external environment image. The target object can be determined in the image based on the position information and the indication information, and then based on the target object and the current position information of the vehicle, the internal database can be queried or a third-party ecological platform interface (such as map service, business information query service, etc.) can be called to obtain the name information of the target object.

[0036] It should be noted that the process of determining the name information of the target object in step S40 of this embodiment can be implemented by the vehicle terminal alone, or it can be implemented through the interaction between the vehicle terminal and the server, that is, the vehicle terminal sends the external environment image, the current position information of the vehicle and the first user voice information to the server, and requests the server to execute the process of determining the name information of the target object based on the external environment image, the current position information of the vehicle and the first user voice information, and then the vehicle terminal receives the name information of the target object returned by the server. The method of this embodiment is not limited to any of the above methods. The specific design depends on the computing resources of the vehicle terminal. If the computing resources of the vehicle terminal are insufficient, it can be implemented on the server. If the computing resources of the vehicle terminal are sufficient, it can be implemented by the vehicle terminal alone.

[0037] Step S15: Prompt the name information of the target object to the user of the vehicle.

[0038] Specifically, the name information of the target object can be converted into a natural and fluent voice using text-to-speech (TTS) technology for broadcasting. Furthermore, in addition to voice broadcasting, the vehicle terminal can also provide visual feedback through the vehicle's central control display screen, displaying the name and / or image of the target object, highlighting the target object on the display screen, or marking the name next to it.

[0039] To summarize, the method of this embodiment is that during the use of the vehicle, the vehicle terminal captures the user's voice command and parses the voice command to obtain the first user's voice information, and obtains the external environment image and the vehicle's current geographic location information. Based on the first user voice information, the external environment image and the geographic location information, the method understands the question raised by the user (i.e., querying the name information of the target object, such as what is the high-rise building in the left front), and obtains the name information of the target object based on the question, and then returns it to the vehicle terminal. The vehicle terminal makes a voice broadcast based on the received perception results (for example, the high-rise building in front of your left is a certain building). In the above manner, the interaction ability between the vehicle and the user can be enhanced, providing the user with a more convenient and intelligent driving experience.

[0040] In some embodiments, the first user voice information also includes: description information of the target object, the description information includes at least one of the following: color information of the target object, type information of the target object, name information of objects surrounding the target object, and type information of objects surrounding the target object.

[0041] Specifically, the description information further provides more target features for determining the target object in the external environment image; the vehicle-mounted terminal or server filters objects of corresponding colors in the image according to the color information provided by the user (such as "red building"); type information (such as "commercial building") helps the vehicle-mounted terminal or server to further narrow the recognition range and only recognize specific types of buildings; if the user mentions the name information of objects surrounding the target object (such as "the building next to the bank"), the vehicle-mounted terminal can use the name information of objects surrounding the target object as an auxiliary clue to locate the target object; the type information of objects surrounding the target object (such as "the building near the park") can also help the vehicle-mounted terminal locate the target object in the image.

[0042] In some embodiments, determining the name information of the target object based on the external environment image, the current position information of the vehicle and the first user voice information includes:

[0043] Sending the first user voice information, the vehicle exterior environment image and the vehicle current location information to the server to request the server to determine the name information of the target object;

[0044] Receiving the name information of the target object returned by the server;

[0045] Specifically, Figure 2 The diagram shows the interaction principle between the vehicle terminal and the server. The vehicle terminal captures the first voice command issued by the user through a microphone or other voice input device installed on the vehicle. The first voice command is a question asked by the user about objects such as buildings or service facilities in the vehicle environment, such as "What is the building in the left front?" When the first voice command is captured, the vehicle terminal will parse the first voice command to obtain the first user voice information, and at the same time, trigger the acquisition of the vehicle environment image and the vehicle's current location information; Generally speaking, a smart vehicle is equipped with multiple cameras. During the use of the vehicle, such as Figure 2As shown, the multiple cameras are used to capture video streams of environments in different directions. After the video streams are preprocessed and compressed by the analog-to-digital conversion module (ADC), they are sent to the vehicle terminal through the RTP protocol. The vehicle terminal determines to intercept the image of the vehicle's external environment from the video stream of the corresponding camera based on the first voice information, and performs image desensitization. For example, if the user asks "What is the building in the left front?", the vehicle terminal will intercept a frame of image from the video stream of the left front camera as the image of the vehicle's external environment and perform image desensitization. The timestamp of the frame of image is consistent with the timestamp when the vehicle terminal captures the first voice command; the vehicle's current location information is the vehicle location information (latitude and longitude information) corresponding to the timestamp when the vehicle terminal captures the first voice command. The vehicle terminal generates a first query request after compressing and encrypting the first user voice information, the image of the external environment and the current location information of the vehicle, and sends the first query request to the server.

[0046] Specifically, the vehicle terminal receives the name information of the target object from the server through the network. After receiving the perception results, the vehicle terminal will process the information so that it can be conveyed to the user in a suitable voice manner. The vehicle terminal uses text to speech (TTS) technology to convert the processed perception results into voice information, and then broadcasts the information to the users in the car through the vehicle audio system. The content of the voice broadcast may be: "The building in front of your left is the landmark building XX Building in City A. It is an Art Deco-style skyscraper built in 1980 and is mainly used for office and sightseeing." Users can obtain specific information about the external environment through hearing without checking the screen or performing other operations, thereby improving safety and convenience while driving.

[0047] The server determines the name information of the target object in the following manner:

[0048] Extracting features of the first user's voice information, the vehicle's external environment image, and the vehicle's current location information to obtain a voice feature vector, an image feature vector, and a geographic location feature vector;

[0049] Fusing the speech feature vector, the image feature vector and the geographic location feature vector to obtain a fused feature vector, and making a decision based on the fused feature vector to obtain the type and probability of each object in the vehicle exterior environment image;

[0050] Generate a first query task according to the type and probability of each object;

[0051] Determine a corresponding preset expert model according to the first query task, call the preset expert model to execute the first query task, query a preset knowledge database or call a third-party ecological platform interface to obtain the name information of the target object.

[0052] Specifically, if Figure 2 As shown, in response to receiving the first query request, the server performs the following operations:

[0053] (1.1) parsing (decompressing and decrypting) the first query request to obtain the first user voice information, the vehicle exterior environment image, and the vehicle current location information;

[0054] (1.2) Feature extraction is performed on the first user voice information, the vehicle exterior environment image, and the vehicle current location information to obtain a voice feature vector, an image feature vector, and a geographic location feature vector (i.e., Figure 2 Text and image tags in );

[0055] Specifically, in this embodiment, for the convenience of description, the image feature vector is defined as V_{image}, which can be extracted by a convolutional neural network (CNN); the voice feature vector is defined as V_{voice}, which can be extracted by a natural language processing (NLP) model (such as a Transformer model); the geographic location feature vector V_{location} is defined, which can be a feature vector encoded by the geographic coordinates.

[0056] (1.3) fusing the speech feature vector, the image feature vector and the geographic location feature vector to obtain a fused feature vector, and making a decision based on the fused feature vector to obtain the type and probability of each object in the vehicle exterior environment image;

[0057] Specifically, before fusion, since the scales of feature vectors of different modalities vary greatly, V_{image}, V_{voice}, and V_{location} are first standardized to ensure that they are at the same level. This can be achieved by using techniques such as BatchNormalization to perform feature normalization. The standardized V_{image}, V_{voice}, and V_{location} are concatenated together to form a multimodal feature vector V_{multi}, which can be expressed as:

[0058] V_{multi}=[V_{image},V_{voice},V_{location}];

[0059] In this embodiment, a multimodal fusion model is designed to fuse V_{image}, V_{voice}, and V_{location}. During the splicing, the multimodal fusion model introduces an attention mechanism to assign different weights to different modalities. The attention mechanism can help the model focus on more important information sources. The attention weight can be expressed as:

[0060] a i =softmax(W a *V i );

[0061] Among them, a i represents the attention weight of the i-th modality, W a is the weight matrix, V i is the eigenvector of the ith mode.

[0062] Furthermore, the attention weights are used to perform weighted summation on the feature vectors of each modality, which can be expressed as:

[0063]

[0064] Among them, V weighted is the weighted sum result;

[0065] Finally, the activation function is used to perform nonlinear transformation on the fused feature vector to capture the complex feature relationship, which can be expressed as:

[0066] V transformed =f(w l *V weighted +b l )

[0067] Among them, V transformed is the fused feature vector, w l and b l are the weight matrix and bias vector respectively, and f() represents the activation function.

[0068] The last part of the multimodal fusion model is a decision layer, which uses reinforcement learning algorithms to optimize the decision process and can be based on V transformed Output the type and probability of each object in the vehicle exterior environment image. An example is given below to illustrate:

[0069] Input: V_transformed (fused feature vector);

[0070] Decision layer processing: The reinforcement learning algorithm processes V_transformed and predicts the object type through a series of neural network layers and strategies;

[0071] Output: The decision layer outputs a series of object types and corresponding probabilities, for example, "building" with a probability of 0.85, "traffic sign" with a probability of 0.10, and "animal" with a probability of 0.05.

[0072] (1.4) generating a first query task according to the types and probabilities of the objects;

[0073] Specifically, according to the type and probability of each object, the relevant feature information of an object with the highest probability is selected as the retrieval information for the expert model to perform knowledge query. In the above example, "building" with a probability of 0.85 is selected as the target object to be queried. At this time, content mapping is performed to generate the first query task. The first query task includes the relevant feature information of the "building" with a probability of 0.85, as well as the first user voice information, the external environment image and the current position information of the vehicle.

[0074] (1.5) Determine a corresponding preset expert model according to the first query task, call the preset expert model to execute the first query task, query a preset knowledge database or call a third-party ecological platform interface to obtain the name information of the target object.

[0075] For example, when the first query task is about the recognition of "buildings", it will be assigned to the corresponding expert model according to the first query task. This expert model is specially designed for identifying and providing building-related information. The selected expert model receives the first query task and starts to perform the following operations: the expert model parses the information in the first query task, including the visual features of the building, the user's voice description, the image of the vehicle's external environment, and the vehicle's location information. The expert model first queries the preset knowledge database, which contains relevant information about the building, such as the building's name, history, and purpose. If a matching result can be found in the knowledge database, the result is output as the name information of the target object; if a matching result cannot be found in the knowledge database, or more detailed or specific information is required, the third-party ecological platform interface is called, such as a map service, to obtain more comprehensive data. Specifically, the name information of the target object may include the name, type, and other relevant attributes of the building, such as "the building in front of you on the left is [building name], it is a [architectural style] building, built in [year of construction], and mainly used for [purpose]".

[0076] During the use of the vehicle, the vehicle terminal captures the user's voice command and parses the voice command to obtain the first user's voice information, and at the same time triggers the acquisition of the external environment image and the vehicle's current geographic location information, and sends the first user's voice information, external environment image and geographic location information to the server through the Internet of Vehicles technology; the server understands the question raised by the user based on the first user voice information, external environment image and geographic location information, and obtains the name information of the target object based on the question, and then returns it to the vehicle terminal, and the vehicle terminal performs voice broadcast according to the received perception results. In the above manner, the interaction ability between the vehicle and the user can be enhanced, providing the user with a more convenient and intelligent driving experience.

[0077] In some embodiments, the method further comprises:

[0078] Step S16, receiving the first location service information or the first vehicle use suggestion related to the target object returned by the server, and prompting the user of the vehicle with the first location service information or the first vehicle use suggestion;

[0079] The first location service information or the first vehicle use suggestion is obtained by the server according to the name information of the target object.

[0080] Specifically, in this embodiment, the vehicle terminal not only receives the name information of the target object, but also receives the location service information or vehicle use suggestions related to these objects. While the server analyzes the target object perception results, the server also provides additional information based on the target object perception results, namely the first location service information or the first vehicle use suggestion. The first location service information is, for example, information about nearby restaurants, parking lots, gas stations, hotels and other service facilities, as well as their business hours, user ratings, contact information, etc. The first vehicle use suggestion is, for example, a suggestion based on factors such as the vehicle's current location, destination, traffic conditions, vehicle status (such as fuel level, power level), such as a recommended driving route, the location of a gas station or charging station, a vehicle maintenance reminder, etc.

[0081] Furthermore, after receiving this information, the vehicle terminal will process it to select the most relevant and useful information to broadcast to the user. The processing process may include filtering, sorting and formatting the information to ensure that the content of the broadcast is concise and clear. The vehicle terminal uses text-to-speech (TTS) technology to convert the processed venue service information or vehicle use suggestions into voice and broadcast it to the user through the vehicle audio system. For example, the broadcast content may be: "The gas station 1 km ahead of you currently has a discount event. It is recommended that you go there to refuel. In addition, the nearby 'Food Paradise' restaurant offers special dishes. You can go there to dine after refueling."

[0082] Through step S16, the vehicle-mounted terminal not only provides direct information about the external environment, but also provides practical service information and personalized vehicle usage suggestions, thereby further enhancing the functionality and user-friendliness of the vehicle-mounted system. This integrated service information provision makes the vehicle not just a means of transportation, but an intelligent partner in the user's daily life.

[0083] In some embodiments, the method further comprises:

[0084] Step S17: acquiring second user voice information based on the voice recognition module of the vehicle, wherein the second user voice information includes instruction information for acquiring location service information or vehicle use suggestions related to the target object.

[0085] Step S18: sending the second user voice information to the server to request the server to determine the location service information or vehicle use recommendation related to the target object.

[0086] Specifically, the vehicle-mounted terminal captures a second voice command issued by the user through a microphone or other voice input device installed thereon. The second voice command is a question asked by the user further about the target object (such as a building or service facility). For example, the first voice command is to ask "What is the building in front of the left?", and the second voice command can be "Is there a coffee shop in this building?", etc. When the vehicle-mounted terminal captures the first voice command, it will create a new session. During this session, if the second voice command is captured, the vehicle-mounted terminal will parse the second voice command to obtain the second user voice information. At this time, the vehicle-mounted terminal will determine whether to create a new session based on the second user voice information. If not, it will not trigger the capture of the external environment image and the acquisition of the current vehicle location information. Instead, the server will use the context information of the current session to process the command. The vehicle-mounted terminal will generate a second query request based on the second user voice information and send the second query request to the server.

[0087] Step S19: receiving the second location service information or the second vehicle use suggestion returned by the server, and prompting the user of the vehicle with the second location service information or the second vehicle use suggestion.

[0088] For example, after receiving the information, the in-vehicle terminal can tell the user through voice broadcast: "There is a large underground parking lot in the shopping mall. There are currently vacancies. The entrance is on the north side. You can turn left at the next intersection." Through steps S17 to S19, the server can use the user's voice commands and conversation memory data to provide accurate and timely service information through expert models, providing users with a more consistent and personalized service experience.

[0089] The server determines the location service information or vehicle usage suggestion related to the target object in the following manner:

[0090] Acquire the memory data of the current session, wherein the memory data includes all interaction data between the vehicle and the server during the session;

[0091] Generate a second query task according to the second user voice information and the memory data;

[0092] Determine the corresponding preset expert model according to the second query task, and call the preset expert model to execute the second query task, query the preset knowledge database or call the third-party ecological platform interface to obtain the second place service information or second car use recommendation related to the target object.

[0093] In response to receiving the second query request, the server performs the following operations:

[0094] (2.1) parsing the second query request to obtain the second user voice information;

[0095] Specifically, the server has short-term / long-term memory capabilities. When the server parses the second query request and only obtains information in the second user's voice mode, the server can determine that it should use the context information of the current conversation to process the second query request to achieve multiple rounds of conversations.

[0096] (2.2) Obtaining memory data of the current session, wherein the memory data includes all interaction data between the vehicle and the server during the session;

[0097] Specifically, the said all interaction data includes all information of the vehicle's interaction with the server in the current session, which may include but is not limited to user voice command records, images of the outside environment, vehicle location information and server response records. The said user voice command records refer to the historical records of all voice commands given by the user in the current session, such as the first user's voice information; the server response records refer to all information returned by the server to the vehicle during the session, including the name information of the target object, venue service information and vehicle usage suggestions, etc.

[0098] The memory data provides the server with the contextual information required to process the second query request. For example, if the user's second voice command is about a specific building mentioned previously, the server can quickly locate the relevant information of the building through the memory data. By analyzing the memory data, the server can better understand the user's intentions and needs, thereby providing more accurate and relevant service information. With the memory data, the server does not need to repeat previously completed calculations or queries, and can respond quickly based on the existing information, thereby improving the overall efficiency of the system. The memory data is stored in the server's temporary storage. The server ensures the security and privacy of this data, which is only used to provide and improve in-vehicle services and will not be disclosed to third parties.

[0099] (2.3) generating a second query task according to the second user voice information and the memory data;

[0100] Specifically, the server generates a second query task by performing content mapping based on the second user voice information and the memory data. For example, in the aforementioned example, the second query task may include the name information of the target object as well as the second user voice information, the image of the vehicle's external environment, and the vehicle's current location information. The server determines the corresponding preset expert model based on the second query task, and calls the preset expert model to execute the second query task, query the preset knowledge database, or call the third-party ecological platform interface to obtain the name information of the target object.

[0101] In one example, suppose a user issues the following first voice command through a vehicle terminal while driving: "What is that building in front of the left?" The server has returned an answer based on the first voice command and the vehicle's location information: "That is a shopping mall in the city center." Then, the user issues a second voice command: "I want to know if there is a parking lot there." The server receives the user's second voice command: "I want to know if there is a parking lot there."

[0102] The server extracts information related to the first query from the memory data, including: "building" related feature information with a probability of 0.85 (here refers to the shopping mall), the first user's voice information ("What is the building in the front left?"), and the external environment image and the vehicle's current location information. The server maps this information to a specific second query task, which is used to determine whether there is a parking lot in the city center shopping mall.

[0103] (2.4) Determine a corresponding preset expert model according to the second query task, and call the preset expert model to execute the second query task, query a preset knowledge database or call a third-party ecological platform interface to obtain second place service information or second vehicle use recommendation related to the target object.

[0104] Specifically, the server determines which preset expert model to use based on the generated second query task. In this example, it may be a model that specializes in processing venue service information, such as a "parking lot information expert model". The server calls the "parking lot information expert model" to execute the query task, and the model performs the following operations:

[0105] Query the preset knowledge database, which contains detailed information about the shopping mall, including parking lot location, capacity, current usage, etc., or call the third-party ecological platform interface, such as the city’s intelligent transportation system, to obtain real-time parking lot information.

[0106] After the expert model is executed, the server returns the acquired parking lot information to the vehicle terminal.

[0107] In some embodiments, the method further comprises:

[0108] Step S110, after prompting the first car usage suggestion or the second car usage suggestion to the user of the vehicle, wait for a preset time. If a third user voice message is received within the preset time, determine whether to execute the vehicle control function corresponding to the first car usage suggestion or the second car usage suggestion according to the third user voice message.

[0109] Specifically, step S110 is an interactive feedback and vehicle control process, which allows the vehicle system to decide whether to execute the corresponding vehicle control function according to the user's further instructions after providing the vehicle use suggestion. Specifically, after receiving the first vehicle use suggestion or the second vehicle use suggestion returned by the server, the vehicle terminal will inform the user through voice broadcast, for example, broadcast: "XX building is in front of you on the left", and at the same time, it will also provide relevant service information or vehicle use suggestions, for example, broadcast "The building is 200 meters away from your destination, you can park, the parking fee is 10 yuan / hour, do you want to navigate to park?".

[0110] After the announcement is completed, the vehicle system will enter a waiting state. The preset time may be a few seconds, which is enough for the user to understand the suggestions and respond. This preset time is designed based on the user experience. It should not be too short, so that the user will not have time to react, nor too long, so as to avoid unnecessary waiting.

[0111] If the user responds to the suggestion of the vehicle-mounted system within the preset time, that is, the vehicle-mounted terminal receives the third voice command issued by the user, parses the third voice command to obtain the third user voice information, and determines whether the user agrees to execute the vehicle control function in the vehicle use suggestion according to the received third user voice information. If the user agrees (for example, the user says: "OK"), the vehicle-mounted system will execute the corresponding vehicle control function, such as navigating to park. If the user disagrees (for example, the user says: "No"), the vehicle-mounted system will not execute the suggested vehicle control function, but will continue to maintain the current driving state or wait for further instructions from the user.

[0112] See also Figure 3 , Embodiment 2 of the present application proposes a method for sensing an external vehicle environment, comprising the following steps:

[0113] Step S21, obtaining an external environment image containing a target object, current position information of the vehicle, and first user voice information uploaded by the vehicle; the first user voice information contains the orientation information of the target object and instruction information for obtaining the name of the target object.

[0114] Step S22: determining the name information of the target object based on the vehicle exterior environment image, the vehicle current position information and the first user voice information.

[0115] Step S23: sending the name information of the target object to the vehicle, so that the vehicle prompts the user with the name information of the target object.

[0116] In some embodiments, the first user voice information also includes: description information of the target object, the description information includes at least one of the following: color information of the target object, type information of the target object, name information of objects surrounding the target object, and type information of objects surrounding the target object.

[0117] In some embodiments, step S22 includes:

[0118] Step S221, performing feature extraction on the first user voice information, the vehicle exterior environment image and the vehicle current location information to obtain a voice feature vector, an image feature vector and a geographic location feature vector;

[0119] Step S222, fusing the speech feature vector, the image feature vector and the geographic location feature vector to obtain a fused feature vector, and making a decision based on the fused feature vector to obtain the type and probability of each object in the vehicle exterior environment image;

[0120] Step S223, generating a first query task according to the type and probability of each object;

[0121] Step S224, determining the corresponding preset expert model according to the first query task, and calling the preset expert model to execute the first query task, querying the preset knowledge database or calling the third-party ecological platform interface to obtain the name information of the target object.

[0122] In some embodiments, step S22 further includes:

[0123] Step S225, obtaining first place service information or first car use suggestion related to the target object according to the name information of the target object, and sending the first place service information or first car use suggestion to the vehicle, so that the vehicle prompts the user with the first place service information or first car use suggestion.

[0124] In some embodiments, the method further comprises:

[0125] Step S241, obtaining second user voice information sent by the vehicle, wherein the second user voice information includes instruction information for obtaining location service information or vehicle use suggestions related to the target object;

[0126] Step S242, obtaining memory data of the current session, wherein the memory data includes all interaction data between the vehicle and the server during the session;

[0127] Step S243, generating a second query task according to the second user voice information and the memory data;

[0128] Step S244, determining a corresponding preset expert model according to the second query task, and calling the preset expert model to execute the second query task, querying a preset knowledge database or calling a third-party ecological platform interface to obtain second place service information or second car use suggestion related to the target object;

[0129] Step S245: sending the second location service information or the second car use suggestion to the vehicle, so that the vehicle prompts the user with the second location service information or the second car use suggestion.

[0130] It should be noted that the method described in Example 2 corresponds to the method described in Example 1, and the specific principles of the server executing the above steps of the method described in Example 2 have been disclosed in detail in the description of the method described in Example 1. Therefore, the relevant technical details of the above steps of the method described in Example 2 can be obtained and understood by referring to the description of the method described in Example 1, so they will not be repeated in this embodiment.

[0131] In some embodiments, the calling of the third-party ecological platform interface to obtain the name information of the target object includes:

[0132] Generate a call request according to the current location information of the vehicle, and send the call request to the third-party ecological platform interface;

[0133] Receive the point of interest information returned by the third-party ecological platform interface, and obtain the name information of the target object according to the point of interest information; wherein the point of interest information is the relevant information corresponding to the location of the target object in the geographic information system.

[0134] Specifically, point of interest (POI) information refers to the relevant information of a specific location in a geographic information system (GIS). The relevant information of this specific location is of special interest or importance to certain users or applications. Point of interest information can include various types of data. Some common points of interest are, for example, commercial facilities, public service places, leisure and entertainment places, and transportation facilities. Specific point of interest information includes name, location, type, address, contact information, business hours, user ratings, service information, etc.

[0135] In an embodiment, the server can obtain the name information of the target object by calling the third-party ecological platform interface, which involves data exchange and integration with external services. The server first needs to generate a call request based on the vehicle's current location information (latitude and longitude coordinates). This request contains necessary information, such as the vehicle's location, the type of request (for example, obtaining nearby points of interest), and possible other parameters (such as search radius, type of interest, etc.). The format of the request follows the API specification of the third-party ecological platform interface to ensure that the data can be correctly parsed and processed.

[0136] After generating the call request, the server will send it to the API interface of the third-party ecological platform through the network. This interface may be a map service, location information service or any service that provides geographic information data. After receiving the call request through this interface, the third-party ecological platform will query the relevant point of interest information in the geographic information system according to the request parameters. This point of interest information includes, for example, the name, address, type, user rating, business hours, etc. of the business. After the query is completed, the third-party ecological platform will return the point of interest information to the server as a response; after receiving the point of interest information from the third-party ecological platform, the server will parse and process it. The server can filter out the most relevant point of interest information based on the specific location of the target object. The server can also combine this information with the name information of the target object to provide richer and more accurate data. For example, if the target object is a building, the server can also return the name of the building, its main functions, the location of the entrance, and other information.

[0137] Embodiment 3 of the present application provides a vehicle-mounted terminal, including a module for executing the vehicle exterior environment perception method described in Embodiment 1, and the module may be a software module, a hardware module, or a combination of software and hardware.

[0138] Embodiment 4 of the present application provides a server, including a module for executing the vehicle exterior environment perception method described in Embodiment 2, and the module may be a software module, a hardware module, or a combination of software and hardware.

[0139] Embodiment 5 of the present application provides a computer program product, including computer program instructions, wherein the computer program instructions instruct a computer device to perform operations corresponding to the method described in Embodiment 1.

[0140] Specifically, the computer program product includes a series of computer program instructions, which can instruct a computer device to execute the vehicle exterior environment perception method described in Embodiment 1. These instructions are codes written in a computer program, which define how to perform specific operations. In this embodiment, these instructions are used to execute the vehicle exterior environment perception method described in Embodiment 1.

[0141] These program instructions are designed to be loaded onto a computer device and instruct the device to perform specific operations, which are the various steps in the method for sensing the external vehicle environment described in the first embodiment.

[0142] In this way, the computer program product provides a complete software solution, which can be run on various computer devices to implement the method for perceiving the vehicle's external environment in the first embodiment.

[0143] Embodiment 6 of the present application provides a computer program product, including computer program instructions, wherein the computer program instructions instruct a computer device to perform operations corresponding to the method described in Embodiment 2.

[0144] Specifically, the computer program product includes a series of computer program instructions, which can instruct a computer device to execute the vehicle exterior environment perception method described in Example 2. These instructions are codes written in a computer program, which define how to perform specific operations. In this embodiment, these instructions are used to execute the vehicle exterior environment perception method of Example 2.

[0145] These program instructions are designed to be loaded onto a computer device and instruct the device to perform specific operations, which are the various steps in the method for sensing the vehicle's external environment as described in the second embodiment above.

[0146] In this way, the computer program product provides a complete software solution, which can be run on various computer devices to implement the vehicle exterior environment perception method of the above-mentioned embodiment 2.

[0147] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for sensing an external vehicle environment, characterized in that: The method comprises: Acquiring first user voice information based on a voice recognition module of the vehicle, wherein the first user voice information includes position information of a target object and instruction information for acquiring a name of the target object; Controlling the camera of the vehicle to acquire an image of the vehicle's exterior environment including the target object; Controlling the positioning module of the vehicle to obtain the current position information of the vehicle; Determine the name information of the target object based on the external environment image, the current position information of the vehicle and the first user voice information; The name information of the target object is prompted to the user of the vehicle.

2. The method according to claim 1, characterized in that The first user voice information also includes: description information of the target object, the description information includes at least one of the following: color information of the target object, type information of the target object, name information of objects surrounding the target object, and type information of objects surrounding the target object.

3. The method according to claim 1 or 2, characterized in that: The determining the name information of the target object based on the vehicle exterior environment image, the vehicle current location information and the first user voice information includes: Sending the first user voice information, the vehicle exterior environment image and the vehicle current location information to the server to request the server to determine the name information of the target object; Receiving the name information of the target object returned by the server; The server determines the name information of the target object in the following manner: Extracting features of the first user's voice information, the vehicle's external environment image, and the vehicle's current location information to obtain a voice feature vector, an image feature vector, and a geographic location feature vector; Fusing the speech feature vector, the image feature vector and the geographic location feature vector to obtain a fused feature vector, and making a decision based on the fused feature vector to obtain the type and probability of each object in the vehicle exterior environment image; Generate a first query task according to the type and probability of each object; Determine a corresponding preset expert model according to the first query task, call the preset expert model to execute the first query task, query a preset knowledge database or call a third-party ecological platform interface to obtain the name information of the target object.

4. The method according to claim 3, characterized in that The method further comprises: receiving first location service information or first vehicle use suggestion related to the target object returned by the server, and prompting the user of the vehicle with the first location service information or the first vehicle use suggestion; The first location service information or the first vehicle use suggestion is obtained by the server according to the name information of the target object.

5. The method according to claim 4, characterized in that The method further comprises: Acquiring second user voice information based on a voice recognition module of the vehicle, where the second user voice information includes instruction information for acquiring location service information or vehicle use suggestions related to the target object; sending the second user voice information to the server to request the server to determine the location service information or vehicle use suggestion related to the target object; receiving the second location service information or the second vehicle use suggestion returned by the server, and prompting the user of the vehicle with the second location service information or the second vehicle use suggestion; The server determines the location service information or vehicle use suggestion related to the target object in the following manner: Acquire the memory data of the current session, wherein the memory data includes all interaction data between the vehicle and the server during the session; Generate a second query task according to the second user voice information and the memory data; Determine the corresponding preset expert model according to the second query task, and call the preset expert model to execute the second query task, query the preset knowledge database or call the third-party ecological platform interface to obtain the second place service information or second car use recommendation related to the target object.

6. The method according to claim 4 or 5, characterized in that: The method further comprises: After prompting the first car usage suggestion or the second car usage suggestion to the user of the vehicle, wait for a preset time. If a third user voice message is received within the preset time, determine whether to execute the vehicle control function corresponding to the first car usage suggestion or the second car usage suggestion according to the third user voice message.

7. A method for sensing an external vehicle environment, characterized in that: The method comprises: Acquire an image of an external environment including a target object, current position information of the vehicle, and first user voice information uploaded by the vehicle; the first user voice information includes the position information of the target object and instruction information for acquiring the name of the target object; Determine the name information of the target object based on the external environment image, the current position information of the vehicle and the first user voice information; The name information of the target object is sent to the vehicle, so that the vehicle prompts the user with the name information of the target object.

8. The method according to claim 7, characterized in that The first user voice information also includes: description information of the target object, the description information includes at least one of the following: color information of the target object, type information of the target object, name information of objects surrounding the target object, and type information of objects surrounding the target object.

9. The method according to claim 7 or 8, characterized in that The determining the name information of the target object based on the vehicle exterior environment image, the vehicle current location information and the first user voice information includes: Extracting features of the first user's voice information, the vehicle's external environment image, and the vehicle's current location information to obtain a voice feature vector, an image feature vector, and a geographic location feature vector; Fusing the speech feature vector, the image feature vector and the geographic location feature vector to obtain a fused feature vector, and making a decision based on the fused feature vector to obtain the type and probability of each object in the vehicle exterior environment image; Generate a first query task according to the type and probability of each object; Determine a corresponding preset expert model according to the first query task, call the preset expert model to execute the first query task, query a preset knowledge database or call a third-party ecological platform interface to obtain the name information of the target object.

10. The method according to claim 9, characterized in that The method further comprises: The first place service information or the first car use suggestion related to the target object is obtained according to the name information of the target object, and the first place service information or the first car use suggestion is sent to the vehicle, so that the vehicle prompts the user with the first place service information or the first car use suggestion.

11. The method according to claim 10, characterized in that The method further comprises: Acquire a second user voice message sent by the vehicle, where the second user voice message includes instruction information for acquiring location service information or vehicle use suggestions related to the target object; Acquire the memory data of the current session, wherein the memory data includes all interaction data between the vehicle and the server during the session; Generate a second query task according to the second user voice information and the memory data; Determine a corresponding preset expert model according to the second query task, call the preset expert model to execute the second query task, query a preset knowledge database or call a third-party ecological platform interface to obtain second place service information or second car use suggestion related to the target object; The second location service information or the second car usage suggestion is sent to the vehicle, so that the vehicle prompts the user with the second location service information or the second car usage suggestion.

12. The method according to claim 11, characterized in that The calling of the third-party ecological platform interface to obtain the name information of the target object includes: Generate a call request according to the current location information of the vehicle, and send the call request to the third-party ecological platform interface; Receive the point of interest information returned by the third-party ecological platform interface, and obtain the name information of the target object according to the point of interest information; wherein the point of interest information is the relevant information corresponding to the location of the target object in the geographic information system.

13. A vehicle-mounted terminal, characterized in that: It comprises a module for executing the method for sensing the vehicle's external environment as described in any one of claims 1 to 6.

14. A server, characterized in that: It comprises a module for executing the method for sensing the vehicle's external environment as described in any one of claims 7 to 12.

15. A computer program product, characterized in that The method comprises computer program instructions, wherein the computer program instructions instruct a computer device to execute operations corresponding to the method according to any one of claims 1 to 12.

Images